Friday, 1 June 2012

pull-debian-source with apt-get or "how to download a Debian package from Ubuntu the hard but fun way"

Have you ever wanted to download a Debian package from an Ubuntu system? Well you are not alone: hidden away in the wonderful "ubuntu-dev-tools" package is the tool for you - the excellent pull-debian-source, written by tumbleweed. This is a really useful tool which is also fast!

$ sudo apt-get install -y ubuntu-dev-tools
$ pull-debian-source hello
pull-debian-source: Downloading hello version 2.8-1
pull-debian-source: Downloading hello_2.8.orig.tar.gz from ftp.debian.org (0.665 MiB)
pull-debian-source: Downloading hello_2.8-1.debian.tar.gz from ftp.debian.org (0.006 MiB)
dpkg-source: info: extracting hello in hello-2.8                   
dpkg-source: info: unpacking hello_2.8.orig.tar.gz
dpkg-source: info: unpacking hello_2.8-1.debian.tar.gz
Or, to download the at package from the sid release:
$ pull-debian-source at sid
pull-debian-source: Downloading at version 3.1.13-1
pull-debian-source: Downloading at_3.1.13.orig.tar.gz from ftp.debian.org (0.117 MiB)
pull-debian-source: Downloading at_3.1.13-1.diff.gz from ftp.debian.org (0.009 MiB)
dpkg-source: info: extracting at in at-3.1.13                      
dpkg-source: info: unpacking at_3.1.13.orig.tar.gz
dpkg-source: info: applying at_3.1.13-1.diff.gz
$ 
However, I had cause to pore over the apt.conf(5) man page recently, and wondered if it would be possible to do the same thing, but using only apt-get(8). Turns out it is...

The first thing to do is create a few directories:
$ dir=~/.cache/pull-debian-source.sh
$ mkdir -p $dir
$ for d in etc/trusted.gpg.d etc/preferences.d var log; do mkdir -p $dir/$d; done
Now, we need a sources.list file so apt-get knows where to download index and package files from. I'm creating one for Debian sid:
$ export conf=$dir/etc/sources.list
$ cat >>$conf<<EOT
deb http://ftp.uk.debian.org/debian/ sid main
deb-src http://ftp.uk.debian.org/debian/ sid main
deb http://ftp.uk.debian.org/debian/ experimental main
deb-src http://ftp.uk.debian.org/debian/ experimental main
EOT
Next, we need to grab a copy of the package cache containing the index files. This is where the magic comes in as we use some of the amazing array of apt options:
$ export options="\
  -o Dir=\"$dir\" \
  -o Dir::Etc=\"$dir/etc\" \
  -o Dir::State=\"$dir/var\" \
  -o Dir::Cache=\"$dir/var\" \
  -o Dir::Log=\"$dir/log\" \
  -o APT::Get::AllowUnauthenticated=yes \
  -o Debug::NoLocking=true"
As you can see, we've specified alternative locations for the directories apt cares about. We've also turned off locking and authentication. The locking option isn't a problem unless you happen to be running multiple copies of apt-get as yourself. The authentication option needs might be a problem for you - check the man page for details. Now we've set the options, lets create the cache:
$ eval "apt-get $options update"
Get:1 http://ftp.uk.debian.org sid InRelease [234 kB]
Get:2 http://ftp.uk.debian.org experimental InRelease [162 kB]
Ign http://ftp.uk.debian.org sid InRelease                    
Ign http://ftp.uk.debian.org experimental InRelease           
Get:3 http://ftp.uk.debian.org sid/main Sources/DiffIndex [7876 B]
Get:4 http://ftp.uk.debian.org sid/main i386 Packages/DiffIndex [7876 B]
Hit http://ftp.uk.debian.org sid/main TranslationIndex
Get:5 http://ftp.uk.debian.org experimental/main Sources/DiffIndex [7819 B]
Get:6 http://ftp.uk.debian.org experimental/main i386 Packages/DiffIndex [7819 B]
Hit http://ftp.uk.debian.org experimental/main TranslationIndex
Hit http://ftp.uk.debian.org sid/main Translation-en
Hit http://ftp.uk.debian.org experimental/main Translation-en
Fetched 407 kB in 2s (194 kB/s)
Reading package lists... Done
W: GPG error: http://ftp.uk.debian.org sid InRelease: No keyring installed in /home/james/.cache/pull-debian-source.sh/etc/trusted.gpg.d/.
W: GPG error: http://ftp.uk.debian.org experimental InRelease: No keyring installed in /home/james/.cache/pull-debian-source.sh/etc/trusted.gpg.d/.
$ 
We got a couple of warnings ("W:" entries above), but they are not fatal. Now we've got a local cache of package meta-data, let's download a package from Debian:
$ eval "apt-get $options source cron"
Reading package lists... Done
Building dependency tree... Done
NOTICE: 'cron' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/git/pkg-cron/pkg-cron.git
Need to get 158 kB of source archives.
Get:1 http://ftp.uk.debian.org/debian/ sid/main cron 3.0pl1-121 (dsc) [1267 B]
Get:2 http://ftp.uk.debian.org/debian/ sid/main cron 3.0pl1-121 (tar) [59.2 kB]
Get:3 http://ftp.uk.debian.org/debian/ sid/main cron 3.0pl1-121 (diff) [97.2 kB]
Fetched 158 kB in 0s (206 kB/s)
dpkg-source: info: extracting cron in cron-3.0pl1
dpkg-source: info: unpacking cron_3.0pl1.orig.tar.gz
dpkg-source: info: applying cron_3.0pl1-121.diff.gz
dpkg-source: info: upstream files that have been modified: 
 cron-3.0pl1/Makefile
 cron-3.0pl1/README
 cron-3.0pl1/compat.c
 cron-3.0pl1/compat.h
 cron-3.0pl1/config.h
 cron-3.0pl1/cron.8
 cron-3.0pl1/cron.c
 cron-3.0pl1/cron.h
 cron-3.0pl1/crontab.1
 cron-3.0pl1/crontab.5
 cron-3.0pl1/crontab.c
 cron-3.0pl1/database.c
 cron-3.0pl1/do_command.c
 cron-3.0pl1/entry.c
 cron-3.0pl1/env.c
 cron-3.0pl1/externs.h
 cron-3.0pl1/job.c
 cron-3.0pl1/misc.c
 cron-3.0pl1/pathnames.h
 cron-3.0pl1/popen.c
 cron-3.0pl1/user.c
$
By the way, if you want to see all the options available to apt, run the following:
$ apt-config dump
APT "";
APT::Architecture "i386";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^kfreebsd-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
APT::NeverAutoRemove:: "^linux-ubuntu-modules-.*";
APT::NeverAutoRemove:: "^gnumach$";
APT::NeverAutoRemove:: "^gnumach-image.*";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Periodic "";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Update "";
APT::Update::Post-Invoke-Success "";
APT::Update::Post-Invoke-Success:: "touch /var/lib/apt/periodic/update-success-stamp 2>/dev/null || true";
APT::Update::Post-Invoke-Success:: "[ ! -f /var/run/dbus/system_bus_socket ] || /usr/bin/dbus-send --system --dest=org.debian.apt --type=signal /org/debian/apt org.debian.apt.CacheChanged || true";
APT::Archives "";
APT::Archives::MaxAge "30";
APT::Archives::MinAge "2";
APT::Archives::MaxSize "500";
APT::Changelogs "";
APT::Changelogs::Server "http://changelogs.ubuntu.com/changelogs";
APT::Architectures "";
APT::Architectures:: "i386";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "1";
APT::Compressor::.::CompressArg "";
APT::Compressor::.::CompressArg:: "";
APT::Compressor::.::UncompressArg "";
APT::Compressor::.::UncompressArg:: "";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "2";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-9n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "3";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-9";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "lzma";
APT::Compressor::lzma::Cost "4";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "-9";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "5";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::lzma "/usr/bin/lzma";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
Acquire::Languages:: "fr";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo > /var/lib/update-notifier/updates-available; fi ";
Unattended-Upgrade "";
Unattended-Upgrade::Allowed-Origins "";
Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-security";
CommandLine "";
CommandLine::AsString "apt-config dump";
$ 

Admittedly, this technique is rather slower than pull-debian-source (atleast for the initial download of the index files), but it is an instructive example of what is possible when you RTFM ;-)

I've bundled the above into a pull-debian-source.sh script. Run it like this:
$ chmod 755 ./pull-debian-source.sh
$ ./pull-debian-source.sh update
$ ./pull-debian-source.sh source hello