Using Debian packages for Python deployments

Virtual environments

There's one more significant thing, which is that Python packages favor virtual environments over system wide installations. Why do we say this?

• easy_install doesn't support removal of packages
• pip does support removal of packages but does not support anything like apt-get autoremove

One of our problems with virtual environments is that we've seen them break in various ways, for example because of security updates to the system-wide Python installation (Google for ImportError: cannot import name urandom). This is one of the reasons why we prefer virtual machines over virtual environments to isolate our production deployments.

Making sense of the Python packaging ecosystem

As mentioned above the Python packaging ecosystem is a bit of a mess. For an overview of the situation and some of the problems, refer to the following external resources:

• The Hitchhiker's Guide to Packaging: Current State of Packaging
• Python Packaging: Hate, hate, hate everywhere
• setuptools versioning - wtf?

I sometimes hear people call Debian package management complex. They certainly have a point, but as a DevOps who wants to do their work properly, Debian and Python are both complex, the difference is that Debian is (mostly) a pleasure to work with... The Python packaging ecosystem is very fragmented and underdocumented while in Debian there is almost always a canonical, documented way to do things properly.

Creating your first Debian package

To create a simple Debian package we only need a single file and a single command. Here's how you get started:

# Create a directory to hold the files contained in the package.
mkdir my-package && cd my-package

# Create a directory with the package's control files.
mkdir DEBIAN

# Create the main control file with package metadata.
cat > DEBIAN/control << EOF
Package: name-of-package
Version: 1.0
Section: universe/web
Priority: optional
Architecture: all
Installed-Size: 1
Maintainer: $USER
Description: Explanation of why name-of-package is so cool
EOF

# Build the package using the accepted naming scheme.
NAME=$(awk '/^Package:/ {print $2}' DEBIAN/control)
VERSION=$(awk '/^Version:/ {print $2}' DEBIAN/control)
ARCH=$(awk '/^Architecture:/ {print $2}' DEBIAN/control)
dpkg-deb --build . $NAME_${VERSION}_$ARCH.deb

Assuming you're on a Debian/Ubuntu system, the above commands should be enough to build a simple package. Any files in the working directory (excluding the special DEBIAN directory) will be included in the package as if the directory containing the DEBIAN package is the root of the file system.

The resulting *.deb file can be installed using dpkg -i $filename, however this won't automatically install dependencies, instead dpkg will error out when dependencies are missing... When this happens you can run apt-get install -f to install the dependencies. After that you can rerun the dpkg command; it should now succeed. Read on if you're interested in a more streamlined experience.

Creating a Debian package repository

As mentioned earlier dpkg doesn't automatically pull in dependencies. If you use apt-get it will do what you expect however apt-get does not support installation of local *.deb archives; it needs a repository. In other words, once you start using dependencies you will want to setup a Debian package repository for your packages! Here's how you get started:

# Create repository layout, copy packages.
mkdir -p repo/binary
cp *.deb repo/binary
cd repo

# Create list of packages.
rm -f Packages Packages.gz Release Release.gpg  # cleanup after previous run
dpkg-scanpackages -m . | sed 's@: \./@: @' > Packages

# Create compressed copy of list.
cat Packages | gzip > Packages.gz

# Generate release file.
rm -f Release Release.gpg  # cleanup after previous run
LANG= apt-ftparchive release . > Release.tmp
mv Release.tmp Release

# Sign release file.
rm -f Release.gpg  # cleanup after previous run
gpg -abs -o Release.gpg Release

There are several gotcha's in the above piece of shell script:

• We cleanup generated files from previous runs because their presence corrupts the generated files
• We modify the output of dpkg-scanpackages to change for example ./test_1.0_all.deb to test_1.0_all.deb (for some reason apt-get doesn't like Packages files with leading ./ fragments)
• We clear the $LANG environment variable so that we are sure the Release file is properly formatted regardless of the value of $LANG
• You need to have a private GPG key to sign the Release file; if you don't have one yet you'll need to create one using the command gpg --gen-key (you may find this GPG quick start useful)

deb http://server-address ./

ssh build-server sudo -i gpg --armor --export | ssh target-host sudo apt-key add -