Cover Image

This post shows how to easily deploy any Python application in form of an ‘omnibus’ Debian package, i.e. one that contains all the application's dependencies, just like in a Java WAR. A basic understanding of Debian packaging, the Linux command prompt, and Python tooling is assumed.

Introduction

In this article, I'll show how to use dh-virtualenv to create self-contained Debian packages to deploy a Python application. The resulting package is very similar to a executable JAR that you can start via java -jar, in that it contains all the moving parts except Python itself, without influencing or being influenced by version requirements of other applications. This also frees you from being restricted to the dependencies and their versions found on your target platforms, and makes porting to several different target environments easier.

The advantage of using a Debian package for deployment as opposed to the native Python tool chain is that you are less dependent on typical development tools and services, i.e. to deploy to QA or production environments you need neither Internet access nor any compiler suites (for extension packages). To achieve the same with direct use of virtualenv and pip, you'd need to have an in-house PyPI repository accessible from production networks, and also release any extension packages as wheels pre-built for the target platform. Removing and updating an application is also much easier with Debian packages.

To use dh-virtualenv, you just have to extend your existing application project with a debian subdirectory – project meta-data like pip requirements and so on will be leveraged to build the final package, i.e. common tasks are delegated to the standard Python eco-system.

Note that just like with any other form of omnibus packaging, you take over the responsibility to release security updates of the contained dependencies in a timely manner.

How does it work?

dh-virtualenv is a debhelper plugin that extends the normal Debian tool chain for package building with the ability to create a Python virtualenv (an isolated Python environment), and then wrap that into the final Debian package.

Depending on the details of the application, you often also have to provide some kind of configuration of the software itself, and possibly some means to run it as a service. This can be done in several ways:

  • add a debian/«pkg».install descriptor to add configuration files to the Debian package.
  • provide a Puppet recipe or Ansible playbook that deploys the package and integrates it into the system.
  • embed (default) configuration into the application's Python package (via the include_package_data option of setuptools).

All those can be combined, e.g. provide defaults via Python package data, and then add external configuration that only provides values specific to the concrete host installation.

A real-world example is the devpi supervisor ERB template that serves both the purpose of passing configuration to the application process (via command line options), and also starting and controlling that process (i.e. handle demonization and automatic startup on boot).

Installing the build tools

Unsurprisingly, you need to install dh-virtualenv to use it. Since it is architecture independant, you can choose to use a recent release offered by Debian sid whatever your build platform is.

If this is your first time to build a Debian package, you also need to add the basic tools for that:

sudo apt-get install build-essential debhelper devscripts equivs

Finally, to take advantage of the available template for easily adding an inital debian directory, install the cookiecutter tool. Note that you can opt to build packages in a Docker container instead, with only Docker as a requirement on your build host.

Packaging an example project

To add the necessary debian directory with minimal effort, you can use the dh-virtualenv-mold cookiecutter. The following commands basically repeat what the integration test script of that project does, namely instantiate a Python project and then add debianization on top of it.

To provide common defaults to cookiecutter, it makes sense to have a ~/.cookiecutterrc file similar to the one I use.

Let's first create a sample project:

mkdir -p ~/tmp/dh-venv-blog
cd ~/tmp/dh-venv-blog
cookiecutter --no-input \
    "https://github.com/borntyping/cookiecutter-pypackage-minimal.git"
cd cookiecutter_pypackage_minimal/
python3 setup.py build

You can of course also use one of your own, then just check that out instead. Next, we add the debian directory:

cookiecutter --no-input \
    "https://github.com/Springerle/dh-virtualenv-mold.git"
dch -r "" # insert proper date & distro

Using --no-input causes the template's defaults to be accepted – it avoids answering all the template's prompts. After all, this is just a demo not requiring sensible inputs. Take the time to have a look at what's in the debian directory.

You're now able to build the package and if that succeeds, print the contained meta data:

dpkg-buildpackage -uc -us -b
dpkg-deb -I ../pyvenv-foobar_*.deb

The last command should show you something like this:

Package: pyvenv-foobar
Version: 0.1.0
Architecture: amd64
Maintainer: Jürgen Hermann <jh@web.de>
Installed-Size: 12877
Pre-Depends: dpkg (>= 1.16.1), python3, python3-venv
Section: contrib/python
Priority: extra
Homepage: https://github.com/jschmoe/foobar
Description: A Python package and its dependencies packaged up as DEB in an isolated virtualenv.

Finally, install the new package via dpkg -i, or upload it to a repository and use it from there with apt.

Real-world examples

These are examples of dh-virtualenv packaging for non-trivial applications:

The last two show how to integrate Python web applications into systemd, instead of using supervisor like the devpi example. The jupyterhub one also demonstrates the integration of a Python project with server-side Javascript (in a NodeJS environment).