What To Expect

In this short series I will explain how to deploy applications with deb packages – but without the usual pain involved. My focus will be deb, because we are using Ubuntu LTS nearly everywhere at dajool. With a few sidenotes from me and a few google searches from you, you'll quickly adopt this to RPM (and perhaps soon FreeBSD). You will be free from any vendor lock-in and you will have the freedom to deploy anywhere: Your own servers, cloud and even your customers data center. There is no special sauce to apply.

I won't go too deep into setup details, but give you a detailed overview. With reasonable effort, you will be able to make your admin happy too!

Who Is Writing This?

My name is Jochen Breuer, I'm CEO and co-founder of dajool. We are specialized in mobile apps for iOS and Android and web services to power these mobile applications. Naturally this screams for a solid and scalable deployment.

The Problems

It seems to be common knowledge, that Python apps – and especially Django apps – need to be deployed using virtualenv and pip with requirements.txt. Most of the times the app itself will be cloned from a repository.

I was absolutely underwhelmed by this approach, since it is hard to maintain and hard to set up.

pip

pip is a great tool during development. You can install different versions of Python packages with minimal effort. But let's face it – for production environment pip sucks. Outages of the Cheese Shop used to be common and still happen from time to time.
Also pip is an additional packaging system you are using on your operating system, which already has one. You have to check updates twice and there could be mixed dependencies. This is especially annoying if you are using virtualenv's and also include packages from rubygems. A simple apt-get update && apt-get upgrade is not possible anymore.
Some packages installed by pip need to be compiled. I don't particularly like the idea to compile packages during deployment on my production systems.

virtualenv

From the description of virtualenv: virtualenv is a tool to create isolated Python environments. The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of libFoo, but another application requires version 2. How can you use both these applications?

There are use cases for virtualenv. A simple deployment of web apps shouldn't be one. It is easily possible to install multiple versions of python packages without virtualenv.
However, if this should really be a problem there is also a solution available, that is – in my opinion – better to maintain. Take a look at LXC. If you consider this overkill, you can still ship virtualenv within your deb package. But I wouldn't encourage that. Inception was a good movie, but shouldn't be replayed with your deployment.

Deployment from git, mercurial aso

While this is a perfectly well approach for testing or staging environments, in production this bears several downsides:

  • You need to respect version numbers, create branches or introduce some other conventions. Just pulling from tip (or head) will most likely break your app at some point in time, since developers not always tend to commit working, well tested code.
  • Repositories become very fat over time. Cloning a repository gives you all the history of a project, which has to be paid in megabytes. Means you are also downloading stuff you don't need to run this app. Shallow copy you say? If you want to stay in your happy repository world, do the full cloning.
  • Additional access restrictions have to be applied to use your repository for deployment and your repo has to be accessible from outside.
  • I'd also like to be in the meeting where you tell the sysadmin of your customer about all this. Because he won't be particularly thrilled to run your fabric scripts on his servers where you ssh into on of his machines moving files around with root permission.

The solution

Ingredients

My Workflow

  1. Update from the repo
    We are using mercurial, so I fetch the sources from our RhodeCode repository. Within the project is a special folder named "_bnd" (build and distribute). That folder hosts everything needed to build and deploy the project. Most of the times this would be an invoke make-file and a bunch of configs and config templates. I will explain this later.
  2. Build the deb file
    With a simple invoke build-deb --config ./config/xyz.ini I'm building the package. fpm is used to actually build the package. Most of the magic is happening here.
  3. Move package to deb repository
    Now the package would be moved to the reprepro (deb) repository. If you think this is overkill, you can just scp the package to your server and skip this step.
  4. Install
    apt-get install xyz-django_0.1234
    All the dependencies this package has defined will be installed automatically.
  5. Get some coffee and enjoy

Next week I'll get into the details. You'll see which security benefits you'll get, how dependencies are handled and how simple config management can be.