Cup-Recipe For (Django) Python Deployment Part 2 - Detailed Overview

Detailed Overview

The last part outlined the bigger picture of the setup. Now we’ll get a bit more into the details.

reprepro

You’ll get several benefits from using reprepro:

• Security
  With a GPG-Key at hand you will be able to sign the packages. Your client systems (the webservers are meant) will check these signatures from the repository server during installation. Also, but this has nothing to do with reprepro directly, deb packages (can) contain checksums within the packages for each file included. If something is bogus with the package apt-get will notify you and stop the installation. This is very important if you are working in a more security sensitive environment.
  With the checksums from the deb package and debsums installed you also get a lightweight IDS. Running debsums in a very frequently started cron job with just your web applications (and dependencies) as a target will let you sleep better. If someone was able to change one of the files, you can let the sirens howl and take the application offline to check out what happend.
  Limiting your frequent debsums check to your web applications and dependencies simply has to do with resources. A full system check takes very long and can produce substantial IO on lower end systems. You should only do this once or twice a week.

• Dependencies
  Installing from a Debian repository with apt-get enables the system to automatically fetch dependencies either from the official repositories or your own reprepro repository. This depends on your priority or simply the version number.

Invoke Makefile And The Magic

The Magic Part 1 - invoke

There are a lot make-like tools out there, but invoke impressed me most with its unobtrusiveness. You can start small and add more functionality as needed. Since everything is plain Python, you can also get really freaky with extending your build task.

For me invoke is something like the basis for the builds. It’s easy to move files around with invoke. But the fun begins when you also generate your configs with Jinja. Everything encapsulated in reusable modules.

The Magic Part 2 – fpm

fpm stands for “Effing package managers” and is packaging for the rest of us (a.k.a packaging done right). With this handy ruby application you are able to build packages for multiple platforms: Everything that uses deb and rpm or Solaris and Mac OS X. You can build from a directory, tar file or pull down packages from RubyGems and the Cheese Shop.

A very minimal example for fpm would look like this:

fpm -s dir -t deb --deb-user root --deb-group root build_dir

A complete example for building a django package with some basic dependencies would look like this:

fpm -s dir \
    -t deb \
    -n my_cool_django_app \
    -v '1.0' \
    -a all \
    --license "All rights reserved." \
    -m "Jochen Breuer <breuer@dajool.com>" \
    --url "http://dajool.com" \
    --deb-user root \
    --deb-group root \
    --config-files /etc/nginx/sites-available/nginx_config \
    --config-files /etc/supervisor/conf.d/my_supervisor.conf \
    --after-install ./post_install.sh \
    --before-remove ./pre_uninstall.sh \
    -d "python2.7" \
    -d "python (>= 2.7.1-0ubuntu2)" \
    -d "python (<< 2.8)" \
    -d "python-django (>= 1.3.1)" \
    -d "python-django-south (>= 0.7.3)" \
    -d "gunicorn (>= 0.13.4-1)" \
    -d "python-django-alternate (>= 1.4.5)" \
    -d "python-grappelli-alternate (>= 2.4.5)" \
    -C ./build \
    etc srv

In this example we would deploy the Django appliation to /srv. The configs for nginx, supervisord aso would go to their designated folder in /etc and are marked as config file with the --config-files option - although this isn’t mandatory since every file in /etc is treated as config file.

The structure in my build folder looks like this – which is also the folder structure that will be generated on the server:

.
├── build
│   ├── etc
│   │   ├── nginx
│   │   │   └── sites-available
│   │   │       └── my.domain.com
│   │   └── supervisor
│   │       └── conf.d
│   │           └── my.domain.com.conf
│   └── srv
│       └── http
│           └── my.domain.com
│               ├── auth
│               ├── htdocs
│               └── lib
│                   └── my_django_application
│                       ├── …

Keep in mind, that none of the configs should be static. If you are moving this project to an other server things might change. This is something we’ll look into in “The Magic Part 3 - Jinaj2 Templates”.

The Magic Part 3 – Jinja2 Templates

As mention in “The Magic Part 1” I’m using Jinja2 to generate configs. So instead of changing the config on the server or moving static configs from my development system to production, I’m simply putting the generated config into the package and everything is in place after the installation.

As an example lets take a look at the supervisor config:

[program:{{ package_name }}] 
command=/usr/bin/python {{ srv_path }}{{ domain }}/lib/{{ django_prj_name }}/manage.py run_gunicorn localhost:{{ port }} 
directory={{ srv_path }}{{ domain }}/lib/{{ django_prj_name}}/
user=www-data
autostart=true
autorestart=true
redirect_stderr=True

Most of the paths, the port, the domain and project name are just variables. This way I can build multiple instances from this application or use different settings for different servers (if needed).
The values for these settings are simply stored in an ini file:

[general]
port = 9006
debug = True
domain = test.dajool.com
django_prj_name = test_app
main_app = test_main_app
admin_title = My Very Nice Test App
var_path = /var/http/
srv_path = /srv/http/
package_name = django-my-test-app

Invoke then gets told which config to use for the build procedure: invoke build_deb --config configs/my.domain.com.ini

Next week, in the last part, I’ll write about deployment and infrastructure management with staltstack as an example.