Celery, gunicorn, and supervisor

I use webfaction to host my web sites, and they are great!

Webfaction provides a “django” application type, which installs Apache + WSGI + Django for you, plus a Python instance. I really appreciate having had this setup, but now I understand what I’m doing well enough to throw it out and restart. Here’s what I have now:

  1. django-supervisor with this config file:
[unix_http_server]
file=/tmp/supervisor.sock
[program:celeryd]
command= /manage.py celery worker -l info -B
[program:gunicorn]
command=gunicorn --bind 127.0.0.1: \
   --workers 3 heitmgt.wsgi
  1. bunch o’ pip requirements:
django-supervisor
django-celery
gunicorn

  1. RabbitMQ/Erlang. Yay?

  2. Startup script that basically runs

python manage.py supervisor -d
python manage.py supervisor stop all
python manage.py supervisor start all

What the heck is going on here?

There is a pretty good reason I didn’t start out with this setup. I had no idea what the heck these things did or why you would need them. Why is everything so complicated??

Here’s the short of it:

  • Gunicorn: A python program that’s happy to serve up wsgi applications.
  • Celery + RabbitMQ: A queuing/processing system that lets you send tasks to be done later. For example, I just started using Django-celery’s database-stored “periodic tasks” to replace cron for some maintenance. (To handle periodic tasks you need a CeleryBeat worker; the “-B” flag to celery worker does this.)
  • Django-supervisor: A wrapper around the program “supervisor” that lets you generate a config file from Django settings files. Supervisor seems to be a guardian for processes–it daemonizes foreground processes. There are also some tricks to this e.g. if DEBUG=True it is supposed to restart the processes.

The move to Gunicorn was mainly because I wanted to get rid of the apache instance and consolidate web management into supervisor. The Celery part, though, is a really big deal. It is extremely helpful/a paradigm shift to have both synchronous requests and asynchronous tasks in the same application.

All that said, I migrated from Webfaction’s provided django apps to a new “custom app listening on port” that’s Gunicorn–and it was much less painful than I thought it would be. Yay!

Updated: