Celery, gunicorn, and supervisor
I use webfaction to host my web sites, and they are great!
Webfaction provides a “django” application type, which installs Apache + WSGI + Django for you, plus a Python instance. I really appreciate having had this setup, but now I understand what I’m doing well enough to throw it out and restart. Here’s what I have now:
- django-supervisor with this config file:
[unix_http_server] file=/tmp/supervisor.sock [program:celeryd] command= /manage.py celery worker -l info -B [program:gunicorn] command=gunicorn --bind 127.0.0.1: \ --workers 3 heitmgt.wsgi
- bunch o’ pip requirements:
django-supervisor django-celery gunicorn
Startup script that basically runs
python manage.py supervisor -d python manage.py supervisor stop all python manage.py supervisor start all
What the heck is going on here?
There is a pretty good reason I didn’t start out with this setup. I had no idea what the heck these things did or why you would need them. Why is everything so complicated??
Here’s the short of it:
- Gunicorn: A python program that’s happy to serve up wsgi applications.
- Celery + RabbitMQ: A queuing/processing system that lets you send tasks to be done later. For example, I just started using Django-celery’s database-stored “periodic tasks” to replace cron for some maintenance. (To handle periodic tasks you need a CeleryBeat worker; the “-B” flag to celery worker does this.)
- Django-supervisor: A wrapper around the program “supervisor” that lets you generate a config file from Django settings files. Supervisor seems to be a guardian for processes–it daemonizes foreground processes. There are also some tricks to this e.g. if DEBUG=True it is supposed to restart the processes.
The move to Gunicorn was mainly because I wanted to get rid of the apache instance and consolidate web management into supervisor. The Celery part, though, is a really big deal. It is extremely helpful/a paradigm shift to have both synchronous requests and asynchronous tasks in the same application.
All that said, I migrated from Webfaction’s provided django apps to a new “custom app listening on port” that’s Gunicorn–and it was much less painful than I thought it would be. Yay!