Django environment setup

I have been doing some programming in “Django,” which is awesome. Django is a Python-based web programming framework.

Over the course of the last couple of months, I’ve worked on my Django development infrastructure. I’m kind of a stickler for doing things the “right” way, and this is what I’ve decided is the right way so far. Here’s how my Django infrastructure is set up:

Directory layout

A “bashmark” and an Emacs bookmark point to the root of my Django tree, which is checked out from git (and which I maintain with git-flow and small time-duration feature branches).

  • Local (.gitignore’d) files:
    • .dir-locals.el: Sets pony-settings to use the virtualenv version of python, and virtualenv-workon and virtualenv-default-directory for Emacs virtualenv.
    • .pyflymakerc: Sets VIRTUALENV, PYLINT, TEST_RUNNER_COMMAND, IGNORE_CODES_PYLNT (‘E1101’), and TEST_RUNNER_FLAGS
    • .venv: Name of virtualenv
  • Makefile (see below), manage.py
  • app/ — where Django apps go, e.g. “app/hello/”. Each application may then have the following files (so far):
    • admin.py
    • ajax.py (used by dajaxice)
    • exceptions.py
    • forms.py
    • migrations/
    • models.py
    • templates/
    • tests.py
    • urls.py
    • views.py
  • bin/ — any command-line files that don’t cleanly fit into another directory
    • get_config.py — creepy program that spits out the identified Django config value, used to get Django database username et al for
    • pg_dump.sh — Shell script that creates a Postrgres dump file & can be added to cron w/o any output on success.
    • post.sh, post_local.sh — Shell scripts that are run after “make pulldown”. In prodcution, post_local.sh restarts the web server, uses curl to pull a test page, and reminds me to do other routine tasks associated with builds.
  • docs/ — “.text” files formatted via Markdown. A Makefile can compile these.
  • «project»/ — the Django project config directory
    • client_secrets.json, urls.py, wsgi.py
    • settings/ — Using this super-awesome __init__.py to let you have multiple settings files including a “common.py” plus an environment-specific one plus one called “local_settings.py” that doesn’t get checked in that has all your passwords in it
  • lib/ — (very simple) Pythony support libraries I’ve written. Currently includes:
    • json_shortcuts.py: to_json() function that correctly serializes datetime.datetime and datetime.date objects
    • middleware.py for custom middleware
    • nose_tweaks.py to make Nose shut up about South logging (plugin called “SilenceSouth“)
    • re_shortcuts.py for super common regexes
    • tests.py
    • time_shortcuts.py
    • unicode_csv.py for simple csv module wrapper that handles Unicode encodings
  • patch/ — diffs that I want to patch from that involve external stuff. Makefile can update these diffs. This is a crazy thing in and of itself and am not sure whether it would be helpful for others. Let’s just say that this helps my Django site look the same as my WordPress site.
  • requirements/ (see below for module list)
    • common.txt
    • dev.txt
    • test.txt
    • test-libs.txt (required by dev.txt and test.txt BUT NOT prod.txt)
    • prod.txt
    • unseen.py (hack to tell me what doesn’t show up in any requirements file that does show up when you run “pip freeze”)
  • static/
    • css/
    • js/
    • images/
  • templates/
    • base.html, {404,500}.html
  • uploads/
  • var/ — in .gitignore but this is where I keep database backups.

Makefiles

I tried using Fabric, and more power to you if you use it, but I didn’t need it. From my limited understanding, Fabric is good for helping you do stuff across systems. However, for someone with a background in cfengine Fabric is very frustrating to me. I wanted a way to automate parts of my build process but I didn’t need to do cross-machine deployments. So, I went back to Make. I don’t know why but I love Make.

Make targets:

  • pulldown — runs the “clean” target, then
git pull
 git fetch --tags
 pip install -r requirements/${DJANGO_ENV}.txt
 python manage.py migrate
 # Needed for permissions to be created/updated:
 python manage.py syncdb --all
 python manage.py collectstatic --noinput
 cd bin && ./post.sh
 # (note: post.sh can then reboot the web server)
  • pushup — runs “git push –all”, “git push origin –tags”
  • clean — deletes .pyc files and runs “clean” for other Makefiles
  • virtualenv — builds the virtualenv you need for the project. (I just built this target and it’s pretty sweet.) This is totally a hack and not very robust. Just so you can get a feel for it (but FYI you can’t copy & paste this cleanly):
test ! -d ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 bash -i -c 'lsvirtualenv > /dev/null'
 curl -O http://pypi.python.org/packages/source/v/\
  virtualenv/virtualenv-${VIRTUALENV_VERSION}.tar.gz
 tar xvf virtualenv-${VIRTUALENV_VERSION}.tar.gz
 python virtualenv-${VIRTUALENV_VERSION}/virtualenv.py \
  --no-site-packages --setuptools \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 rm virtualenv-${VIRTUALENV_VERSION}.tar.gz
 rm -rf virtualenv-${VIRTUALENV_VERSION}
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && add2virtualenv .'
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && pip install -r requirements/${DJANGO_ENV}.txt'
 echo "export DJANGO_ENV='${DJANGO_ENV}'" >> \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "export DJANGO_SETTINGS_MODULE='${DJANGO_PROJECT}.settings'" \
  >> ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "${VIRTUALENV_ALIAS}" > .venv

Modules/requirements

  • common.txt (for all environments) includes:
    • Django
    • South (totally the best thing since sliced bread and extremely the type of thing you don’t know you need until 3 months after you need it)
    • django-extensions (even better than South. runserver_plus, shell_plus, and graph_models are my faves)
    • pytz (see my “Time” post)
    • django-dajaxice (still not sure how I feel about this but it has been pretty useful so far)
    • Scrapy (is awesome)
    • coverage (really cool but I am not using it enough yet–along with django-nose, this will tell you thoroughly you’re testing)
  • dev.txt:
    • Werkzeug (O. M. G.)
    • pep8
    • pyflakes
    • pygraphviz
    • pylint
    • jedi (this helps with Emacs autocompleter stuff)
  • test-libs.txt (non-production envs):
    • django-nose
    • factory-boy (really good for testing)

Emacs

Using Aquamacs. ~/.emacs.d is checked into its own git repository currently shared via dropbox.

  • pony-mode
  • virtualenv
  • yasnippet
  • markdown-mode
  • jedi
  • python (vanilla 23.3 ‘python python-mode)
  • json-mode
  • flymake
  • django-nxhtml-mumamo-mode (worth every letter)

Things I Now Try To Do

  • Use factories rather than fixtures for tests (I thought factories were dumb until I started changing models & had to mess with raw .json — this kills the programmer)

Updated: