Preparing a Django development workflow between Ubuntu and OSX

By March 26, 2012 Tutorial No Comments

After a few years in the development world I think I finally find a workflow I am confortable with. I started with Flash and some PHP long time ago, but I must say that Python and HTML/Javascript are my favorite couple right now.

The problem

On the left side of the Ring… OSX!
I really like working with OSX, coding stuff it’s much easier in richer environments rather than in vi or emacs, although I have been learning some interesting shortcuts on vi lastly, It’s almost impossible to match the speed I can get on Sublime Text (2) or TextMate for example ( I do also understand that speeding up the workflow on vi will come with the experience and time which I not really aimed to spent :) )

On the right side of the Ring… Ubuntu!
On the other side most of the servers are running unix based operative systems, in this environments there’s no rich visual interface in order to take advantage of every cycle to the CPU to to backend stuff and the editors ( vi, nano, emacs ) are really minimal stuff ( yes, yes, you can add some cool plugins and stuff, but at the end, wanting or not, you are getting any simple editor you could find on a richer visual interface )

The solution

The solution is pretty simple :) This post is about how I managed to develop a Python Django + MySQL – NginX based Backend, coding from OSX Sublime Text 2 using a NFS connection to a VM Ubuntu Server [Preproduction] and using GIT to publish to a Linode Ubuntu Server [Production]

Requirements

To follow this tutorial you’ll need:

- A linode ( or equivalent ) production server
- Parallels Desktop ( or equivalent ) preproduction server

Conventions:

I will be using:

$Pre> command running on Preproduction Server
$Pro> command running on Production Server
$OSX> command running on OSX

{{var}} I will be enclosing variables to indicate this should be changed to your own settings

Github as a repository.
iTerm instead of the default Console provided by OSX

Mysql as database.

Summary:

Step 1 – Ubuntu Installation

Step 2 – Referring the servers

Step 3 – OSX to VM communication

Step 4 – Create folder structures

Step 5 – Installing MySql

Step 6 – Create a Mysql user for Django

Step 7 – Install Django and dependencies

Step 8 – Create Django base application and basic module

Step 9 – Install NginX

Step 10 – Restart-Django script

Step 11 – Install South

Step 12 – Enabling the Django Admin

Step 13 – Configure your static folder

Step 14 – Restart

Step 15 – Editing Ubuntu code from OSX via NFS

Step 16 – Mounting the drive on OSX

Step 17 – GIT – Using a repository for you code

Let’s start

Step 1 (Ubuntu Installation)

You need to have the same Operative System in both servers, in this case I will be using “Ubuntu 10.04 LTS”

To install Ubuntu in the production server you just have to install it via the web interface, here you can find a nice tutorial about how to deploy a Linux Distribution on a linode.

To install Ubuntu in the production server you have to download the same version of the operative system from ubuntu and install it to your virtual machine.

Note: Even if you download the same version of the Ubuntu Server like the one you installed into the linode the servers may be slightly different.

Step 2 (Referring the servers)

$Pre> ifconfig

The {{Pre_IP}} will be from now own the network IP of the virtual machine

$OSX> sudo vi /etc/hosts

The {{domain_name}} will be from now own the domain we will use to refer the preproduction server, for example project.prepro.is

Add {{Pre_IP}} {{domain_name}} as a new line in the editor and save the changes.

Step 3 (OSX to VM communication)

One of the differences I was talking about between distributions ( Linode and Ubuntu’s downloaded one ) is that the want you download may not have ssh installed by default. In this step we will download and install the package to allow ssh communication.

$Pre>sudo apt-get install openssh-server
$OSX>ssh {{pre_user_name}}@{{domain_name}}

The {{pre_user_name}}, {{pre_user_password}} will be from now own the name, password you gave when installing the Ubuntu Distribution to the VM.

Step 4 (Create folder structures – based on this )

$Pre>sudo mkdir /home/sites
$Pre>sudo mkdir /home/sites/public_html
$Pre>sudo mkdir /home/sites/public_html/{{domain_name}}

The {{domain_name}} will be from now own the name, password you gave when installing the Ubuntu Distribution to the VM.
The {{base_dir}} will be from now on ‘/home/sites/public_html/’

$Pre>sudo mkdir -p /home/sites/public_html/{{domain_name}}/{public,private,log,backup,static,conf}

Note: /home/sites/public_html is the default prefix I use for some of my sites, you can change this for whatever you want, such as /home/www/ , /var/www/ , etc…

Step 5 (Installing MySql)

$Pre>sudo apt-get install mysql-server mysql-client
$Pre>mysql -u root -p

In MySql

mysql> create database {{database_name}};
The {{database_name}} will be from now own the database name you want to work with for example 'project'.

Step 6 ( Create a Mysql user for Django – based on this )

mysql>CREATE USER ‘{{mysql_username}}’@'localhost’ IDENTIFIED BY ‘{{mysql_password}}’; mysql>GRANT ALL PRIVILEGES ON `{{database_name}}` . * TO ‘{{mysql_username}}’@'localhost’; mysql>FLUSH PRIVILEGES;

The {{mysql_username}} and {{mysql_password}} will be from now own the database user and password you want django to work with the database.

Step 7 (Install Django and dependencies)

$Pre>sudo mkdir /home/installers
$Pre>cd /home/installers
$Pre>sudo wget http://media.djangoproject.com/releases/1.3/Django-1.3.1.tar.gz
$Pre>sudo tar -zxvf Django-1.3.1.tar.gz
$Pre>sudo rm Django-1.3.1.tar.gz
$Pre>sudo python Django-1.3.1/setup.py install

Note: At the moment of writing this tutorial Django 1.4 has just been released, it may not be compatible to following this tutorial, but I think it won’t be any problem.

Installing the ‘Mysql – Python’ module

$Pre>sudo apt-get install python-mysqldb

Installing ‘Setup Tools’ and ‘Easy Install’

$Pre>sudo wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
$Pre>sudo sh setuptools-0.6c11-py2.6.egg

$Pre>sudo wget http://www.saddi.com/software/flup/dist/flup-1.0.2-py2.6.egg
$Pre>sudo easy_install flup-1.0.2-py2.6.egg

Note: If this code or urls does not work for you, this is because your python version is different form 2.6. In that case you can do 2 things:

- Install Python 2.6 :)
- Use the base of the URLs provided above and look for the file that matches your Python version

Step 8 (Create Django base application and basic module)

Django Application

$Pre>cd {{base_dir}}/{{domain_name}}/private
$Pre>sudo django-admin.py startproject {{app_name}}

The {{app_name}} will be from now own the name you chosed for you app, such as project.

Django Module
In order to start rendering some cool website we will create a basic module called ‘website’.

$Pre>cd {{app_name}}
$Pre>sudo python manage.py startapp website

Things to edit on the settings.py file
(you’ll find this file in the {{base_dir}}/{{domain_name}}/private/{{app_name}}/settings.py
for example /home/sites/public_html/project.prepro.is/private/project/settings.py
)

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '{{database_name}}', # Or path to database file if using sqlite3.
'USER': '{{mysql_username}}', # Not used with sqlite3.
'PASSWORD': '{{mysql_password}}', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
}

Step 9 (Install NginX – based on this)

$Pre>sudo aptitude install nginx
$Pre>sudo /etc/init.d/nginx start
$Pre>sudo vi {{base_dir}}/{{domain_name}}/conf/nginx.conf

nginx.conf

server {
listen 80;
server_name {{domain_name}};

access_log {{base_dir}}/{{domain_name}}/log/access.log;
error_log {{base_dir}}/{{domain_name}}/log/error.log;

location /static
{
root {{base_dir}}/{{domain_name}};
}

location /
{
# host and port to fastcgi server
fastcgi_pass 127.0.0.1:8081;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_pass_header Authorization;
fastcgi_intercept_errors off;
}
}

Linking to nginx

$Pre>sudo ln -s {{base_dir}}/{{domain_name}}/conf/nginx.conf /etc/nginx/sites-available/{{domain_name}}
$Pre>sudo ln -s /etc/nginx/sites-available/{{domain_name}} /etc/nginx/sites-enabled/{{domain_name}}
$Pre>sudo /etc/init.d/nginx restart

Note: The Nginx configuration file is stored inside the conf folder created at the beginning. I do it like this because this way I can keep the file in my GIT repository ( Subversion and GIT does not follow symlinks, so the only way to add them to the repository is by having them on the folder physically and linking them to the application – Nginx in this case )

Step 10 (Restart-Django script – based on this )

Every once in a while you’ll have to restart django to see that you changes on the code take effect. Every time you start django it creates a process that must be killed in order to be efficient.

$Pre>sudo vi /usr/local/bin/{{app_name}}-restart

Paste this on the file:

#!/bin/bash
PROJDIR="{{base_dir}}/{{your_domain}}/private/{{app_name}}"
PIDFILE="$PROJDIR/{{app_name}}.pid"
cd $PROJDIR
if [ -f $PIDFILE ]; then
kill `cat -- $PIDFILE`
rm -f -- $PIDFILE
fi
exec python ./manage.py runfcgi host=127.0.0.1 port=8081 pidfile=$PIDFILE --settings=settings

Give execution permissions

$Pre>sudo chmod +rx /usr/local/bin/{{app_name}}-restart
$Pre>sudo {{app_name}}-restart

Now if everything went well, you should see the result by going to this url (you should see an It Worked! message):

http://{{domain_name}}

Note: if you want to see which process is django running you can execute:

$Pre> ps -ef | grep "fcgi"

It will give you a result similar to this:

1000 3014 1 0 13:22 ? 00:00:00 python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
1000 3015 3014 0 13:22 ? 00:00:00 python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
1000 3016 3014 0 13:22 ? 00:00:00 python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
1000 3017 3014 0 13:22 ? 00:00:00 python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
1000 3018 3014 0 13:22 ? 00:00:00 python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
1000 3019 3014 0 13:22 ? 00:00:00 python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
1000 3105 1370 0 13:29 pts/0 00:00:00 grep --color=auto fcgi

To kill the django proces look for the 1 (parent) in the third column, take the second column number ( in this case 3014 ) and kill the process

$Pre> kill 3014

Step 11 (Install South)

Without using South, every time you edit any model you have to delete the table on the database, and this means losing all the data.

In a short and non-technical description, South is a module that let you update your models without all the headaches you usually find during this process.

$Pre> sudo easy_install South
$Pre> sudo vi {{base_dir}}/{{domain_name}}/private/{{app_name}}/settings.py

Add south to INSTALLED_APPS:

INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes'8,
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'south',
# Uncomment the next line to enable the admin:
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
)

An run this commands

$Pre> cd {{base_dir}}/{{domain_name}}/private/{{app_name}}
$Pre> sudo python manage.py syncdb

To apply South to your app modules you only have to run this command (in this case, I’ll use website module as an example):

$Pre>sudo python manage.py schemamigration website --initial

Now I can edit website model modules without being afraid of losing any data.

Step 12 (Enabling the django admin):

First we add the admin inside the installed apps.

$Pre> sudo vi {{base_dir}}/{{your_domain}}/private/{{app_name}}/settings.py

Uncomment the line in bold:

INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'south',
# Uncomment the next line to enable the admin:
'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
)

And now edit urls.py in order to enable /admin url

$Pre> sudo vi {{base_dir}}/{{your_domain}}/private/{{app_name}}/urls.py

Uncoment the lines in bold:

from django.contrib import admin admin.autodiscover()

urlpatterns = patterns('',
...
url(r'^admin/', include(admin.site.urls)),
)

You can check now the admin interface going here:

http://{{domain_name}}/admin/

Note: It is recommended to use a different name instead of admin. If you see an error going to the root preproduction page (http://{{domain_name}}/) this is because you didn’t create any view yet, or associated any view with the specified regular expression in urls.py

But wait, there is no styles! If you use chrome or firefox you can see that some resources failed to load (/static/admin/css/base.css, …) this is because you must put the admin files inside your static folder or define a path in your settings file.

If you have downloaded Django ( or checked out from subversion ) this files are included in the tar. You only have to move or link them to your static folder ( you may want to move them, as far as I know GIT and SVN doesn’t follow symlinks and styles wouldn’t be on the repository )

The files will be in the uncompressed folder or the folder where you did check out the repository, you can use this command to find files on unix

$Pre>sudo find / -name login.css

/home/installers/Django-1.3.1/django/contrib/admin/media/css/login.css

So now we want to move the media folder to our static and change the folder name to admin.

$Pre>sudo cp -Rf /home/installers/Django-1.3.1/django/contrib/admin/media {{base_dir}}/{{domain_name}}/static/admin

If you reload the page you will se still doesn’t work, this is because static files weren’t configured properly on settings yet.

Step 13 (Configure your static folder)

You only have to open your settings file

$Pre> sudo vi {{base_dir}}/{{domain_name}}/private/{{app_name}}/settings.py

And edit this line

STATIC_ROOT = '{{base_dir}}/{{domain_name}}/static/'

Note: This modifications are directly related to this line added to the nginx configuration:

...
location /static
{
root {{base_dir}}/{{domain_name}};
}
...

In this case the root directive says that every request starting with /static will be redirected to {{base_dir}}/{{domain_name}}/static ( another command such as alias would SUBSTITUTE the contents of the url )

Step 14 (Restart)

Finally we only have to restart django:

Remember we created an executable? We only have to execute to reload django

$Pre> {{app_name}}-restart

Step 15 (Editing Ubuntu code from OSX via NFS)

First of all, we need to install the necessary packages to establish an NFS connection

$Pre> sudo apt-get install portmap
Not sure portmap is really needed.
$Pre> sudo apt-get install nfs-kernel-server

Important Note: Since this is a configuration for a local development server ( Virtual Machine ) I won’t be putting too much attention into security stuff.

In order to be able to edit files via NFS your OSX userid and groupid must much the user created on the Virtual Machine aka Preprodction.

$Pre>id

uid={{unix_id}}({{unix_username}}) gid={{unix_gid}} ({{unix_username}})…
The {{unix_id}},{{unix_username}} and {{unix_gid}} will be from now own the preproduction user id, preproduction username and preproduction user groupid.

WHAT I COULDN’T MAKE WORK

This is what people tended to be on the past, but it is not working anymore, as far as I know map_static doesn’t work anymore in nfs v.4. I decided to post this code too in order to see if someone can make it work, or to avoid others to fall on the same trap.

$Pre>sudo mkdir /etc/nfs
$Pre>sudo vi /etc/nfs/mac.map ( where mac is the name of the accessing computer)

Add this lines:

# mapping for client: mac
# remote local
uid 501 1000
gid 20 1000

Finally open expots:

$Pre> sudo /etc/exports

And add this line

/home/sites/public_html/{{domain_name}} *(rw,sync,insecure,map_static=/etc/nfs/mac.map)

Restart

$Pre>sudo /etc/init.d/nfs-kernel-server restart
$Pre>sudo /etc/init.d/portmap restart

WHAT WORKS FOR ME

Open exports

$Pre>sudo vi /etc/exports

Add this line

{{base_dir}}/{{your_domain}} *(rw,async,no_subtree_check,insecure,all_squash,anonuid=1000,anongid={{unix_gid}})

If you created you files as a root, you may have to change the owner like this ( this step may not be required in your case )

$Pre>sudo chown -R {{unix_username}}:{{unix_username}} {{base_dir}}/{{domain_name}}
$Pre>sudo /etc/init.d/nfs-kernel-server restart

Step 16 (Mounting the drive on OSX)

There is several ways to do this

OSX > Disk Utility > File > NFS Mounts...

OSX > Finder > Go > Connect to Server....

I did it command-style :)

$OSX> sudo mount -t nfs {{domain_name}}:{{base_dir}}/{{app_name}} /path/to/local/folder (for example /Users/your_username/temp)

Now if you open finder you should be able to see a new mounted drive.

Note: If you want to unmounted you can do it by typing this on the command line:

$OSX>sudo umount -f /path/to/local/folder

Step 17 (GIT – Using a repository for you code)

Github has a GREAT documentation on how to create local repositories, etc…

$Pre>sudo apt-get install git-core
$Pre>sudo git config --global user.name "Your Name"
$Pre>sudo git config --global user.email "Your Email"
$Pre>sudo git init

$Pre>sudo git add private
$Pre>sudo git add static
$Pre>sudo git commit -m 'inital import.'
$Pre>sudo git remote add origin {{your_repo.git}}
$Pre>sudo git push -u origin master

Final Step:

The only step left is to deploy on production, this is basically repeat all the steps commented above ( changing $Pre> for $Pro> ) but instead of the Step 8 ( Creating the django application ) you would be cloning the repository developed in preproduction )

As a last modification you have to remove all configurations in the settings.py to prepare it for a production environment ( set DEBUG = False, etc… )

Extra Tip:
As an extra tip I found that using OSX iTerm I can configure my terminals to have different background colors and opacity. Sometimes when dealing with to many terminal windows you don’t know which is which anymore so now I use red color for production and green for preproduction.

It looks like this:


By the way, if you find that I made a mad mistake somewhere, I can improve something comment this post in order to be able to fix and improve my work

Final Note:
Watching all this variables enclosed in ‘{}’ is pretty annoying, isn’t it?

I made this for two reasons:

- In case you use this code you only have apply search and replace to match to your possible configuration
- Is the same code I use on the server, I created a python script that translates this into a valid commands and templates in order to deploy faster. Wait Wait Wait!!! there are frameworks such as Fabric to do that! I know, I know, but this was an excuse to learn python. Coming on the next post.

Hope it helps someone!!

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: