Archive for the 'webeng' Category

Tornado with VirtualEnv and Pip Quickstart

Friday, October 9th, 2009

Friendfeed’s open source Tornado web server is great, and is incredibly easy to get up-and-running. Just install tornado, write your app and run it.

At some point, however, you’d want more structure in your project and manage dependencies to ease deployment. This is where virtualenv and pip shines. For a few more steps, you can bootstrap your project and have the warm fuzzy feeling that you can easily deploy the stuff when the code is ready.

Installing virtualenv and pip

If you haven’t set up virtualenv, do so (as root):

# easy_install virtualenv

Decide where you’d put your project directory. I’ll use /path/to/myapp for now. The next step is to create a virtualenv where all your Python packages are stored. I like to use the convention of a directory called root where all dependencies are installed. I’d generally also use it as the prefix for any cmmi packages that I’d like to contain within the project.

$ cd /path/to/myapp
$ virtualenv --no-site-packages root

Activate the environment that we just created:

$ . root/bin/activate
(root)[wil@wasabi /path/to/myapp]$ 

From now on, all packages installed with easy_install will be placed in this virtualenv.

Next, we will install pip into this virtualenv:

(root)[wil@wasabi /path/to/myapp]$ easy_install pip

Once pip is installed, as long as you’ve got your virtualenv activated, anything installed with pip will also go into the right place (without your having to remember to use the -E command line argument.)

Installing Tornado

Tornado (as of the current version) needs two mandatory dependencies, i.e. pycURL and simplejson. Make sure you have the right libcURL version installed on your system (using apt-get or other mechanism) and pick the compatible pyCURL version.

(root)[wil@wasabi /path/to/myapp]$ pip install pycurl==7.16.4
(root)[wil@wasabi /path/to/myapp]$ pip install simplejson

Now we’ll install tornado proper. I chose to go with the bleeding edge and ask pip to install from the git trunk.

(root)[wil@wasabi /path/to/myapp]$ pip install -e \

Should you not want that, you can tell pip to install from the tarball URL instead (at least until tornado gets added to PyPI.)

(root)[wil@wasabi /path/to/myapp]$ pip install \

Tornado is installed!

Every now and then, it’s a good idea to save your pip dependencies by running

(root)[wil@wasabi /path/to/myapp]$ pip freeze > pip-req.txt

Start your project

What I like about this is that the project directory has all the dependencies contained within a single directory (root). This is really just my convention; I’d create a src directory where my application code lives.

(root)[wil@wasabi /path/to/myapp]$ mkdir src
(root)[wil@wasabi /path/to/myapp]$ cd src
(root)[wil@wasabi /path/to/myapp/src]$ 

Let’s test drive Tornado:

(root)[wil@wasabi /path/to/myapp/src]$ cp ../root/src/tornado/demos/helloworld/ .
(root)[wil@wasabi /path/to/myapp/src]$ python

From browser, visit your host at port 8888 to verify.

That’s it!

Multipart Web Requests

Tuesday, October 2nd, 2007

Andy Skelton posted a proposal for a mechanism by which a web server could send related objects to a resource in response to a single request. It’s quite an interesting idea, although I’m not sure how much we could gain given that most clients today do HTTP pipelining.

It could reduce loads on the server, but I’m not sure either. Most static content is served very efficiently on the server using the sendfile (2) system call. By mixing static content with dynamic (the HTML which has to know what its embedded objects are), it may be difficult to implement efficiently.

That said, it could have some merits, and a more in-depth analysis would be needed and I’d be interested to see how it pans out.

Proposal: IDNA Browsers Advertising Capability in User-Agent header

Wednesday, July 12th, 2006

This is a proposal to the major browser producers supporting IDNA to advertise their IDN capabilities. One way of doing it is to include a token in the User-Agent HTTP request header. According to RFC2616, the User-Agent header is used for

“statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations.”

This will serve at least 2 purposes:

1. Allows web sites to encode anchor URLs in the native page encoding or HTML entities (&#xXXXX;) for clients that advertises IDNA support. For other clients, punycode can be used for baseline compatibility.

2. Provides IDN versioning information. There have been sprouts of discussions over the next version of Nameprep profile (or even a new IDNA, maybe even a new IDN solution). This would be a good preparation for things to come, regardless of which direction the future takes us.

Below are some examples of popular browser User-Agent strings I pulled from my logs (and my proposed addition):

Firefox: *
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20060508 Firefox/ IDN/RFC3490

Googlebot (which I believe is IDNA compliant):
Mozilla/5.0 (compatible; Googlebot/2.1;+ IDN/RFC3490

Internet Explorer 6:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) IDN/RFC3490

Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] IDN/RFC3490

Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/312.8 (KHTML, like Gecko) Safari/312.6 IDN/RFC3490

Other possibilities for advertising this capability exists, such as a custom header X-IDN-Version. The same information could possibly be exported to Javascript via the navigator.userAgent field, or some other custom Javascript API (this is more an example of what should NOT be done rather than a recommendation.)

* To achieve the result in Firefox, type this into the URL bar (in a single line):


This should take effect immediately. Go to and verify that “IDNA/RFC3490” is displayed in the “HTTP User Agent” box.