Archive for the 'Uncategorized' Category

Import Google Reader Starred Items into Delicious

Thursday, July 4th, 2013

Now that Google Reader is gone, you might want to import your starred posts into Delicious. I’ve written a Python script to do just that.

Just head to Google Takeout and download your reader data, and use this script to do the import.

For more details, see the github repository: https://github.com/wil/reader-to-delicious.

Querying the DNS by hand

Thursday, August 30th, 2012

Well, almost.

I wanted to get acquainted with using socat for UDP, so here’s something whipped up with the help of DNSPython:


python -c "import dns.message, dns.rdataclass, dns.rdatatype; "\
"print dns.message.make_query(dns.name.Name(['dready', 'org', '']), dns.rdatatype.A).to_wire()" \
| socat - UDP:8.8.4.4:53 \
| python -c "import sys, dns.message; "\
"print '\n'.join(str(m) for m in dns.message.from_wire(sys.stdin.read()).answer)"

What’s with the J in Emails?

Thursday, December 1st, 2011

This has bothered me ever since I saw it appearing in emails:

I’d love that J

WTF is that “J”? Does it stand for “joke”? “Jesus”?

After a while it became apparent that it’s somewhat equivalent to a smiley face, but I was still puzzled by it until I peeked under the hood today and found an email sent from Outlook with the following bit in the HTML part:

I'd love that <span style="font-family:Wingdings">J</span>

A-ha!

When rendered using the Windings font, indeed you get a smiley face:

I'd love that J

And the text/plain part of the email actually does contain the regular :), so you’d only see the “J” showing up if your device is trying to display the HTML version but it doesn’t have the Windings font available.

Find For-Else Statements in Python Code

Monday, October 24th, 2011

I came across Ned Batchelder’s post on the for/else construct, which explains its purpose in a different way from the standard Python docs. I’m fairly certain that more than once in my Python career I have been confused, or as one of the commenters put it: “Whenever I see it, I usually require an extra mental cycle to remember what it does”.

It got me paranoid and I wanted to quickly scan through my code to make sure that I haven’t got a lurking bug somewhere. This is the script that I came up with, which takes a .py file and look in its AST to identify such constructs and report them:

This is the first time I’m using the ast module, and it took a grand total of 30 minutes to find the module, read the documentation, experiment on the shell and come up with the script. I’d say that’s a pretty good testament to the power and intuitiveness of Python!

Hope you find it useful!

Hash Ring fork

Saturday, July 10th, 2010

Amir Salihefendic‘s hash_ring module is awesome, but it doesn’t install cleanly with pip (dies with ez_setup import error) and spews some deprecation warning on Python 2.6+ (md5 module deprecated in favour of hashlib.)

In order to keep my deployment recipe sane, I’ve decided to fork the project on github.

Now, all you need is the following in requirements.txt:

-e git://github.com/wil/hash_ring.git#egg=hash-ring

Nginx + Django + FastCGI + SSL

Saturday, July 10th, 2010

My favourite method for deploying Django sites in production is with Nginx at the frontend talking FastCGI to a dedicated Django process (either flup or supervisord) via a UNIX or TCP socket.

There are two recurring issues that I keep running into costing hours of fruitless Googling. Hopefully, this will help others:

  • SCRIPT_NAME is evil! At least it’s useless in Django, and if present as a FastCGI parameter, Django will trip that away from your request URI and you’re almost certainly going to get / no matter what URI you navigate to! So, save SCRIPT_NAME for your PHP stuff, and leave it out of the parts where you expect to be talking to Django.
    If you have include fastcgi_params in your nginx.conf, make sure it’s not hiding behind the fastcgi_params file.
  • If you want SSL to work, i.e. for Django to know that it is serving in https mode, and to build proper Location header for redirects, you need to include the following in the appropriate section of your nginx.conf:

    fastcgi_param HTTPS on;

    Obviously, if you’re running both plain and SSL version, you shouldn’t be sharing that section or Django will just thinking that it is in SSL mode even when it’s not.

Trials and Tribulations of Global Communities

Friday, April 30th, 2010

I’m not about to write a lengthy essay on online cultures. Just an interesting observation on some events on the Tornado mailing list.

So, I lurk in that mailing list because Tornado is one of those awesome technologies in the Python world, and there are usually gems flying around.

As with most discussion groups I join, English is the norm. In fact, some have gone as far as to argue that English should be the lingua franca of hackers. To a great extent, I agree.

So, all is well until one day some Chinese messages started appearing on the list. Now, I read and write Chinese but I can’t making a living with Chinese geek-speak. Those messages didn’t bother me though I was thinking it must be annoying to non-Chinese speaking subscribers. And true enough, we got a message from one of the subscribers today:

我们看不懂你的中文信息,写英语,请。

Obviously the output of machine translation with an input to the effect of:

We cannot understand your Chinese messages, write in English, please.

I find it pretty amusing, and hope our fellow Chinese hackers don’t take offense.

git conflict resolution

Wednesday, April 28th, 2010

Logging this mostly for myself.

When a conflict occurs during a pull operation from remote repository, we get this:

$ git pull kumo1 develop
From ssh://kumo0/home/wil/...
 * branch            develop    -> FETCH_HEAD
Auto-merged src/.../file.py
CONFLICT (content): Merge conflict in src/.../file.py
Automatic merge failed; fix conflicts and then commit the result.

What git did was to fetch objects from the remote repository, and tried to merge it in the branch that you specified. Sometimes the merge operation fails due to a conflict, and the conflicting edits are left in the file. It is then up to you to eyeball the file, straighten it and then “commit the result” (as the message said.)

However, if you tried to commit that file (after fixing the conflict), you’d get this:

$ git commit -m "my fixes" src/.../file.py
fatal: cannot do a partial commit during a merge.

What you’d want is to add the -i argument to the git commit command, which tells it to stage the additional file before committing.

logcheck update on FreeBSD

Saturday, January 23rd, 2010

Logging this quickly for posterity.

If like me, you just updated the logcheck port on your FreeBSD to version 1.2.69_1 and found that it’s broken, you might have run into the same permission problem I did.

Apparently, the port installed some files with too restrictive permissions and the error message from logcheck does not help. The files in question are some dynamically interpreted Perl plug-ins to detect rotated files in various schemes.

Just change the permissions as follows and it should all work again:


# chmod 644 /usr/local/share/logcheck/detectrotate/*.dtr

Tornado with VirtualEnv and Pip Quickstart

Friday, October 9th, 2009

Friendfeed’s open source Tornado web server is great, and is incredibly easy to get up-and-running. Just install tornado, write your app and run it.

At some point, however, you’d want more structure in your project and manage dependencies to ease deployment. This is where virtualenv and pip shines. For a few more steps, you can bootstrap your project and have the warm fuzzy feeling that you can easily deploy the stuff when the code is ready.

Installing virtualenv and pip

If you haven’t set up virtualenv, do so (as root):

# easy_install virtualenv

Decide where you’d put your project directory. I’ll use /path/to/myapp for now. The next step is to create a virtualenv where all your Python packages are stored. I like to use the convention of a directory called root where all dependencies are installed. I’d generally also use it as the prefix for any cmmi packages that I’d like to contain within the project.

$ cd /path/to/myapp
$ virtualenv --no-site-packages root

Activate the environment that we just created:

$ . root/bin/activate
(root)[wil@wasabi /path/to/myapp]$ 

From now on, all packages installed with easy_install will be placed in this virtualenv.

Next, we will install pip into this virtualenv:

(root)[wil@wasabi /path/to/myapp]$ easy_install pip

Once pip is installed, as long as you’ve got your virtualenv activated, anything installed with pip will also go into the right place (without your having to remember to use the -E command line argument.)

Installing Tornado

Tornado (as of the current version) needs two mandatory dependencies, i.e. pycURL and simplejson. Make sure you have the right libcURL version installed on your system (using apt-get or other mechanism) and pick the compatible pyCURL version.

(root)[wil@wasabi /path/to/myapp]$ pip install pycurl==7.16.4
(root)[wil@wasabi /path/to/myapp]$ pip install simplejson

Now we’ll install tornado proper. I chose to go with the bleeding edge and ask pip to install from the git trunk.

(root)[wil@wasabi /path/to/myapp]$ pip install -e \
  git+git://github.com/facebook/tornado.git#egg=tornado

Should you not want that, you can tell pip to install from the tarball URL instead (at least until tornado gets added to PyPI.)

(root)[wil@wasabi /path/to/myapp]$ pip install \
  http://www.tornadoweb.org/static/tornado-0.2.tar.gz

Tornado is installed!

Every now and then, it’s a good idea to save your pip dependencies by running

(root)[wil@wasabi /path/to/myapp]$ pip freeze > pip-req.txt

Start your project

What I like about this is that the project directory has all the dependencies contained within a single directory (root). This is really just my convention; I’d create a src directory where my application code lives.

(root)[wil@wasabi /path/to/myapp]$ mkdir src
(root)[wil@wasabi /path/to/myapp]$ cd src
(root)[wil@wasabi /path/to/myapp/src]$ 

Let’s test drive Tornado:

(root)[wil@wasabi /path/to/myapp/src]$ cp ../root/src/tornado/demos/helloworld/helloworld.py .
(root)[wil@wasabi /path/to/myapp/src]$ python helloworld.py

From browser, visit your host at port 8888 to verify.

That’s it!