Import Google Reader Starred Items into Delicious

July 4th, 2013

Now that Google Reader is gone, you might want to import your starred posts into Delicious. I’ve written a Python script to do just that.

Just head to Google Takeout and download your reader data, and use this script to do the import.

For more details, see the github repository: https://github.com/wil/reader-to-delicious.

Querying the DNS by hand

August 30th, 2012

Well, almost.

I wanted to get acquainted with using socat for UDP, so here’s something whipped up with the help of DNSPython:


python -c "import dns.message, dns.rdataclass, dns.rdatatype; "\
"print dns.message.make_query(dns.name.Name(['dready', 'org', '']), dns.rdatatype.A).to_wire()" \
| socat - UDP:8.8.4.4:53 \
| python -c "import sys, dns.message; "\
"print '\n'.join(str(m) for m in dns.message.from_wire(sys.stdin.read()).answer)"

Speed up Your SSH Connections for Free

August 8th, 2012

Adding the following directives to your .ssh/config should reduce your SSH connection establishment time significantly:

        ControlMaster auto
        ControlPersist yes
        ControlPath "/tmp/ssh-c-%r@%h:%p"

This works by having all SSH sessions sharing a single “control” connection.

The ControlMaster auto directive tells OpenSSH client to start a control master if one is not already running, otherwise it will reuse a previous one.

The ControlPersist yes directive tells the control master process to continue running in the background even though you’ve exited from the session, allowing subsequent sessions to reuse its connection.

The ControlPath is the control socket rendezvous point for masters and slaves. If you’re on a shared host, you might want to set it to a private location such as in your ~/.ssh/ directory instead of /tmp.

In my case, I stuck these into the wildcard Host section:

Host *
        Compression yes
        ServerAliveInterval 60
        ServerAliveCountMax 5
        ControlMaster auto
        ControlPersist yes
        ControlPath "/tmp/ssh-c-%r@%h:%p"

I’ve always known that OpenSSH had this functionality, but hadn’t bothered to try it out until now and it’s sooo very handy!

Dot Sydney GovCampNSW Session Debrief

December 15th, 2011

Recently I attended GovCampNSW, several days after Cloud Registry, in partnership with Sedari and CoCCA, submitted our response to the RFP for .sydney, .melbourne and .victoria gTLDs.

GovCampNSW turned out to be one of the most inspiring unconferences I’ve attended, to the credit of the (un)organisers, volunteers, and supporting partners. For the uninitiated:

GovCampNSW is an invitation to be part of an emerging conversation, that may inspire and shape new opportunities for innovation in government in Sydney, in NSW and beyond.

It is an opportunity to talk with a mix of people – from inside and outside government, from the worlds of technology and policy, of community and universities – to talk about shaping an agenda for innovation in NSW and to make a start on that agenda.

My agenda for attending was simple: to gauge the level of awareness of new gTLDs among fellow attendees and take home some pertinent lessons.

It was shaping up to be a great day. I arrived bright and early and bumped into Paul Wallbank, with whom I had an interesting conversation about new gTLDs over coffee! I also managed to speak to the Information Commissioner Deirdre O’Donnell briefly about Whois privacy and the .sydney project.

When the time came to nominate sessions on the white board, I scribbled in one of the slots:
.sydney manifesto ~ public-citizen collaboration

The question I had in mind was “how should the government of NSW involve the stakeholders in its design of the city’s digital identity, including the name space planning and governance aspects of the dotSydney gTLD?”

After the crowd had settled in our little cozy break out spot, I presented the story:

I was hopeful that the discussions would help craft some important points to inform the government about what matters to the stakeholders; hence, the “collaborative manifesto” in the slide title.

Following are my notes from the session. While they are far from being a “manifesto”, it highlights several important areas that may not be apparent to those of us who have been working in the industry for far too long.

  • On the Environment and Scoping Study:
  • Paul Wallbank: the $185,000 ICANN application fee smells like a scam
  • Paul Wallbank: How do you define Sydney, and the granularity of it? Who owns it? The NSW government? Or the people?
  • John Wells advised us to look at auDA’s Community Geographic Domain Name which has an interesting governance model
  • Several participants expressed the concern that: do businesses need to do defensive registration their product or name in every city that the business has a presence in, that also happen to have its own TLD?
  • There were also concerns about pricing of domains
  • Judy: .sydney could foster a sense of community
  • Some people agreed that there’s immense peer pressure — if major metropolitan cities like New York City, Paris and London have their own top-level domains, Sydney must have one too.

It it obvious that there is a lot of work to be done on outreach and public engagement in order to ensure that the .sydney, .melbourne and .victoria TLDs are well-integrated into the respective communities that they represent. We are keen to extend the efforts to the greater public sphere.

Please leave a comment if you have any ideas, suggestions or criticisms.

What’s with the J in Emails?

December 1st, 2011

This has bothered me ever since I saw it appearing in emails:

I’d love that J

WTF is that “J”? Does it stand for “joke”? “Jesus”?

After a while it became apparent that it’s somewhat equivalent to a smiley face, but I was still puzzled by it until I peeked under the hood today and found an email sent from Outlook with the following bit in the HTML part:

I'd love that <span style="font-family:Wingdings">J</span>

A-ha!

When rendered using the Windings font, indeed you get a smiley face:

I'd love that J

And the text/plain part of the email actually does contain the regular :), so you’d only see the “J” showing up if your device is trying to display the HTML version but it doesn’t have the Windings font available.

Find For-Else Statements in Python Code

October 24th, 2011

I came across Ned Batchelder’s post on the for/else construct, which explains its purpose in a different way from the standard Python docs. I’m fairly certain that more than once in my Python career I have been confused, or as one of the commenters put it: “Whenever I see it, I usually require an extra mental cycle to remember what it does”.

It got me paranoid and I wanted to quickly scan through my code to make sure that I haven’t got a lurking bug somewhere. This is the script that I came up with, which takes a .py file and look in its AST to identify such constructs and report them:

This is the first time I’m using the ast module, and it took a grand total of 30 minutes to find the module, read the documentation, experiment on the shell and come up with the script. I’d say that’s a pretty good testament to the power and intuitiveness of Python!

Hope you find it useful!

Hash Ring fork

July 10th, 2010

Amir Salihefendic‘s hash_ring module is awesome, but it doesn’t install cleanly with pip (dies with ez_setup import error) and spews some deprecation warning on Python 2.6+ (md5 module deprecated in favour of hashlib.)

In order to keep my deployment recipe sane, I’ve decided to fork the project on github.

Now, all you need is the following in requirements.txt:

-e git://github.com/wil/hash_ring.git#egg=hash-ring

Nginx + Django + FastCGI + SSL

July 10th, 2010

My favourite method for deploying Django sites in production is with Nginx at the frontend talking FastCGI to a dedicated Django process (either flup or supervisord) via a UNIX or TCP socket.

There are two recurring issues that I keep running into costing hours of fruitless Googling. Hopefully, this will help others:

  • SCRIPT_NAME is evil! At least it’s useless in Django, and if present as a FastCGI parameter, Django will trip that away from your request URI and you’re almost certainly going to get / no matter what URI you navigate to! So, save SCRIPT_NAME for your PHP stuff, and leave it out of the parts where you expect to be talking to Django.
    If you have include fastcgi_params in your nginx.conf, make sure it’s not hiding behind the fastcgi_params file.
  • If you want SSL to work, i.e. for Django to know that it is serving in https mode, and to build proper Location header for redirects, you need to include the following in the appropriate section of your nginx.conf:

    fastcgi_param HTTPS on;

    Obviously, if you’re running both plain and SSL version, you shouldn’t be sharing that section or Django will just thinking that it is in SSL mode even when it’s not.

httpd exited on signal 11 – solved

July 4th, 2010

tl;dr

It’s probably mod_php, and try recompiling PHP without zend_multibyte. If you have WITH_MULTIBYTE=true in /var/db/ports/php5/options, change it to WITHOUT_MULTIBYTE=true and rebuild the port.

Long Story

The Apache 2.2 processes on my FreeBSD box keep dying intermittently, with messages such as:


Jul 3 08:08:37 yama kernel: pid 13526 (httpd), uid 80: exited on signal 10
Jul 3 08:25:58 yama kernel: pid 33050 (httpd), uid 80: exited on signal 11

I’ve googled around and tried various extensions.ini reordering tricks without luck. Eventually, I figured out how to get Apache to dump core. First, you need to tell Apache where to dump core by adding the CoreDumpDirectory directive to your httpd.conf. Restart apache.

Then, check to make sure that the kern.coredump and kern.sugid_coredump sysctl variables are set to 1. Because Apache calls setuid and setgid to the configured user after listening on port 80 (privileged port), FreeBSD by default will refuse to dump core on segfaults. Setting kern.sugid_coredump will make it do so.


$ sysctl kern.coredump
kern.coredump: 1
$ sysctl kern.sugid_coredump
kern.sugid_coredump: 0
$ sudo sysctl kern.sugid_coredump=1
kern.sugid_coredump: 0 -> 1

Of course, you’d also need to make sure that your ulimit doesn’t prevent it from dumping core:


$ ulimit -c
unlimited

Great! Now wait for Apache to segfault (or, if you have a reproducible test case, hit it!)

You should now find httpd.core in the directory that you configured in httpd.conf (in my case, it is /tmp.)

Do the usual gdb stuff:


$ gdb /tmp/httpd.core /usr/local/bin/httpd

In my case, I found zend_multibyte_read_script near the top of the stack, so it’s pretty clear that PHP is to blame.
Did I mention that I abhor PHP? Someday I will get rid of it entirely!

This led me to more googling and found that I may have accidentally enabled Zend Multibyte support (well, its purpose is not very well documented and I just assumed that since I do lots of i18n work, it’ll come in handy. Okay so it’s my fault after all but it still shouldn’t crash!)

Changing WITH_MULTIBYTE=true to WITHOUT_MULTIBYTE=true in /var/db/ports/php5/options and rebuilding the port made the problem go away.

Hopefully, the above will help someone in a similar situation.

Trials and Tribulations of Global Communities

April 30th, 2010

I’m not about to write a lengthy essay on online cultures. Just an interesting observation on some events on the Tornado mailing list.

So, I lurk in that mailing list because Tornado is one of those awesome technologies in the Python world, and there are usually gems flying around.

As with most discussion groups I join, English is the norm. In fact, some have gone as far as to argue that English should be the lingua franca of hackers. To a great extent, I agree.

So, all is well until one day some Chinese messages started appearing on the list. Now, I read and write Chinese but I can’t making a living with Chinese geek-speak. Those messages didn’t bother me though I was thinking it must be annoying to non-Chinese speaking subscribers. And true enough, we got a message from one of the subscribers today:

我们看不懂你的中文信息,写英语,请。

Obviously the output of machine translation with an input to the effect of:

We cannot understand your Chinese messages, write in English, please.

I find it pretty amusing, and hope our fellow Chinese hackers don’t take offense.