GovCampNSW turned out to be one of the most inspiring unconferences I’ve attended, to the credit of the (un)organisers, volunteers, and supporting partners. For the uninitiated:

GovCampNSW is an invitation to be part of an emerging conversation, that may inspire and shape new opportunities for innovation in government in Sydney, in NSW and beyond.

It is an opportunity to talk with a mix of people – from inside and outside government, from the worlds of technology and policy, of community and universities – to talk about shaping an agenda for innovation in NSW and to make a start on that agenda.

My agenda for attending was simple: to gauge the level of awareness of new gTLDs among fellow attendees and take home some pertinent lessons.

It was shaping up to be a great day. I arrived bright and early and bumped into Paul Wallbank, with whom I had an interesting conversation about new gTLDs over coffee! I also managed to speak to the Information Commissioner Deirdre O’Donnell briefly about Whois privacy and the .sydney project.

When the time came to nominate sessions on the white board, I scribbled in one of the slots:
.sydney manifesto ~ public-citizen collaboration

The question I had in mind was “how should the government of NSW involve the stakeholders in its design of the city’s digital identity, including the name space planning and governance aspects of the dotSydney gTLD?”

After the crowd had settled in our little cozy break out spot, I presented the story:

I was hopeful that the discussions would help craft some important points to inform the government about what matters to the stakeholders; hence, the “collaborative manifesto” in the slide title.

Following are my notes from the session. While they are far from being a “manifesto”, it highlights several important areas that may not be apparent to those of us who have been working in the industry for far too long.

• On the Environment and Scoping Study:
  • Dominic Campbell: almost as if they were trying to obscure the whole thing with a name like that
  • Emily Mitchell and others wondered about the results of the study
• Paul Wallbank: the $185,000 ICANN application fee smells like a scam
• Paul Wallbank: How do you define Sydney, and the granularity of it? Who owns it? The NSW government? Or the people?
• John Wells advised us to look at auDA’s Community Geographic Domain Name which has an interesting governance model
• Several participants expressed the concern that: do businesses need to do defensive registration their product or name in every city that the business has a presence in, that also happen to have its own TLD?
• There were also concerns about pricing of domains
• Judy: .sydney could foster a sense of community
• Some people agreed that there’s immense peer pressure — if major metropolitan cities like New York City, Paris and London have their own top-level domains, Sydney must have one too.

It it obvious that there is a lot of work to be done on outreach and public engagement in order to ensure that the .sydney, .melbourne and .victoria TLDs are well-integrated into the respective communities that they represent. We are keen to extend the efforts to the greater public sphere.

Please leave a comment if you have any ideas, suggestions or criticisms.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

I’d love that J

WTF is that “J”? Does it stand for “joke”? “Jesus”?

After a while it became apparent that it’s somewhat equivalent to a smiley face, but I was still puzzled by it until I peeked under the hood today and found an email sent from Outlook with the following bit in the HTML part:

I'd love that <span style="font-family:Wingdings">J</span>

A-ha!

When rendered using the Windings font, indeed you get a smiley face:

I'd love that J

And the text/plain part of the email actually does contain the regular :), so you’d only see the “J” showing up if your device is trying to display the HTML version but it doesn’t have the Windings font available.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

It got me paranoid and I wanted to quickly scan through my code to make sure that I haven’t got a lurking bug somewhere. This is the script that I came up with, which takes a .py file and look in its AST to identify such constructs and report them:

This is the first time I’m using the ast module, and it took a grand total of 30 minutes to find the module, read the documentation, experiment on the shell and come up with the script. I’d say that’s a pretty good testament to the power and intuitiveness of Python!

Hope you find it useful!

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In order to keep my deployment recipe sane, I’ve decided to fork the project on github.

Now, all you need is the following in requirements.txt:

-e git://github.com/wil/hash_ring.git#egg=hash-ring

Related posts:

1. Find For-Else Statements in Python Code I came across Ned Batchelder’s post on the for/else construct, which explains its purpose in a different way from the...

Related posts brought to you by Yet Another Related Posts Plugin.

There are two recurring issues that I keep running into costing hours of fruitless Googling. Hopefully, this will help others:

• SCRIPT_NAME is evil! At least it’s useless in Django, and if present as a FastCGI parameter, Django will trip that away from your request URI and you’re almost certainly going to get / no matter what URI you navigate to! So, save SCRIPT_NAME for your PHP stuff, and leave it out of the parts where you expect to be talking to Django.
  If you have include fastcgi_params in your nginx.conf, make sure it’s not hiding behind the fastcgi_params file.
• If you want SSL to work, i.e. for Django to know that it is serving in https mode, and to build proper Location header for redirects, you need to include the following in the appropriate section of your nginx.conf:
  
fastcgi_param HTTPS on;


  Obviously, if you’re running both plain and SSL version, you shouldn’t be sharing that section or Django will just thinking that it is in SSL mode even when it’s not.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

It’s probably mod_php, and try recompiling PHP without zend_multibyte. If you have WITH_MULTIBYTE=true in /var/db/ports/php5/options, change it to WITHOUT_MULTIBYTE=true and rebuild the port.

Long Story

The Apache 2.2 processes on my FreeBSD box keep dying intermittently, with messages such as:


Jul 3 08:08:37 yama kernel: pid 13526 (httpd), uid 80: exited on signal 10
Jul 3 08:25:58 yama kernel: pid 33050 (httpd), uid 80: exited on signal 11


I’ve googled around and tried various extensions.ini reordering tricks without luck. Eventually, I figured out how to get Apache to dump core. First, you need to tell Apache where to dump core by adding the CoreDumpDirectory directive to your httpd.conf. Restart apache.

Then, check to make sure that the kern.coredump and kern.sugid_coredump sysctl variables are set to 1. Because Apache calls setuid and setgid to the configured user after listening on port 80 (privileged port), FreeBSD by default will refuse to dump core on segfaults. Setting kern.sugid_coredump will make it do so.


$ sysctl kern.coredump
kern.coredump: 1
$ sysctl kern.sugid_coredump
kern.sugid_coredump: 0
$ sudo sysctl kern.sugid_coredump=1
kern.sugid_coredump: 0 -> 1


Of course, you’d also need to make sure that your ulimit doesn’t prevent it from dumping core:


$ ulimit -c
unlimited


Great! Now wait for Apache to segfault (or, if you have a reproducible test case, hit it!)

You should now find httpd.core in the directory that you configured in httpd.conf (in my case, it is /tmp.)

Do the usual gdb stuff:


$ gdb /tmp/httpd.core /usr/local/bin/httpd

In my case, I found zend_multibyte_read_script near the top of the stack, so it’s pretty clear that PHP is to blame.
Did I mention that I abhor PHP? Someday I will get rid of it entirely!

This led me to more googling and found that I may have accidentally enabled Zend Multibyte support (well, its purpose is not very well documented and I just assumed that since I do lots of i18n work, it’ll come in handy. Okay so it’s my fault after all but it still shouldn’t crash!)

Changing WITH_MULTIBYTE=true to WITHOUT_MULTIBYTE=true in /var/db/ports/php5/options and rebuilding the port made the problem go away.

Hopefully, the above will help someone in a similar situation.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

So, I lurk in that mailing list because Tornado is one of those awesome technologies in the Python world, and there are usually gems flying around.

As with most discussion groups I join, English is the norm. In fact, some have gone as far as to argue that English should be the lingua franca of hackers. To a great extent, I agree.

So, all is well until one day some Chinese messages started appearing on the list. Now, I read and write Chinese but I can’t making a living with Chinese geek-speak. Those messages didn’t bother me though I was thinking it must be annoying to non-Chinese speaking subscribers. And true enough, we got a message from one of the subscribers today:

我们看不懂你的中文信息，写英语，请。

Obviously the output of machine translation with an input to the effect of:

We cannot understand your Chinese messages, write in English, please.

I find it pretty amusing, and hope our fellow Chinese hackers don’t take offense.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

When a conflict occurs during a pull operation from remote repository, we get this:

$ git pull kumo1 develop
From ssh://kumo0/home/wil/...
 * branch            develop    -> FETCH_HEAD
Auto-merged src/.../file.py
CONFLICT (content): Merge conflict in src/.../file.py
Automatic merge failed; fix conflicts and then commit the result.

What git did was to fetch objects from the remote repository, and tried to merge it in the branch that you specified. Sometimes the merge operation fails due to a conflict, and the conflicting edits are left in the file. It is then up to you to eyeball the file, straighten it and then “commit the result” (as the message said.)

However, if you tried to commit that file (after fixing the conflict), you’d get this:

$ git commit -m "my fixes" src/.../file.py
fatal: cannot do a partial commit during a merge.

What you’d want is to add the -i argument to the git commit command, which tells it to stage the additional file before committing.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The first thing you should do is to see if HTTP Keep-Alive is supported by your server.

Second, and this may seem obvious, but I’ve seen too many developers forget to leave a hint to the browser to cache the results. As a result, the page becomes sluggish due to a feature that’s meant to be responsive.

See what happens behind the scene when you sign up for a new Twitter account. Suppose you try to register the username “wil”, but it’s taken. For each character you type, the browser makes HTTP requests to check that the username in the input field to see if it’s available.

So that’s one for “w”, “wi”, “wil”. Then you find that all 3 are taken. Ok, perhaps time to add a numeric suffix? “wil1″ – nope, delete the “1″, and we’re back to “wil” again. Guess what? Another HTTP request is sent to Twitter for the same string “wil”!

Had Twitter set an “Expires” header to usernames that are taken, the browser wouldn’t have had to make that round trip!

Below are the headers sent by Twitter for the https://twitter.com/users/username_available URI (courtesy of Hurl):

In the case of signup forms, usernames are rather involatile pieces of data, so it’s a prime optimization target. As your namespace becomes more scarce, you’ll tend to have people trying more strange combinations, increasing the number of requests to your servers.

In my case, I’ve applied it to our TLD management platform. Domain names that are registered gets a 10 minute cache timeout value (which is heaps short but good enough to ensure a snappy UI operations). However, with domain names, it’s a lot more volatile but we’re not guaranteeing success at the point of registering a name so anywhere between 1-5 minutes is usually sufficient.

A simple view snippet in Django does the trick and goes a long way to making your users happy.

/ Blogging from a HSDPA connection

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().

Okay, this means that my program forked, and the child did not immediately exec(). Instead, it went on to call some Core Foundation function. Naturally, I’d want to find out what is it that it’s doing! Time to fire up our trusty gdb.

macmac:~ $ sudo gdb -p 52772
GNU gdb 6.3.50-20050815 (Apple version gdb-960) (Sun May 18 18:38:33 UTC 2008)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin".
Attaching to process 52772.
Reading symbols for shared libraries . done
Reading symbols for shared libraries .................................... done
0x9354a6fa in select$DARWIN_EXTSN ()
(gdb) b __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__
Breakpoint 1 at 0x938df314
(gdb) cont
Continuing.
Breakpoint 1, 0x938df314 in __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ ()
(gdb) bt
#0  0x938df314 in __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ ()
#1  0x938d97bd in CFRunLoopGetCurrent ()
#2  0x95bf09fa in +[NSThread currentThread] ()
#3  0x95befe39 in _NSInitializePlatform ()
#4  0x918568b8 in _class_initialize ()
#5  0x9185678c in _class_initialize ()
#6  0x91855239 in _class_lookupMethodAndLoadCache ()
#7  0x918656d6 in objc_msgSend ()
#8  0x9185dbdf in call_load_methods ()
#9  0x918570d3 in load_images ()
#10 0x8fe02e38 in __dyld__ZN4dyld12notifySingleE17dyld_image_statesPK11mach_headerPKcl ()
#11 0x8fe0e7cf in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#12 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#13 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#14 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#15 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#16 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#17 0x8fe0e8c9 in __dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextE ()
#18 0x8fe02202 in __dyld__ZN4dyld15runInitializersEP11ImageLoader ()
#19 0x8fe0bbdd in __dyld_dlopen ()
#20 0x935042c2 in dlopen ()
#21 0x001b09ac in _PyImport_GetDynLoadFunc ()
#22 0x001a35a4 in _PyImport_LoadDynamicModule ()
.. (snipped) ..

This tells us that Python was trying to call the dlopen() function to load a dynamic library. I know that because the stacktrace went from a bunch of CPython functions to dlopen() (at stack #20) and eventually led up to the function that we were told to break on. Now it’s time to find out which library it is trying to load.

(gdb) frame 20
#20 0x935042c2 in dlopen ()
(gdb) info frame
Stack level 20, frame at 0xb007d6f0:
 eip = 0x935042c2 in dlopen; saved eip 0x1b09ac
 called by frame at 0xb007d9b0, caller of frame at 0xb007d6d0
 Arglist at 0xb007d6e8, args:
 Locals at 0xb007d6e8, Previous frame's sp is 0xb007d6f0
 Saved registers:
  ebx at 0xb007d6e4, ebp at 0xb007d6e8, eip at 0xb007d6ec

If you had the source code for dlopen(), gdb would’ve helpfully printed the arguments for you, but alas we don’t have the source code!

Thankfully, we know what the function looks like from the dlopen man page:

     void *dlopen(const char* path, int mode);

So we know that the library we’re looking for is in the first argument, which is a pointer to a C string. And we will find it.

Some googling later, I found the layout for darwin stackframes:

Sweet! According to this, the first function argument is in EBP+8:

(gdb) info registers ebp
ebp            0xb007d6e8	0xb007d6e8

Aha! But you didn’t really need to do that; the info frame command output already told us that locals are at 0xb007d6e8.

0xb007d6e8 + 8 = 0xb007d6f0, so we shall get the address stored at this address:

(gdb) x/a 0xb007d6f0
0xb007d6f0:	0xb007da67

Now print the string at the target address:

(gdb) x/s 0xb007da67
0xb007da67:	 "/Users/wil/src/proj/root/lib/python2.5/lib-dynload/icglue.so"

Et voila! This happens to be the glue code for Python to access the Mac OS X Internet Config settings. In some previous foray, I learned (to my surprise) that Python’s urllib (on Darwin) actually uses IC to find out your proxy settings and use it! For some reason, I never expected it to do so but I guess it makes sense for a seamless user experience.

Did I find out why it’s giving me that fork/exec message? No, not really but it doesn’t matter much.
(Update: I do know the reason for this; it’s because some code started fetching from HTTP after fork, but that’s the expected behavior, and I don’t wish to reexec. Program continues to work, so I just have to ignore it in my development environment. This will be deployed to a Linux or FreeBSD which is why I said it doesn’t matter much.)

Now, I’m convinced that there is a better way to do it so if you have any idea, please leave a comment.

p.s. This can probably be easily achieved with DTrace, but it makes my head spin.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.