Archive for the 'hacks' Category

Debugging without source on Mac

Thursday, March 11th, 2010

This is probably newbie stuff for hardcore C programmers, but I’m logging here for posterity and for my own benefit. I don’t pretend to be one, but recently found myself needing to find out the cause of a curious message while running Python on my Mac OS X 10.5.8:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().

Okay, this means that my program forked, and the child did not immediately exec(). Instead, it went on to call some Core Foundation function. Naturally, I’d want to find out what is it that it’s doing! Time to fire up our trusty gdb.

macmac:~ $ sudo gdb -p 52772
GNU gdb 6.3.50-20050815 (Apple version gdb-960) (Sun May 18 18:38:33 UTC 2008)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin".
Attaching to process 52772.
Reading symbols for shared libraries . done
Reading symbols for shared libraries .................................... done
0x9354a6fa in select$DARWIN_EXTSN ()
Breakpoint 1 at 0x938df314
(gdb) cont
(gdb) bt
#1  0x938d97bd in CFRunLoopGetCurrent ()
#2  0x95bf09fa in +[NSThread currentThread] ()
#3  0x95befe39 in _NSInitializePlatform ()
#4  0x918568b8 in _class_initialize ()
#5  0x9185678c in _class_initialize ()
#6  0x91855239 in _class_lookupMethodAndLoadCache ()
#7  0x918656d6 in objc_msgSend ()
#8  0x9185dbdf in call_load_methods ()
#9  0x918570d3 in load_images ()
#10 0x8fe02e38 in __dyld__ZN4dyld12notifySingleE17dyld_image_statesPK11mach_headerPKcl ()
#11 0x8fe0e7cf in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#12 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#13 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#14 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#15 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#16 0x8fe0e775 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#17 0x8fe0e8c9 in __dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextE ()
#18 0x8fe02202 in __dyld__ZN4dyld15runInitializersEP11ImageLoader ()
#19 0x8fe0bbdd in __dyld_dlopen ()
#20 0x935042c2 in dlopen ()
#21 0x001b09ac in _PyImport_GetDynLoadFunc ()
#22 0x001a35a4 in _PyImport_LoadDynamicModule ()
.. (snipped) ..

This tells us that Python was trying to call the dlopen() function to load a dynamic library. I know that because the stacktrace went from a bunch of CPython functions to dlopen() (at stack #20) and eventually led up to the function that we were told to break on. Now it’s time to find out which library it is trying to load.

(gdb) frame 20
#20 0x935042c2 in dlopen ()
(gdb) info frame
Stack level 20, frame at 0xb007d6f0:
 eip = 0x935042c2 in dlopen; saved eip 0x1b09ac
 called by frame at 0xb007d9b0, caller of frame at 0xb007d6d0
 Arglist at 0xb007d6e8, args: 
 Locals at 0xb007d6e8, Previous frame's sp is 0xb007d6f0
 Saved registers:
  ebx at 0xb007d6e4, ebp at 0xb007d6e8, eip at 0xb007d6ec

If you had the source code for dlopen(), gdb would’ve helpfully printed the arguments for you, but alas we don’t have the source code!

Thankfully, we know what the function looks like from the dlopen man page:

     void *dlopen(const char* path, int mode);

So we know that the library we’re looking for is in the first argument, which is a pointer to a C string. And we will find it.

Some googling later, I found the layout for darwin stackframes:

Mac OS X stack frame layout

Sweet! According to this, the first function argument is in EBP+8:

(gdb) info registers ebp
ebp            0xb007d6e8	0xb007d6e8

Aha! But you didn’t really need to do that; the info frame command output already told us that locals are at 0xb007d6e8.

0xb007d6e8 + 8 = 0xb007d6f0, so we shall get the address stored at this address:

(gdb) x/a 0xb007d6f0
0xb007d6f0:	0xb007da67

Now print the string at the target address:

(gdb) x/s 0xb007da67
0xb007da67:	 "/Users/wil/src/proj/root/lib/python2.5/lib-dynload/"

Et voila! This happens to be the glue code for Python to access the Mac OS X Internet Config settings. In some previous foray, I learned (to my surprise) that Python’s urllib (on Darwin) actually uses IC to find out your proxy settings and use it! For some reason, I never expected it to do so but I guess it makes sense for a seamless user experience.

Did I find out why it’s giving me that fork/exec message? No, not really but it doesn’t matter much.
(Update: I do know the reason for this; it’s because some code started fetching from HTTP after fork, but that’s the expected behavior, and I don’t wish to reexec. Program continues to work, so I just have to ignore it in my development environment. This will be deployed to a Linux or FreeBSD which is why I said it doesn’t matter much.)

Now, I’m convinced that there is a better way to do it so if you have any idea, please leave a comment.

p.s. This can probably be easily achieved with DTrace, but it makes my head spin.

Java SSL Implementation does not detect disconnected socket

Saturday, November 22nd, 2008

To be exact, Sun’s JSSE SSL/TLS implementation ignores the EOFException that is generated when its underlying socket was closed properly (that is if the server initiates a proper shutdown) and you try to write to it.

This is what you get when debug is turned on (using the switch):

Thread, WRITE: TLSv1 Application Data, length = 499
Thread, received EOFException: ignoredThread, called closeInternal(false)
Thread, SEND TLSv1 ALERT:  warning, description = close_notify
Thread, WRITE: TLSv1 Alert, length = 18

In order to detect the closed connection, you actually need to perform a read on the SSLSocket’s input stream:

Socket sock = factory.createSocket(host, port);
// factory is an SSLSocketFactory() instance
// ...
// ...
// At some point, peer has already shutdown the connection
// ...
// write a byte
OutputStream out = socket.getOutputStream();
// no exception here
// still no exception
// read something from it
InputStream in = socket.getInputStream();
int n =;
// n becomes -1 here

One would expect the write to throw an exception, since the underlying socket was closed. It seems that the Apache HttpClient folks did discover this:

Due to what appears to be a bug in Sun’s older (below 1.4) implementation of Java Virtual Machines or JSSE there’s no reliable way of telling if an SSL connection is ‘stale’ or not. For example, the HTTP 1.1 specification permits HTTP servers in ‘keep-alive’ mode to drop the connection to the client after a given period inactivity without having to notify the client, effectively rendering such connection unusable or ‘stale’. For the HTTP agent written in Java there’s no reliable way to test if a connection is ‘stale’ other than attempting to perform a read on it. However, a read operation on an idle SSL connection on Sun JVM older than 1.4 returns ‘end of stream’ instead of an expected read timeout. That effectively makes the connection appear ‘stale’ to HttpClient, which leaves it with no other way but to drop the connection and to open a new one, thus defeating HTTP 1.1 keep-alive mechanism and resulting in significant performance degradation (SSL authentication is a highly time consuming operation). The problem appears to have been fixed in Sun’s Java 1.4 SSL implementation. Sockets which are not using HTTPS are unaffected on any JVM.

However, in many network protocols, there is a fixed command-response sequence so reading from the socket before writing does pose a problem. What the Apache HttpClient folks did was to wrap the socket InputStream in a BufferedInputStream, then mark the stream position, try to read a single byte from it with a timeout of 1ms. If read() returns -1 the socket is considered stale, and it resets the stream to the marked position. This is less than ideal, but probably works.

As far as I know, this problem exists on JDK versions 1.4 and 1.5.

There you go, I just wanted to throw it out there for anyone who happens to stumble on the same problem. Personally, I’ve spent a good afternoon chasing down this problem, from googling to hunting for the JSSE source code (which I gave up finding after discovering that the JDK 1.6 source doesn’t include it.)

Memcached on Solaris

Thursday, February 7th, 2008

If you observe memcached exhibiting strange behavior while running under Solaris, you should try upgrading to the latest version of libevent.

I experienced a problem while testing my application on the excellent Joyent Facebook Accelerator, which runs Solaris Nevada snv_67 X86, has memcached 1.2.2 with libevent 1.3b2 installed by default.

My memcached usage is pretty low and I refresh the cache often to keep it from going stale, but somehow I still get lots of cache misses. By elimination, I ruled out the possibility of faults on the python memcache module, or memcached version (tried the latest 1.2.4 compiled against the libevent-1.3b2 and it still had the same problem.) When I connected my app to the memcached instance running on my FreeBSD box, though, the problem doesn’t exist.

Eventually, it turns out that memcached disconnects the client and all I got from the python memcache module was:

  File "/opt/local/lib/python2.5/site-packages/", line 846, in recv
    'read returned 0 length bytes' % ( len(buf), foo ))

Well, that’s another bug. The above foo should really be rlen, but fixing that only proved that memcached always disconnects the client after sending 66887 bytes in response to a get.

After some poking around (by inserting prints and running memcached in foreground mode), it was apparent to me that memcached was getting an error from libevent, so I upgraded libevent and problem was solved.

Hope this helps anyone who may run into the same problem (as I couldn’t find any clue in the googs.)

GearSaver Reloaded

Monday, August 6th, 2007

The GearSaver Bookmarklet was not that useful since there was no easy way to use the saved data. Since this is an itch that I’d really like to be able to scratch, I reached further and improved on it to the point that I can now write this post in a GearSaver-enabled textarea. Added features are:

  • uses jQuery – I have no intention of working around browser DOM quirks more than I already have to
  • displays previously saved versions
  • ability to preview a version and apply to current textarea content
  • ability to delete versions

Eventually, I hope to be able to provide the script as:

  • ad-hoc inclusion into any page – this already works in the current version
  • as a bookmarklet – so that you can enable any page
  • as a greasemonkey script – so that you can automatically have it turned on on the sites that you want it

So, without further ado, you can try it out…

1. Here’s a textarea

2. Make sure you have Google Gears installed.

3. Click here to enable GearSaver.

4. Gears will ask you whether to allow to write data

Please report any bugs / suggestions.

Update 20070812: Updated to use new and improved version 0.4

Monday, January 29th, 2007

From the innovative mind of Simon Willison comes IDProxy:, launched today, is my attempt at speeding up the process. It uses Yahoo!’s Browser-Based Authentication API to allow you to sign in with a Yahoo! account, then lets you create one or more OpenIDs (of the form to use with sites that support the OpenID standard.

Basically, it’s an OP that authenticates against Yahoo – call it an intermediate IdP, Proxy OpenID Provider, OpenID-Yahoo bridge, whatever.. it rocks!

Skype Reverse-Engineered

Friday, July 14th, 2006

VoIPWiki reports that the Skype protocol was successfully cracked by a company in China, who promised to release a demo version by end of August. This is amazing – not the news itself, but the extent to which they have gone. Several reverse-engineering reports have been issued but no one has built a working prototype until now.

This goes to show that: obfuscation is obfuscation – it will not last, not if you’re as popular as Skype.

What does this mean?

Many are predicting / recommending that the eBay company documents its protocol. I can safely say that they will not. This is only news to techies; it may make its way to NYT but people will forget. In fact, as long as it works, they will not lose any market share.

However, it would be great if the Chinese company would publish their findings so that the protocol can be enhanced by others and a more efficient, and certainly open protocol can be created to benefit users.

Think about the 3 main advantages that Skype possesses:
1. Proprietary protocol – which allows them to have tight control over the evolution of the protocol.

2. Good NAT traversal technology – this is the main point. This is critical to their success.

3. Good codec – the GIPS codecs work extremely well. This also ties back to the first point – they know what codecs their clients can support which is much better what gets negotiated in a heterogenous SIP environment.

It’s all in the app. Now, I’d like to see an open source effort is dedicated to making a product work well with the protocol design rather than wasting time trying to obfuscate it!

Using XML-Stream without

Monday, March 7th, 2005

I needed to write a notification script for AMMS, so that a mail server administrator can be notified by IM when the MTA is down.

So I set out to install the popular Net::Jabber module, but soon found it to be tricky. I was using Perl 5.6.1, and chasing the dependency tree led me to XML::Stream, which listed Perl 5.8 as a pre-requisite.

The thing is, the only feature from Perl 5.8 required by XML::Stream is the module. I found the Unicode::UTF8simple module, which was created exactly to fill that gap for us late adopters.

So, here it is, if you wish to install XML::Stream on Perl 5.6 (may work on other versions but I have not tested), you can apply this patch to lib/XML/ before running perl Makefile.PL.

p.s.: Had I used the ever-”useful” CPAN shell to install Net::Jabber, I guess it would’ve proceeded to build a new Perl distribution on my humble Pentium 300Mhz laptop with 32MB of memory.

SLUG Code Con

Tuesday, August 10th, 2004

Note to self: Get your ^ss there.

“Code Con is intended to be a low cost conference, with a focus on developers doing presentations of working code, rather than on companies with products to sell.”

You can’t get much more low cost than this:

Frazer Beach is located in Munmorah State Conservation Area and is managed by NPWS. It features a beautiful sandy beach, rock shelves for fishing, fabulous waves for surfing, amazing rocks for photographing, numerous sea caves and walking tracks to explore, and even a blow hole at high tide. All this is just 2 hours north of Sydney CBD by car.

At $10 a night? You bet I’ll be there!

Great Hackers…

Thursday, July 29th, 2004

are a rare find, but this article by Paul Graham (of “A Plan For Spam” fame) tells you how to sniff them out, how to manage them and be like one of them if you wish. Better yet, get the book.

This is a magnibrilliant article, and echos many of my beliefs. That is not to say that I’m one of those that he speaks about – if not for the fact that I do program Windows applications and use the Java programming language. I do, however, share the same ideals.

While we’re on the topic of Java programmers and employment and how they relate to the variation in wealth, I ran some simple queries on SEEK (AU/NZ’s job search site) today. The findings for various programming languages are as follows:

Java: 315 jobs
Python: 4 jobs
Perl: 87 jobs
C++: 108 jobs
C#: 88 jobs
PHP: 22 jobs
COBOL: 62 jobs
VB: 171 jobs

Eliminating some duplicates from multiple agencies commissioned to fill the same vacancy, we can see that Java has at least a factor of 20-50 openings more than other types of programming jobs. For a while, I was contemplating whether to sell out to the J2EE hype (flame me!!!) but the article has inspired me to stay true to my path. The sky brightens up again.

I Love Hacking

Friday, June 18th, 2004

I can spend hours, even days or months if the project is interesting enough, the more challenging the better.

Created this section to file hacks of my own, as well as interesting ones that I discover elsewhere. I would also post up any ideas of interesting hacks, so people can comment on or pick up and develop further. This sort of thing would probably be better off in a Wiki, but I don’t have time to install one yet (actually, two weeks ago I found one that belongs to a old abandoned project in my cgi-basement.)

Of course, viWord and IDN-OSS gets a mention here since they are technically hacks. (They each have their own section under the Projects hierarchy.)