They are usually cluttered with distracting animated gifs, elaborate signatures and tons of useless stats about the posters (novice/expert level, im status, joining date, etc.)

They are notoriously cumbersome to navigate, let alone find what you need. The forum administrators and moderators know it, so most forums use stickies as a band-aid. For the casual visitor, stickies are usually road signs that tell you where to look for certain things, a summary of important points or FAQs. Needless to say, they’re pretty effective when you compare them with the rest of the mess, but a band-aid nonetheless.

I’d blame the user interface for this awkward communication medium. It’s not so much due to its age, because I’d rather read stuff on old skool NNTP with vi key bindings than to navigate these forums. Rather, it’s the constant need for paging (displaying page 1 of 32!!), which means that you can’t effectively use the browser’s in-page find feature. Search is usually broken or otherwise less than relevant. No threading, nor ability to easily filter messages. I could go on.

Perhaps it’s just a matter of tweaking the skin and applying sane user interface design. Certainly, there are some better-designed forums out there that are less painful to use. I haven’t seen much innovation in that area in a long time, actually not since my first encounter with them. I suspect people actually like and have come to expect those cumbersome features.

Forum communities are an entire subculture of their own, and I don’t expect a shift anytime soon. Yet, I can’t help but wonder if we can do better.

₁ In case you’re wondering, I intentionally used the word “forums” instead of “fora”.

Related posts:

1. EchIDNA Speaks Korean 안녕하세요! There are two pieces of great news I’d like to report, both of which relate to the Korean Internet...
2. Whirlpool, aaNet, Exetel, and XXX Here’s a piece of interesting news from Whirlpool — Australia’s most popular community driven broadband news site. Quoting a forum...

Related posts:

1. links for 2009-02-18 Whoosh Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python. May be an easier-to-integrate...
2. Google Web Toolkit (AJAX library) When I first saw the news about the newly released Google’s Web Toolkit: Google Web Toolkit (GWT) is a Java...
3. Python 3.0: Text vs. Data Instead Of Unicode vs. 8-bit Python 3.0 (Py3K) is out. I’m with Sam Ruby — this seemingly simple change of paradigm from “Unicode vs. 8-bit”...

Seth Godin’s book Tribes: We Need You to Lead Us looks like a good read, especially for marketers, crowd-herders, and entrepreneurs. Along with the book, he also started an invitation-only triiibal network on Ning, and got the folks to write an ebook called The Tribes Casebook (free download).

There’s a particular essay in there written by Dr. Saleh AlShebil titled When Technology Fails: A Language gets Born in an Online Tribe. Dr. AlShebil wrote about how an ASCII-based language (that he calls Araby) was born due to the lack of Arabic language input support on early instant messaging networks. These are transliterations of Arabic into Latin alphabets, not unlike l33t but grew out of different motivations.

Here’s what it looks like (source):

So, واحد (one) sounds roughly like “wahed”, and you’d write it as “wa7ed”.

Quoting Dr. AlShebil (emphases added):

Arabic language alphabet is comprised of 28 letters. Some of these letters do not have an equivalent “sound” in English. So what did our online tribe do? They began looking for numbers and other keystrokes that can somehow resemble what the real Arabic letter “looks” like. Let me explain…

For instance, the Arabic letter “ﻉ” is pronounced as A’aa when used in a word and it got replaced with the number “3” since “3” looks like an inverted “ﻉ”. So the word Arabic which is written “Araby” (in Arabic sounding English) and begins with “ﻉ” was then written as “3raby.”

…This new form of tribal net lingo began to spread like wildfire. It would probably be a safe assumption to say that any Arab who is online today (especially the youth) is pretty familiar with it. Using it was not limited to chat and instant messaging but has also swelled to include any form of writing in online communities and even in mobile text messaging (sms). The Arabic net lingo virus caught on to Arabic websites that even wanted their domain names to sound or “look” Arabic.

As mentioned above, this is similar to l33t-speak, and also the lesser-known ギャル文字 (Gyaru-Moji).

Now, I dig subcultures like these, but don’t you think there’s something wrong with the emergence of a new lingo that could potentially erode a language like Arabic just because technology couldn’t support it?

Is this serious enough to erode the Arabic language? Maybe I’m exaggerating but one can imagine youths forgetting how to spell correctly in Arabic script because they’re so used to using “Araby”.

This is the case for why internationalization is important for the Internet (and technology in general.) More importantly, it is the prime motivation behind Internationalized Domain Names, which is in turn a primary contributor to the need for new TLDs.

Internationalization is not for vanity or luxury, it’s a necessity to preserve culture.

Related posts:

1. .test IDN TLD (For Real!) Tina Dam and I caught up during ICANN San Juan on the current state of IDN TLD work within ICANN...
2. What’s in an i-name? The project that I’ve been working for almost a year at NeuStar is very much centered around a marketing term...
3. IDN in Dot Net RFP The .NET RFP evaluation report by Telcordia is disappointing because: IDN was not given the weight that it deserves; it...

Related posts:

1. links for 2009-02-19 JavaScript Madness: Dynamic Script Loading (tags: bookmarklet js loading script webdev ajax) Scope in JavaScript | Digital Web Magazine...
2. I’m Flocked Just downloaded flock and am totally lovin’ it. Being able to manage my flickr and photobucket account with drag-n-drop...
3. Distinguished Business Leader Series Yesterday, Alvin and I attended the “Distinguished Business Leader Series” talk jointly organized by the Singapore Prime Minister’s Office and...

This was a controlled experiment combining 3 technologies (2 of which from the Open Stack but hybridized) under the hood to create a streamlined signup experience that goes like this:

1. Someone at Plaxo invites you to join by entering your Gmail address
2. You get an invitation email from Plaxo
3. You click on the link
4. Plaxo knows that you’re a GMail user (and likely still signed in), so it presents you with the following screen:
   
   I believe that since Plaxo already has your Gmail address, it is already somehow encoded in here to save you from having to type it in, but I haven’t tried it so I’m not sure
5. Clicking “Sign up with my Google Account” brings you over to Google with the following screen:
   
6. Clicking “Continue Sign-in” tells Plaxo that you are indeed the holder of the Gmail address, at the same time authorizing Plaxo to import your address book from Google.
7. That’s it! You’re signed up to Plaxo and your Gmail address book is available in Plaxo.

The result was a staggering 92% return rate (from the Google authorization confirmation screen above), of which 92% continued with the sign up and allowed Plaxo to import their contacts from their Google address book. The results were so impressive that Plaxo’s business folks stopped the tech folks from turning off the experiment!

Indeed these results are impressive by today’s standard of endless signup forms and social networking fatigue. I would whole-heartedly agree that through this clever experiment, Plaxo has met their goals of making it better for the user, the identity provider, as well as the relying site.

The technologies that made these possible were:

• OpenID for proving who you are (to Plaxo that you do indeed own the GMail address.)
• OAuth (implemented as an extension to OpenID) was used to grant Plaxo access to your contacts stored on Google; and
• Google Contacts API for actually importing them into Plaxo (would be nice to see Portable Contacts being adopted by Google)

Individually, those technologies are good at what they’re designed to do but when combined with a simple hint such as “the user is a GMail account holder, and is probably still signed in to the service”, it could be very powerful.

Still, my biggest takeaway from the slides are:

• 17% (of Plaxo signups) come from GMail account holders; and
• 73% come from the top 4 (Yahoo, Microsoft, Google, and AOL)
• all of them being OpenID Providers

This shows that you can already take advantage of the fact that a large percentage of users already own an OpenID, who may be more willing to sign up to your service than they otherwise wouldn’t have if faced with another tedious registration form.

While many (including myself) have criticized OpenID that there are more providers than relying parties, Plaxo has proven (with impressive numbers) that with a little ingenuity and optimization of UX, sites can reap the benefits of being an RP!

Related posts:

1. OpenID for Drupal There was a thread on the OpenID list around the subject of OpenID support in Drupal. Previously, I’ve experimented with...
2. On Mobile OpenID in Japan This presentation by =zigorou (Toru Yamaguchi) titled “Considering OpenID for Mobile” (Thanks =peterd and =nat) is particularly interesting for me...
3. This blog is OpenID enabled! One thing led to another. After reading my previous entry on “mod_python OpenID Access Control”, Nate Olson contacted me...

Notice how the words “suspension”, “service”, “trouble” (and the partially visible “Shinkansen”) were brutally split at arbitrary boundaries without any use of hyphen. While this is common practice for the Japanese language, it certainly makes it hard for English readers to scan the information being presented.

I can only vaguely remember early computing systems where editors didn’t do word wrapping (anybody care to refresh my memory?)

Related posts:

1. On Mobile OpenID in Japan This presentation by =zigorou (Toru Yamaguchi) titled “Considering OpenID for Mobile” (Thanks =peterd and =nat) is particularly interesting for me...
2. Google Web Toolkit (AJAX library) When I first saw the news about the newly released Google’s Web Toolkit: Google Web Toolkit (GWT) is a Java...
3. Why Internationalize? Seth Godin’s book Tribes: We Need You to Lead Us looks like a good read, especially for marketers, crowd-herders,...

It does take some getting used to but trust me it’s a wise investment.

The following settings (in .vimrc) works well for me:


:let g:html_tag_case = 'lowercase'
:let g:do_xhtml_mappings = 'yes'


Plus the ability to quickly insert a customized blank HTML template is just godsend!

I’ve procrastinated way too long, but if you’re like me and use Vim to edit HTML a lot, you should definitely check it out!

No related posts.

I remember reading about Apple supporting emoji on the iPhone OS 2.2. Now that I’ve upgraded, I decided to try it out but for could never find it after hunting through the keyboard preferences. Googling showed that these cute little emoticons are only available for Softbank users. Thankfully, Steven Troughton-Smith has figured out that by editing a file on your iPhone backup, the “emoji” option suddenly shows up under Settings -> General -> Keyboard -> International Keyboards -> Japanese!

Now that I have the ability to enter these emoji on my iPhone, I figure I’d try it out by sending an email to myself. Alas, all I get is a list of of boxes. Time to look at the message content (relevant fields):

Content-Type: text/plain;
  charset=cp932;
  format=flowed
Content-Transfer-Encoding: base64
X-Mailer: iPhone Mail (5G77)
Mime-Version: 1.0 (iPhone Mail 5G77)
Subject: trying out emoji
Date: Sat, 31 Jan 2009 19:11:03 +0800

aGV5IHRoZXJlIPGQDQoNCvWc9d3wSPB78unzR/SY9Jv3RPCW9vnwlQ==

ok, pretty strange that it’s sent in cp932 encoding, but we’ll see:

>>> from base64 import b64decode
>>> s = b64decode('aGV5IHRoZXJlIPGQDQoNCvWc9d3wSPB78unzR/SY9Jv3RPCW9vnwlQ==')
>>> s
'hey there \xf1\x90\r\n\r\n\xf5\x9c\xf5\xdd\xf0H\xf0{\xf2\xe9\xf3G\xf4\x98\xf4\x9b\xf7D\xf0\x96\xf6\xf9\xf0\x95'
>>> s.decode('cp932')
u'hey there \ue10b\r\n\r\n\ue407\ue448\ue008\ue03b\ue220\ue23b\ue347\ue34a\ue528\ue055\ue520\ue054'
>>>

The Unicode code points look like they do correspond to the Softbank private use ones, so I’m going to use the emoji4unicode package to convert it to HTML. The following Python script will convert it to the various carrier’s representation:

import emoji4unicode
import carrier_data

s = b64decode('aGV5IHRoZXJlIPGQDQoNCvWc9d3wSPB78unzR/SY9Jv3RPCW9vnwlQ==')
uni = s.decode('cp932')

sd = carrier_data.GetSoftbankData()
dd = carrier_data.GetDocomoData()
kd = carrier_data.GetKddiData()

emoji4unicode.Load()
def find_symbol(pua, carrier):
    for sym in emoji4unicode.GetSymbols():
        uni = sym.GetCarrierUnicode(carrier)
        if uni and uni == pua:
            return sym

def map_symbol(sym, carrier, cdata):
    uni = sym.GetCarrierUnicode(carrier)
    if not uni: # no mapping for this carrier
        return sym.GetTextFallback()
    else:
        if uni.startswith(">"): # mapped
            if len(uni) > 5:
                raise Exception, "cannot handle this yet"
            uni = uni[1:]

        return cdata.SymbolFromUnicode(uni).ImageHTML()

softbank = []
docomo = []
kddi = []

for u in uni:
    hex = "%04X" % ord(u)
    if u > '\x7F':
        sym = find_symbol(hex, "softbank")

        softbank.append(sd.SymbolFromUnicode(hex).ImageHTML())
        kddi.append(map_symbol(sym, "kddi", kd))
        docomo.append(map_symbol(sym, "docomo", dd))
    elif u == '\n':
        softbank.append('<br />')
        kddi.append('<br />')
        docomo.append('<br />')
    else:
        softbank.append(u)
        kddi.append(u)
        docomo.append(u)

print "Softbank:<br />", ''.join(softbank).encode('utf-8')
print "<hr />KDDI:<br />", ''.join(kddi).encode('utf-8')
print "<hr />DoCoMo:<br />", ''.join(docomo).encode('utf-8')

Results below:

Softbank:
hey there

KDDI:
hey there

DoCoMo:
hey there

[サンタ][<][イチゴ][ナス][サル][イルカ][クジラ]

As you can see, DoCoMo has the least number of emoji’s, so many of the characters like “Santa”, “Strawberry”, “Eggplant”, “Monkey”, “Dolphin” and “Whale” are substituted by the fallback text format.

Do note that the python script is not optimized at all, and loops through every emoji in the database for each character it needs to convert. Also, if you need such functionality in your application, there are various libraries out there that already does the mapping well. This is just an experiment.

Related posts:

1. Emoji to be encoded in Unicode The Unicode Technical Committee is working on encoding emoji (絵文字) in the Unicode Standard and ISO10646. It has spurred loads...
2. On Mobile OpenID in Japan This presentation by =zigorou (Toru Yamaguchi) titled “Considering OpenID for Mobile” (Thanks =peterd and =nat) is particularly interesting for me...
3. iPhone-blogging I’ve always wanted to post to my own blog from the mobile phone rather than using third party blogging platforms....

Then, all of a sudden, now it’s all coming back. Every other minute, I’m getting hit by ideas and everything I do, I just feel like blogging it. I could’ve easily spurted in under 140 characters on Twitter or something, but it doesn’t feel right. I’m being possessed by the need to elaborate.

Is this the reverse of the Writer’s block — Writer’s drainage?

Let’s see if I can define it:

Writer’s Drainage

a phenomenon whereby a person who seldom writes suddenly gets streams of inspiration in during a period of time.

If that’s the case, I sure hope this is a chronic case because I certainly need the practice.

Moving on to the next post…

Related posts:

1. Linksys WRT54GP2 Router with ATA I seldom get a toy like this, so I’m very excited! This is essentially a combination of the WRT54G wireless...
2. iPhone-blogging I’ve always wanted to post to my own blog from the mobile phone rather than using third party blogging platforms....
3. IP Migration Our excellent hosting provider, Simpli.BIZ, has had in the past a few glitches due to the fault of AboveNet, their...

Not to worry, the Unicode consortium is a veteran when it comes to dealing with the hairy issues of creating standards that work across languages, cultures and geographic regions. They simply can’t please everyone.

To me, the motivation for this is clear — interoperability. The current state of affairs in the Japanese mobile industry leaves a lot to be desired: across the carriers, there exist different sets of supported emoji’s, different private-use characters, substitution mappings, and code pages (user-defined characters in Shift_JIS, really). As one can imagine, the results is chaos, and as I software engineer, I really don’t want to imagine what those poor software engineers have to do to make it “just work” when a message cross the carrier boundaries.

To illustrate my point, let’s look at what Google does when you send a message with some emoji characters from GMail to each of DoCoMo, Softbank, and au.

The screenshot above is for DoCoMo, but I also repeated the experiment for Softbank and au. From the bounce message, one can easily tell that what is saved as a sent message and what gets actually transmitted to each of the carriers’ SMTP server are all distinct in their encodings.

What’s saved in Gmail when you click on “Show original” embeds the graphics using standard mime techniques (multipart/related with CID URIs a.k.a RFC 2111) with an extension attribute called goomoji in the HTML version, which carries part of Unicode private use character assigned for it. For example, the crab is assigned U+FE1E3 in Google, so its goomoji value is 1E3.

What’s sent to DoCoMo is a different story altogether:

It’s a standard multipart/alternative message with 2 parts: text/plain and text/html, both encoded in Shift_JIS. Decoding the text/plain part gives:

>>> import base64
>>> sjis = base64.b64decode("W4NKg2pd+aT56ApYT1hPIIFfKF4tXimBXgoKaHR0cDovL3hyaS5uZXQvPXdpbAo=")
>>> sjis
'[\x83J\x83j]\xf9\xa4\xf9\xe8\nXOXO \x81_(^-^)\x81^\n\nhttp://xri.net/=wil\n'
>>> print sjis.decode("shift_jis", 'ignore')
[カニ]
XOXO ＼(^-^)／

http://xri.net/=wil

Since DoCoMo doesn’t have the decapods in their emoji set, it gets encoded as カニ (Japanese for crab) in square brackets. Next comes the double musical notes , which is assigned a user defined Shift_JIS value of F9A4 in DoCoMo (explains why I had to pass the ‘ignore’ parameter to the decode method above, Python has no way to map that that sequence to Unicode and therefore barfs). Same goes for the tulip . Ignoring the plain text “XOXO”, the last emoticon is a hug face , which Google assigned a code point to but none of the carriers use a graphic to represent. In fact, this is mapped to a Kao-moji (顔文字 - “face words”).

For au (KDDI), the message was also sent in multipart/alternative with text/plain and text/html parts but this time encoded in ISO-2022-JP. Similar situation here, where crab => [カニ], KDDI’s version of ISO-2022-JP for the musical notes and tulip emoji, and kaomoji for the hugs.

Similar deal for Softbank, but the charset specified is PDC, but it smells just like Shift_JIS with user-defined characters to me.

I hope by now you have an appreciation of the kind of fiddling that Google engineers had to do in order to get their messages to display properly on Japanese mobile phones, just because the carriers decided to go invent their own mappings and character sets. It’s no wonder that Google and Apple, with its recently announced emoji support in iPhone, are among those supporting this effort.

The ongoing work can be found here and all the emoji’s is available here and here in gory details.

If I had the time and luxury (read: paid) to participate, I would. I wish them all the best and hope to see a good set of emoji’s in Unicode soon.

Related posts:

1. Emoji Encoding Conversion between Carriers I remember reading about Apple supporting emoji on the iPhone OS 2.2. Now that I’ve upgraded, I decided to...
2. Python 3.0: Text vs. Data Instead Of Unicode vs. 8-bit Python 3.0 (Py3K) is out. I’m with Sam Ruby — this seemingly simple change of paradigm from “Unicode vs. 8-bit”...
3. On Mobile OpenID in Japan This presentation by =zigorou (Toru Yamaguchi) titled “Considering OpenID for Mobile” (Thanks =peterd and =nat) is particularly interesting for me...