I remember reading about Apple supporting emoji on the iPhone OS 2.2. Now that I’ve upgraded, I decided to try it out but for could never find it after hunting through the keyboard preferences. Googling showed that these cute little emoticons are only available for Softbank users. Thankfully, Steven Troughton-Smith has figured out that by editing a file on your iPhone backup, the “emoji” option suddenly shows up under Settings -> General -> Keyboard -> International Keyboards -> Japanese!

Now that I have the ability to enter these emoji on my iPhone, I figure I’d try it out by sending an email to myself. Alas, all I get is a list of of boxes. Time to look at the message content (relevant fields):

Content-Type: text/plain;
  charset=cp932;
  format=flowed
Content-Transfer-Encoding: base64
X-Mailer: iPhone Mail (5G77)
Mime-Version: 1.0 (iPhone Mail 5G77)
Subject: trying out emoji
Date: Sat, 31 Jan 2009 19:11:03 +0800

aGV5IHRoZXJlIPGQDQoNCvWc9d3wSPB78unzR/SY9Jv3RPCW9vnwlQ==

ok, pretty strange that it’s sent in cp932 encoding, but we’ll see:

>>> from base64 import b64decode
>>> s = b64decode('aGV5IHRoZXJlIPGQDQoNCvWc9d3wSPB78unzR/SY9Jv3RPCW9vnwlQ==')
>>> s
'hey there \xf1\x90\r\n\r\n\xf5\x9c\xf5\xdd\xf0H\xf0{\xf2\xe9\xf3G\xf4\x98\xf4\x9b\xf7D\xf0\x96\xf6\xf9\xf0\x95'
>>> s.decode('cp932')
u'hey there \ue10b\r\n\r\n\ue407\ue448\ue008\ue03b\ue220\ue23b\ue347\ue34a\ue528\ue055\ue520\ue054'
>>>

The Unicode code points look like they do correspond to the Softbank private use ones, so I’m going to use the emoji4unicode package to convert it to HTML. The following Python script will convert it to the various carrier’s representation:

import emoji4unicode
import carrier_data

s = b64decode('aGV5IHRoZXJlIPGQDQoNCvWc9d3wSPB78unzR/SY9Jv3RPCW9vnwlQ==')
uni = s.decode('cp932')

sd = carrier_data.GetSoftbankData()
dd = carrier_data.GetDocomoData()
kd = carrier_data.GetKddiData()

emoji4unicode.Load()
def find_symbol(pua, carrier):
    for sym in emoji4unicode.GetSymbols():
        uni = sym.GetCarrierUnicode(carrier)
        if uni and uni == pua:
            return sym

def map_symbol(sym, carrier, cdata):
    uni = sym.GetCarrierUnicode(carrier)
    if not uni: # no mapping for this carrier
        return sym.GetTextFallback()
    else:
        if uni.startswith(">"): # mapped
            if len(uni) > 5:
                raise Exception, "cannot handle this yet"
            uni = uni[1:]

        return cdata.SymbolFromUnicode(uni).ImageHTML()

softbank = []
docomo = []
kddi = []

for u in uni:
    hex = "%04X" % ord(u)
    if u > '\x7F':
        sym = find_symbol(hex, "softbank")

        softbank.append(sd.SymbolFromUnicode(hex).ImageHTML())
        kddi.append(map_symbol(sym, "kddi", kd))
        docomo.append(map_symbol(sym, "docomo", dd))
    elif u == '\n':
        softbank.append('<br />')
        kddi.append('<br />')
        docomo.append('<br />')
    else:
        softbank.append(u)
        kddi.append(u)
        docomo.append(u)

print "Softbank:<br />", ''.join(softbank).encode('utf-8')
print "<hr />KDDI:<br />", ''.join(kddi).encode('utf-8')
print "<hr />DoCoMo:<br />", ''.join(docomo).encode('utf-8')

Results below:

Softbank:
hey there

KDDI:
hey there

DoCoMo:
hey there

[サンタ][<][イチゴ][ナス][サル][イルカ][クジラ]

As you can see, DoCoMo has the least number of emoji’s, so many of the characters like “Santa”, “Strawberry”, “Eggplant”, “Monkey”, “Dolphin” and “Whale” are substituted by the fallback text format.

Do note that the python script is not optimized at all, and loops through every emoji in the database for each character it needs to convert. Also, if you need such functionality in your application, there are various libraries out there that already does the mapping well. This is just an experiment.

No related posts.

Not to worry, the Unicode consortium is a veteran when it comes to dealing with the hairy issues of creating standards that work across languages, cultures and geographic regions. They simply can’t please everyone.

To me, the motivation for this is clear — interoperability. The current state of affairs in the Japanese mobile industry leaves a lot to be desired: across the carriers, there exist different sets of supported emoji’s, different private-use characters, substitution mappings, and code pages (user-defined characters in Shift_JIS, really). As one can imagine, the results is chaos, and as I software engineer, I really don’t want to imagine what those poor software engineers have to do to make it “just work” when a message cross the carrier boundaries.

To illustrate my point, let’s look at what Google does when you send a message with some emoji characters from GMail to each of DoCoMo, Softbank, and au.

The screenshot above is for DoCoMo, but I also repeated the experiment for Softbank and au. From the bounce message, one can easily tell that what is saved as a sent message and what gets actually transmitted to each of the carriers’ SMTP server are all distinct in their encodings.

What’s saved in Gmail when you click on “Show original” embeds the graphics using standard mime techniques (multipart/related with CID URIs a.k.a RFC 2111) with an extension attribute called goomoji in the HTML version, which carries part of Unicode private use character assigned for it. For example, the crab is assigned U+FE1E3 in Google, so its goomoji value is 1E3.

What’s sent to DoCoMo is a different story altogether:

It’s a standard multipart/alternative message with 2 parts: text/plain and text/html, both encoded in Shift_JIS. Decoding the text/plain part gives:

>>> import base64
>>> sjis = base64.b64decode("W4NKg2pd+aT56ApYT1hPIIFfKF4tXimBXgoKaHR0cDovL3hyaS5uZXQvPXdpbAo=")
>>> sjis
'[\x83J\x83j]\xf9\xa4\xf9\xe8\nXOXO \x81_(^-^)\x81^\n\nhttp://xri.net/=wil\n'
>>> print sjis.decode("shift_jis", 'ignore')
[カニ]
XOXO ＼(^-^)／

http://xri.net/=wil

Since DoCoMo doesn’t have the decapods in their emoji set, it gets encoded as カニ (Japanese for crab) in square brackets. Next comes the double musical notes , which is assigned a user defined Shift_JIS value of F9A4 in DoCoMo (explains why I had to pass the ‘ignore’ parameter to the decode method above, Python has no way to map that that sequence to Unicode and therefore barfs). Same goes for the tulip . Ignoring the plain text “XOXO”, the last emoticon is a hug face , which Google assigned a code point to but none of the carriers use a graphic to represent. In fact, this is mapped to a Kao-moji (顔文字 – “face words”).

For au (KDDI), the message was also sent in multipart/alternative with text/plain and text/html parts but this time encoded in ISO-2022-JP. Similar situation here, where crab => [カニ], KDDI’s version of ISO-2022-JP for the musical notes and tulip emoji, and kaomoji for the hugs.

Similar deal for Softbank, but the charset specified is PDC, but it smells just like Shift_JIS with user-defined characters to me.

I hope by now you have an appreciation of the kind of fiddling that Google engineers had to do in order to get their messages to display properly on Japanese mobile phones, just because the carriers decided to go invent their own mappings and character sets. It’s no wonder that Google and Apple, with its recently announced emoji support in iPhone, are among those supporting this effort.

The ongoing work can be found here and all the emoji’s is available here and here in gory details.

If I had the time and luxury (read: paid) to participate, I would. I wish them all the best and hope to see a good set of emoji’s in Unicode soon.

No related posts.

What’s inconsistent in this new version though is that the new bytes type still contains many of the methods with text semantics that should only make sense as string methods: e.g. capitalize() and islower(). I suspect these are provided as convenience methods, which is fine. But one would imagine that these byte methods will work by decoding the bytes using the default encoding of your locale, then performing the operations on the resulting string. As it turns out from my trials, it seems to assume that your bytes are encoded in Latin1:


>>> greek_beta = "Β" # This is the uppercase greek letter "beta", not regular B.
>>> greek_beta.isupper()
True
>>> greek_beta.islower()
False
>>> greek_beta.lower()
'β'
>>> greek_beta_bytes = greek_beta.encode('iso8859-7')
>>> greek_beta_bytes
b'\xc2'
>>> greek_beta_bytes.isupper()
False
>>> greek_beta_bytes.islower()
False
>>> greek_beta_bytes.lower()
b'\xc2'
>>> greek_beta_bytes.upper()
b'\xc2'


This is definitely a gotcha that may lead to hard to find bugs. So, it is best to avoid using those methods on bytes objects.

Otherwise, this change is definitely more “correct” in that Py3K forces you to know the type of your variables earlier or at the interfaces (to the outside world) so errors like these are less likely to sneak up from your back. For example, you can no longer use the + (sequence concatenation operator) to mix text and data. Whereas in Python 2.x, you can do:


>>> name = 'Wil'
>>> greet = lambda n: u'Hello ' + n
>>> greet(name)
u'Hello Wil'
>>> name = u'François'.encode('utf-8')
>>> name
'Fran\xc3\xa7ois'
>>> greet(name)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)


What went wrong was that you’re relying on Python’s automatic Unicode conversion, which uses the standard ASCII codec (which is a good thing) to promote your str to unicode.

In Python 3.x, you will be greeted by the TypeError: Can't convert 'bytes' object to str implicitly message if you tried to pass a bytes object to the function. This will happen on any bytes object, so the error is easier to catch.

In this new version, the unicodedata module is upgraded to Unicode version 5.1.0.

Now, I’m not ready to run production code on Py3K yet but it would be nice if Django (my favourite Python-based web framework) can run on it. It looks like Martin von Löwis has started the porting.

Related posts:

1. Tornado with VirtualEnv and Pip Quickstart Friendfeed’s open source Tornado web server is great, and is incredibly easy to get up-and-running. Just install tornado, write your...