I started playing with Erlang last year. Mostly that meant reading the Joe Armstrong book, looking at ejabberd and writing a little code. Sadly, I’ve not had the chance to go much beyond the “playing” stage. Anyway, I’ve got a soft spot for functional languages like Erlang since my Programming Language Theory class in undergrad where we used ML. I especially like the way Joe and the rest of the people behind Erlang have built it for concurrency via tiny processes that share nothing and have provided a framework for building apps that know how to operate correctly in soft-realtime. It’s a very different way of thinking about building systems and seems to be remarkably effective.
The Erlang guys have to be feeling pretty good to hear that Facebook has used Erlang as a core component of their new chat service. (High Scalability also has a good writeup.) As Facebook engineer Eugene Letuchy describes it, their implementation uses XHR long polling which means tons of open HTTP connections. Spread this out over 70 million potential users and it’s not hard to see that Apache would break down pretty quickly. Basically it sounds like they have tons of Erlang processes servicing these connections and holding messages and presence events for users in memory if there’s not an open connection to the client.
Eugene mentions the challenge of delivering presence information as being more difficult than real-time messaging. (Something I thought a lot about when building Effusia.) He lays out the issues inherent in broadcasting presence on every state change in the form of a nasty worst-case asymptotic complexity:
The naive implementation of sending a notification to all friends whenever a user comes online or goes offline has a worst case cost of O(average friendlist size * peak users * churn rate) messages/second, where churn rate is the frequency with which users come online and go offline, in events/second.
However, he doesn’t really go into any detail on how they solved this problem. I can only assume they used some form of periodic polling on a need-to-know basis and/or coalescing friend presence updates in such a way that they’re only occasionally sent to a user.
A few other interesting notes… Apparently they used C++ to do the chat logs as Erlang is not that great at raw I/O. They also apparently use Thrift to glue everything together. (Reminds me I need to look into Thrift in more detail.)
