The Counts

countingMy attention was recently focused (thanks John) on an article by Flickr’s Kellan, describing how Flickr generates system wide unique IDs for user data like photos and comments. In short they have a dedicated ID generator that issues unique, bigint(20) numbers. They implemented this by keeping an almost empty table in MySQL and have the server do a non-standard REPLACE INTO and return the SELECT LAST_INSERT_ID(); of the table. The article claims that they needed a dedicated ID generator because they could not come up with a way to make independent servers generate a system wide unique ID and to prevent the introduction of a single-point-of-failure they came up with a way to make multiple independent servers generate a system wide unique ID. It is “a great example of the Flickr engineering dumbest possible thing that will work design principle.” (Kellan Elliott-McCrea) [Ticket Servers: Distributed Unique Primary Keys on the Cheap] But there is no such thing as dumbest-possible-thing. Give me a candidate dumbest-possible-thing and I’ll add a client-server layer and presto you have something even dumber. Like solving a multi-machine problem by building a multi-machine server.
However I am in favor of system design that goes for the “simplest possible thing that will work (but not simpler than that).” And others might rightfully argue that it’s impossible to prove a candidate simplest thing to actually be the simplest. Oh, I am such a hypocrite, and I did do nothing in my life that comes even remotely close to an installation like Flickr, they rock! But it’s amusing nonetheless.

  1. John says:

    And the really funny thing is – they now have two dedicated servers with mysql, just for these numbers. Instead, they could have installed postgresql on those two servers, and have a proper “sequence” implementation and replace that code with a simple select nextval('a');. You can do multiple servers with CREATE SEQUENCE a INCREMENT BY 2 START 1;

Leave a Reply