Doug Hamlin's Brain

This post has been sitting in my drafts for a while. The Clay Shirky video embedded below is one of my all-time favorite videos about social media and technological sociology. I had a few versions of what I wanted to write about this video, but forget all that; the more I think about it, the more I realize it doesn’t matter; just watch this video if you haven’t seen it before. Or, if you have seen it before, watch it again. 

What matters here isn’t technical capital; it’s social capital. These tools don’t get socially interesting until they get technologically boring. It isn’t when the shining new tools show up that their uses start permeating society; it’s when everybody is able to take them for granted. Because now that media is increasingly social, innovation can happen anywhere that people can take for granted the idea that we’re all in this together.
—Clay Shirky

Rex has lately been hosting guest bloggers over at Fimoculous, including one with the initials and Twitter name ADM. In my day job, I often am faced with tactical details like “what do we name our Twitter account.” If I worked for Archer Daniels Midland or Association for Downloadable Media, I would wonder what best to name that organization’s Twitter account that fits within a 15-character minimum. @ADM is the obvious choice that is already taken. Twitter needs to do something about that, like it does about a handful of problems. 

A terrifying thought hit me Monday while watching this flyover video of Apple’s new $1 billion 500,000 square foot data center in North Carolina.

Forget sophisticated digital attacks on our infrastructure; there is absolutely nothing preventing extreme anti-government thugs like Joe Stack from flying planes into the data centers that increasingly are synonymous with computing, the Internet, the cloud and our lives. 

Data centers are famous for their tight access restrictions, but as we learned in last week’s IRS attack, an aerial attack doesn’t care about access cards and finger print scans. 

Just consider this:

• An accidental outage at a Texas data center can grind parts of the Internet to a halt;
• The U.S. is not prepared for a sophisticated digital attack, never mind thugs in planes;
• Just today, part of Google’s cloud failed.

Hitting a few key data centers could be catastrophic. No wonder companies don’t like sharing information about these campuses.

Scariest part is I can’t possibly be the first person to have this thought.

Twitter’s geolocation guru Raffi Krikorian recently gave an interview to O’Reilly ahead of next month’s Where 2.0 conference in San Jose. The interview is obviously heavy on how Twitter is working with geo, but the last question, or more specifically, the last answer struck a chord and goes straight to the problem of trying to deal with the massive amounts of real-time data a lot of web players are dealing with today.

James Turner: What do you see as the technical side of geolocation, in terms of what’s going to be the new interesting technologies coming along, and how they’re going to be used?

Raffi Krikorian: From Twitter’s standpoint, it’s how do you accept all of this real-time data, index and analyze it and spread it throughout our system in almost real-time. People have traditionally built a bunch of GIS-like systems on top of PostgreSQL or on top of MySQL, and that’s fine, but it doesn’t scale after a while. After you throw a couple million or a couple hundred million entries at it, the amount it takes for one of those databases to process that, to insert it, all I have to do is select against it, and you can understand it’s untenable for real-time operation. And by real-time, I mean sub-second operation.

So the stuff that we’re doing is more geared towards how can you accept tweets that are coming in at what you can imagine to be an incredibly fast rate. Tweets are coming in, figure out their location, attach appropriate metadata data to it. Store it in our database. Span it out to anyone who wants to look at it. Run research and analytics on it and index it in their search index, and do this all within a couple of seconds on the way through the system. I think there’s a lot of interesting stuff being done out there on how things are being stored, how things are being indexed. But I think our personal contribution will be how do you do it at that kind of speed?

Of course, so far Twitter has been doing an admirable job (Fail Whales excluded) of providing uptime on a service with (forget a couple million) 9 billion rows in its statuses table — a table that’s growing by 50 million rows per day.

Those 600 tweets per second are certainly what turned Twitter away from a SQL cluster to using rival Facebook’s Cassandra system. On Tuesday, Twitter’s Ryan King explained why Twitter is turning to Cassandra.

We have a system in place based on shared mysql + memcache but its quickly becoming prohibitively costly (in terms of manpower) to operate. We need a system that can grow in a more automated fashion and be highly available.

My day job involves a lot of media monitoring with such products at Radian 6. Speaking last week with one of R6′s competitors, I was told that no one in the media monitoring space can really do real-time monitoring — there’s just too much data. I think that’s overstating the challenge a little bit, but it will surely be some time before a company can say it’s “real-time” without employing an army of engineers.

Love this. Newsweek published an internal e-mail thread about who should and shouldn’t be labeled a terrorist.

For anyone who has ever had insomnia, this study reported in today’s Times.

What they found was that subjects took slightly longer to fall asleep on nights they were instructed to distract themselves by counting sheep or were given no instructions at all. But when they were told to imagine a relaxing scene — a beach, for example — they fell asleep an average of 20 minutes sooner than they did on other nights. Counting sheep, the scientists suggested, may simply be too boring to do for very long, while images of a soothing shoreline or tranquil stream are engrossing enough to concentrate on.