We weren't imagining things: Twitter's Fail Whale has been resurrected as of late, and it's your fault.
Well. Maybe it isn't your fault, but according to Twitter's API lead, Alex Payne, the Twitter team has been keeping an eye on the mammal's return, which they attribute to recent traffic surges.
"Weekly when we get together and talk as a company we measure downtime in minutes," said Payne in an interview with Internet Evolution yesterday at Twitter's appropriately posh San Francisco headquarters. "We have a number of other metrics -- the time to send a message, receive a message... We look at the number of Fail Whales we throw every day. We literally have a graph and can see it hour by hour. We're very, very cognizant of it."
Payne -- who spoke against the backdrop of a loaded bicycle rack and vintage telephone booth -- attributes Twitter's recent downtime less to the amount of messages being sent over the system and more to the third-party clients using its API.
"People sending updates doesn't hurt us half as much as just people making heavy use of API clients," says Payne. "If someone has a TweetDeck up and they've got 40 different searches for every variation of their company... We want people to be doing that stuff, we love when people do that stuff. But that's what gets really expensive. Every 30 seconds they're going out and hitting us 10 different times."
Part of the problem is, with Twitter now having reached mainstream status, the surges in traffic are unpredictable, leaving the team unprepared.
"We know for big events -- the presidential inauguration, Super Bowl, Apple announcements -- we're going to see a spike in traffic and we can brace ourselves a little bit," says Payne.
Nevertheless, Twitter's engineers are working to keep it intact. Part of this process has included using programming language Scala on top of Ruby, a language Payne expressed optimism about during our first interview last year. Since then, Twitter has had to limit Ruby to the front-end of the system.
"It was kind of depressing," says Payne. "We were at the Ruby conference a couple years ago and their lead designer for language got up and acknowledged Twitter and that we were growing really fast and were an example of Ruby hitting the big time -- but went on to say 'we're not that interested in working on performance issues.'
"So in that respect we kind of felt like we needed to keep the language in a part of system that isn't stressed."
Apart from introducing Scala, Twitter engineers are breaking out the system's functionality into smaller chunks that are easier to maintain -- as well as working to better handle API requests. For example, as of yesterday a feature on Twitter's home page called "@ replies" was changed to "mentions" in an effort to give its search API a traffic break.
"Clients like TweetDeck and Twhirl were making one request to the API to get replies and another request to the search API to find mentions," says Payne. "We wanted to consolidate on those two to cut down on the traffic."
With the work being done to sustain it, Payne says he isn't too worried about Twitter's ability to handle volume, long-term. But he and his colleagues have nevertheless grown weary of the dreaded oceanic mammal.
"It's sort of great that the failure in our system has become a brand in and of itself and users love it and celebrate it," he says of the Fail Whale's celeb status. "But yeah we're sick of it. We don't want to see it anymore."
— Nicole Ferraro, Site Editor, Internet Evolution
Email This
