Here is how David Karp, founder of Tumblr, recently commented on Tumblr’s downtime to TechCrunch’s Erick Schonfield.

“Karp admits that the company was “unprepared” for that kind of hockey-stick hypergrowth, but with a new $30 million round in the bank, he says his team is working round the clock to keep scaling and catching up with all the sudden demand. Karp says the growth is coming in part from college students, who really took to the service only since September, 2009 or so and, more recently, international growth in Europe, Japan,and Brazil. He also tells me separately that 65 percent of those pageviews come from Tumblr users looking at their Dashboards (which shows the stream of posts from other people on Tumblr they follow).”

Good.

Also I just found this post on Karp’s blog that has this interesting bit.

“Ah, yes – an incredible opportunity and challenge!

The really impressive piece is that our engineers have been keeping up with this surge in traffic while serving fewer and fewer errors every week. It’s been a rough couple of months, but we’re almost there.”

“Opportunity and challenge” is the perfect way to put it. Karp gets it. Now if only Tumblr assigned someone on the staff to do updates and share stats on these “fewer errors every week” via the main Staff blog? Oh wait, they already did.

Tumblr did a 180. Congrats.

“Of course, all companies experience episodes like this. How they handle the situation is what counts. I’m not talking about fixing the problem—you have to fix it; that’s a given. I’m talking about how you communicate with your customers, how you accept responsibility, and how you make things right. That’s what people remember.”

Jason was talking about Campfire in this article, a paid service, used by companies to communicate remotely and is a vital part of their workflow in many cases. Arguably Tumblr isn’t such a service and is, for the most part, free to use. So does this quote apply? I leave that to you to decide.

Bijan Sabet, partner at Spark Capital and one of the lead investors in Tumblr, today on his tumblog:

“i gotta talk to @davidkarp about this. if i’m reading this correctly, it looks like Tumblr is *growing* by 200M page views per week.”

You’ll be hard pressed to find him talking too much about their downtime.

And again (because I feel as though this post could come across the wrong way), I’m willing to bet all of the money in my pockets that the Tumblr team is doing everything they can to keep their service up and running – I just think they and their investors should be talking about it more. They should be talking about how they’re pulling out all of the stops, pushing all of their resources and people at the problem, their successes and failures in that area. Simply ignoring the subject and constantly trying to talk only about the good things that are going on smells like propaganda and spin. And I seriously doubt that is intentional but that is the way it smells from here.

Note to self: Learn from this.

I even called Zach Inglis out for his tirade against the Tumblr team. Now I’m thinking, perhaps, he was completely justified. Or, maybe, I spoke too soon. However, I also believe that the Tumblr team (or perhaps just the investors) and its core users want two very different things for Tumblr.

One of Tumblr’s main investors and mentors has been Union Square Ventures. USV is an incredibly adept team of venture capitalists who, for the most part, have made some excellent bets over the years and whose opinions I respect. Put simply, guys like Fred Wilson “get it” without even breaking a sweat. Well, at least he makes it look easy. That being said USV obviously cares very much about the success of Tumblr – I just believe it is a different type of success then what the core users want. Investors, by and large, want to see growth and eventually profitability while core users want stability and for things to work better and better over time for them.

In early 2010 USV reupped their bet on Tumblr by “doubling down” on them. They’ve put a cool $10m into Tumblr alone. Wilson, in his post in April 2010 about how Tumblr had gotten to 1bn pageviews per month, wrote a very short reason why they’ve made that bet.

“There are some lessons here. First, make your software super easy to use. Second, you don’t need hundreds of employees to build a big time web service. You can keep it lean and scale if you have the right team. That’s how Tumblr got to a billion page views and we just made a bet that they will be able to take that number a lot higher.“

Emphasis mine. USV thinks that Tumblr can increase the number of pageviews from 1bn per month to, well, a lot more. And they think that will help their investment. They don’t care, too much, about how the service gets there just that they increase that number dramatically and – I can only assume – get a much larger round of financing or exit.

I’m not saying that USV doesn’t care if Tumblr gets their downtime in check. You can’t serve 1bn+ pageviews per month if you’re down. I think USV cares very much about the stability of the Tumblr platform – I just think they are focused on the wrong thing which could end up trickling down to the Tumblr team. If the Tumblr team is focused on metrics they will end up losing what made Tumblr’s team so great to begin with – the passion for making something great, simple, and different from everything else out there.

I could be dead wrong. Perhaps the team at Tumblr is focused on exactly that and that the dreams of the investors don’t trickle down too far. I hope USV (and the rest of the investors in Tumblr) understand very well how to stay out of the hair of the core team so that they can continue to do what they are great at. But there must be some reason by Marco Arment (one of the 2-man-team that made Tumblr great to begin with) left to do his own thing and continuously touts that he doesn’t want to take investment for Instapaper. Is he jaded? Has the Tumblr team “sold out”? We’ll see.

Oh, and I’m not picking on Fred Wilson either. I would point to other Tumblr investors that have commented about the growth of Tumblr, like Bijan Sabet, but he powers his blog with Tumblr which means his search simply doesn’t work. Maybe he’s hoping that the millions of dollars that his company Spark Capital has invested in Tumblr will fix that?

Again, I hope I’m wrong and I want Tumblr to succeed. I love the service and would pay money to keep it up and stable. Lets hope someday they give all of us the opportunity to do just that.

On Twitter they only have 40,000 followers and as far as I know they have millions of users. So obviously most of their community has no idea what is going on.

As someone who has been in this type of situation with Viddler (though our longest period of downtime has only been a few hours in 5 years) I can sympathize with the amount of effort that is going on behind the scenes by their entire team. However, let this be a lesson for the rest of us in what not to do.

Update: And just like that they’ve sent out a second tweet. With their own Tumblr-powered blog being down I suppose their options were limited in how they can communicate with their community.

Update again: Tumblr has now published a post-mordem report on what happened, how they’re dealing with it, and how they hope it doesn’t happen again. No new information though the words do seem sincere.

I’m sure Tumblr, and everyone that was watching this all happen, learned a lot of lessons during this event.

Nothing is stated on their Twitter account. Their staff blog is silent (yet not down). Of course, neither is Marco Arment’s Tumblog (he’s a staff member). And yet my dashboard is down, my Tumblog is down, and so are many others.

The message appears to suggest that they know about this downtime. But I don’t think so, because they didn’t let any of us users know about it and they usually do an incredible job with this type of thing.

The fact that the staff blog is up, Marco’s blog is up, and pretty much everything else is down (even photo slideshows), suggests to me that they’ve had some sort of outage that only affects some users and not all users. Perhaps they separate out VIP users? That’d make sense.

Hope Tumblr comes back up soon.

Update: As of 7:49a the error message on the dashboard has changed to this:

It seems like an automatically generated message from Squid. Squid, I believe, actually will notify the technical team at Tumblr with an email, SMS message, or whatever they have set up to notify them when an error such as this occurs.

Update again: As of 8:11a EST it is back. We’ll see if anything is said as to why. But, I also saw a lot of Tumblogs that mentioned the downtime and some funny error messages they were getting. So maybe there was some planned downtime?

Over the past few days the number of Twitter statuses popping up from those I follow on Twitter have been scarce.  At first I thought it was because I was beta testing the latest version of Hahlo.  Then, I thought it was a problem with the API and so Twitterrific would also have the issue.  But, I’ve finally taken a moment to look into the problem and I am, indeed, missing the majority of status updates on Twitter.

Recently on Twitter

The image here (pictured) is what I see on my Twitter profile page. You are looking at roughly 12 hours of Twitter activity from, what should be, 43 people on Twitter. However, you’re only seeing activity from four people. This should mean that these four people are the only people that have updated Twitter in the last 12 hours.

But its way off. Turns out nearly every person I follow has updated their Twitter status within the last 12 hours. If you poke around their profile pages, you’ll see that’s true.

The reason I’m writing about this, is because I have yet to see anything on the Twitter blog about this, and I’m certain that I can’t be the only one affected by this. I’ll try my best to make sure that the team at Twitter sees this blog post.

Are you experiencing the same thing?

Update 11:43am EST: I submitted a support request to Twitter in hopes of making them aware of the issue.

Update 8:45am EST on April 21: Still no response from Twitter on their blog. I’m really not sure why they’ve chosen to go silent about this issue, but it is the direct opposite of a good example. Dave Winer is also very surprised at the silence. Ev, Biz, Alex, dudes?

Update: 12:57pm EST on April 21: Twitter has now posted an update to their blog about the issues. Thanks team Twitter for keeping us up-to-date.

Update: 1:05pm EST on April 21:

How to fix your Twitter cache

It is really simple. Follow someone you are not already following. And then un-follow them. Refresh your Twitter page and you’re all set. Your cache has now been restored.

Note: As several people have pointed out, and as I have now discovered myself, this fix only works for “past tweets” that you missed. It does not allow new ones to come through.

Update: 3:25pm EST on April 21: So this post has been linked to from Techmeme, my comment on the Twitter blog, Mashable, and elsewhere. Thanks for all of the links.

Update: 4:15pm EST on April 21: This post has been dugg. Digg it up, if you’d like.

“37signals responds to downtime, perfectly. They start with an explanation of what happened, then apologize with the promise to compensate where warranted, and assure it won’t happen again, all with human feeling. Learn.” — (view bookmark | view their post)

Pulling this off is no easy task – though for a remarkably customer service conscience group like 37signals perhaps this comes pretty naturally. I wanted to take a second to show some bad examples of this type of response, so that you can see the contrast (and I’m sure I could be one of these examples if I was harder on myself).

Recently Flickr had some downtime that they knew they were going to have so they gave fair warning about it. This is a good thing. However, their maintenance took longer than they thought it would, and I think they might have stepped over the “snarky remark” edge just slightly. Just so we’re all clear, I love Flickr. I’ve met some of their staff members and each of them are good people. Here is a snippet from their downtime notice post.

“Do you remember when we said we were almost back online? Well, that time we were joking, but this time is for real!”

Personally I think they could have skipped the “every few hours” approach to updating and just waited until the service was updated to bring the community up-to-speed (more on this below). Snarky remarks like the above don’t help too much. How can this be avoided though? You don’t want to be completely unhuman. Let’s look at how 37signals brought the human-feeling into their post, with this line.

“Again, we’re truly sorry for this interruption. This is not how Fridays are supposed to be.”

During their downtime they also updated their users as best they could (this particular situation was relatively out-of-their hands) and while they injected some heartfelt messages into those updates, I think they could have saved that for this post.

Another bad example would be to remain silent and have your service degrade, well, not so gracefully. Blogger recently had some outage and their users just saw a weird message and there was no updates from the Blogger staff. Silence isn’t a good tactic at all.

Points to remember

Based on the good example of 37signals and the bad examples above, I think that we should all strive to do the following when web services go down – and I’ve ordered these by importance (in my opinion).

• Degrade gracefully. When downtime occurs, forward to some sort of friendly message that is easily updatable by staff members to let the community know what is going on.
• Keep explanations short and simple. Don’t update every 5-seconds (especially if you have nothing to report), and don’t be long winded. Sometimes “we’re working on it” is sufficient. Oh, and each update should have a timestamp.
• Don’t give false expectations. I’ve learned this the hard way. Even if your engineers tell you that it will take an hour, there is no need to say that publicly. Keep the “we’re close” messages to a minimum too.
• Be human. Try your best to explain the situation in human terms and be warm.

Once the service is back up and running, and a longer explanation is warranted, you can look no further than 37signals post for inspiration.

One thing we can’t see is whether or not 37signals did any contacting of their users behind the scenes. Since their product is a pay-for service, they could have very well personally contacted some of their larger accounts to let them know what is going on. Or, after they were back up, they could have reimbursed them beyond the offer they made publicly. Things like this go a very long way.

Please notice that I believe this task to be extremely hard to pull off well and that I think both Flickr and Blogger are great services.

I’m hoping that I can take all of these points and learn from them the next time we have any troubles at Viddler. In the past we’ve handled these situations fairly well, but I know we can improve a lot by learning from others good and bad examples.

Thoughts?

Addendum: It appears that I am not the only one that thinks 37signals did a great job. Not only do they have numerous comments on the post, but Dan Benjamin also thought so.

Side note: The team at Media Temple called me the very moment they knew something was up. Talk about service, these guys are great.

From what I’ve been able to determine, this effected all of Media Temple (mt) and DreamHost, as well as several smaller hosts (obviously those that resell for the above).

It is Friday night anyways. Go out, grab a beer, and watch some fireworks. That is what I am going to do.

Update: – We’re up and running.

[tags]9rules, media temple, mt, dreamhost, hosting, downtime, power outage[/tags]