Comments on: Learning how to respond to downtime

By: Hi, I’m Colin. » Blog Archive » Twitter is definitely down, sorta

Mon, 21 Apr 2008 12:50:38 +0000

[...] not sure why they’ve chosen to go silent about this issue, but it is the direct opposite of a good example. Dave Winer is also very surprised at the silence. Ev, Biz, Alex, [...]

By: Diana LeRoi-Schmidt

Diana LeRoi-Schmidt — Tue, 22 Jan 2008 15:55:42 +0000

I think 37 Signals did a great job with damage control and apologies for the outage on their system. My employer just recently started using Basecamp, and I didn’t hear any freaking out from management offices, they all seemed to calmly wait for the system to come back up.

I feel like sites when they’re going to either have a scheduled down time, or are experiencing an unexpected outage, have some sort of message on their website, and if it’s unexpected, give regular updates.

I’m much more understanding and willing to roll with something when there is an explanation. Even if it’s the fact that the company is dealing with the fact that they didn’t have the resources to support the traffic they received. Ok, it can be remedied in the future.

Twitter in particular, I think is walking the fine line of upside down birdies a few too often with not a whole lot of reasons behind it, and then followed by a scheduled down time.

By: albert

albert — Mon, 21 Jan 2008 23:19:09 +0000

the most annoying part of the recent flickr outage (and it seems to be running slow now too which is what happened right before the outage) was that they didn’t announce it beforehand. i believe it was due to some scheduled upgrade/maintenance. whammo, no notice. it crawled to a halt and then the “flickr is having a massage” message.

By: Colin Devroe

Colin Devroe — Mon, 21 Jan 2008 16:04:45 +0000

Chris Masto: Yeah, I definitely overlooked using that as a recent example, and a great example it is. When you are dealing with people’s lively hood, or money, there is no room to joke – but there is room for being human and compassionate.

By: Colin Devroe

Colin Devroe — Mon, 21 Jan 2008 16:02:42 +0000

Owen: I agree with you in some regard. I think some services do not need to notify the users until they try to access it, while others definitely should.

For example: When a service is used to “service others”, they definitely should. Say for example credit card authentication services. Normally the customers, or clients, of these companies don’t use their services online but their customer’s customers do. In this case, the companies should definitely contact them directly to give them a heads up.

Perhaps in Joyent’s case they should have contacted you. In 37signals case – maybe they should only do so as a courtesy to their largest accounts (perhaps all pay accounts even). Flickr could contact all of their Pro users, or – all of their Pro users that have more than 100,000 views on their photos. Maybe Viddler, now that we offer revenue sharing, should contact all users who have revenue sharing turned on.

There are many ways each service could use to determine how and when they should notify their users of downtime – but I think that the first priority should be to degrade gracefully which is used to notify those trying to access the service what is going on.

By: owen

owen — Mon, 21 Jan 2008 15:02:19 +0000

Joyent’s Strongspace and Bingodisk services have been down outside of expectations for a full week until today. I did not even realize that I was having a problem with backups to their system until Thursday when I tried to access it, couldn’t, and then checked their status log.

Moral: Sending email to affected customers during downtime is absolutely essential because not everyone cares to read your site’s status log.

I have yet to hear of compensation to users for this downtime. I expect that for me it’ll be nothing, since I have one of their “lifetime” accounts and have the luxury of being screwed in every respect in regard to service.

By: Chris

Chris — Mon, 21 Jan 2008 14:56:35 +0000

It’s not exactly “downtime”, but certainly related to the idea of how to tell your customers about something bad that happened. Can’t leave out the recent DreamHost debacle. http://blog.dreamhost.com/2008/01/15/um-whoops/ People reacted very strongly to the “joking” tone of that post compared to the severity of a $7.5 million billing error.