Bonobo Conspiracy Web comic #685: As if millions of horny women-of-a-certain-age cried out and were silenced

Okay, guys, this isn't rocket science (and I've done rocket science, and that isn't rocket science either, but that's another story). You have a business that basically consists of a Web site. If customers visit the Web site you sell ad impressions and fucking user pic expansions and all the other things your business model is based on. If customers can't visit your Web site, you lose. So you put your servers in a professionally-run data centre, and that is supposed to mean, among other things, that your servers get power. All the time. No whining and no excuses.

Even semi-serious amateur Web sites (like, for instance, Bonobo Conspiracy) have service level agreements with their upstream providers. In BoCon's case, the hosting company guarantees 99.9% uptime per calendar month, or the month is free. If it's worth it for me to pay a little extra to get that kind of insurance, it makes sense for an actual for-profit business whose Web site is absolutely mission-critical, to get something similar. Real, serious businesses, have battery backups that last long enough to spin up their dedicated diesel generators, as well as hot standby servers at remote sites, and service level agreements with real teeth that don't admit to whining and exceptions, and the bottom line is that things like power failures Don't Happen.

But as I write this on Tuesday about 8:50pm Eastern, Livejournal's essential mission-critical site, the Web site which is the entirety of their business, is down. Supposedly because of a power failure in their data centre. Either they didn't pay for reliable power, or the people they paid, let them down in a many thousands of dollars way. It's been down for a number of hours. Right when approximately 99.93% of Livejournal users (rough estimate), are just finishing reading Harry Potter and the Deathly Hallows and ready to log in and talk about it. Right when a whole lot of people who shelled out for permanent accounts, hoping to give the company the chance to buy back the loyalty lost in the ~~Strikethrough 07~~ debacle, are just getting their credit card bills and having second thoughts about whether it was the right decision. Livejournal couldn't have picked a worse time to go off the air.

I notice something else interesting when I look at Six Apart's status update page. See if you notice it too, in this screen shot:

It's all about TypePad. TypePad gets the detailed status updates, with other Six Apart projects, including Livejournal, relegated to single-sentence summaries below the fold and not shown on this screenshot. TypePad gets top billing in the handy status box at the upper left - where Livejournal is not even mentioned. It's not just cosmetic, either. TypePad also got priority for being actually brought back up, apparently coming at least sort of online several hours ago, while Livejournal and Vox are still down. TypeKey (according to the note at the bottom of the page, not shown in screenshot) was up as of 1:55pm; I think that means it wasn't down in the first place, not that it came up first. The bottom line: Livejournal isn't Six Apart's priority. No wonder it fell and hasn't gotten up.

Comments

no-richard from 75.116.83.170 at Thu, 26 Jul 2007 07:53:43 +0000:
Jason: TypePad and many external blogs rely on TypeKey for commenter authentication. A TypeKey outage could disrupt activity on weblogs in eastern Siberia; reducing external blog disruptions was the highest priority.

Matt: I know, the cheap customer will decide that the vendor doesn't really regard him highly, which is true enough, and then take his cash elsewhere. From the vendor's POV it's just a self-fulfilling prophecy. Having supported products that were aimed at some VERY big (as in, multi-Billion dollar) corporations, I remember that when they told us to jump through flaming hoops, we poured the fuel and lit it ourselves. The $200 customer didn't butter our bread; the $15,000 customer did.

acelightning from 68.82.9.59 at Thu, 26 Jul 2007 08:29:34 +0000:
http://valleywag.com/tech/breakdowns/365-mains-credibility-outage-282257.php

according to that theory, the colo facility used a really cockamamie backup power system that relied on flywheels, rather than batteries, to keep the equipment running until the diesel generators could start. there were several very short outages in quick succession, long enough to use some of the momentum from the flywheels, but not long enough to trigger the auto-startup for the generators. when the power went down for the long term, there wasn't enough oomph left in the flywheels to keep everything going until the generators started. hell, the power backup i have for my little home LAN is better-designed than that! (i copied, as best i could on a small scale, the power backup systems from the radio stations where i used to work.)

This form is for posting public comments to be read by other people who visit this Web site. If you have a software support question, or other material directed to the page author instead of to the general public, please send email instead.

All the data you enter, and your IP address, will be saved and displayed. Don't enter secret information. HTML is not accepted; it will be displayed as plain text. Your comment will only be added if you enter valid data in all required fields; if it isn't, use the back button and try again.

I, and I alone, reserve the right to remove postings for any reason.

#685: As if millions of horny women-of-a-certain-age cried out and were silenced

Comments

Add Comment