One million characters wiped

Jibb · 10-07-2006, 07:41 PM

Today I received the premiere issue of Massive Magazine. Here is a portion of a very interesting article. I cant type the entire thing by hand, so this a paraphrase. With a couple minor exceptions, these words are not my own - it is a direct quote:

-------------------------------------------------------------------

It's late at night, a Wednesday between last year's Thanksgiving and Christmas. Everquest 2 has been humming along for just over a year. But tonight, someone logs onto server and loads his character. He notices something strange. A quest is missing from his log. Hmm. Odd...Another player notices a piece of missing equipment...Another player may have a skill gone from the toolbar...The petitions stack up quickly...It is 3 a.m. on what some people will remember as Black Thursday. Within 24 hours, one million virtual lives will nearly be lost.

Ten hours after the first petition, shortly after 1 p.m. on a Thursday afternoon, the call was made that the problem couldnt be fixed while the servers where running. They were all shut down...It was time for a character roll back...

Back then, there were systems for backing up and restoring data, but not as many as there are now. Hartsman says everything is triple redundant these days, but in the early days of EQ2 some of the back-up systems wernt as thorough. In this situation, they had to go to the tapes. Physical magnetic tapes, which are the Absolute Last Resort.

As servers were repaired they were put back online. There was an intricate procedure for wiping a server and then restoring the back-up. According to Hartsman, someone restoring the servers did two steps of the procedure out of order, which ended up wiping the magnetic tapes before they were copied to the servers. The result was a set of empty servers and blank magnetic tapes. Around one million characters were essentially deleted. Gone. Wiped. Poof.

As anyone who heard stories about recovering data from reformatted hard drives knows, data is rarely "completely gone". It's just a matter of how far your willing to go to get it back. Sony was willing to go as far as it took. The company contacted the hardware manufacturers and they sent their best troubleshooter.

While they worked to resolve the problem, the team tried to put up hourly updates to keep the players informed. Hartsman notes that they never said, "We lost a million characters". It was something they wernt going to announce unless and until it was certian they wouldnt be able to get them back.

While the troubleshooter studied the problem, the tech support was working on salvaging whatever details of the characters they could from web data, which tracked many, but not all, character stats. It would have been a painstaking and incomplete process to reconstruct characters that way, and it was likely gaps would have remained. But it was their best hope if nothing else worked.

In the end, the trouble shooter saved the day and the data was recovered. Some servers took substantially longer to go online, but there was no substantial loss of character data, and if the the problem had been diagnosed earlier, there would have been even less data loss.

All acounts were credited with a few free days, and at the time no one was the wiser about how bad it could have been. "Literally there was a cascade of something like four different errors that led to this level of screwed-up-ness", said Hartsman. "I mean, how do you explain this without seeming incompetent? It was a perfect storm of crap".

Cupcake · 10-07-2006, 07:50 PM

Wow, I can't even begin to imagine being in Hartsman's shoes that night. To say he probably needed a stiff drink after that is probably an understatement!

That probably would have been the end of EQ2 right there.

Calthine · 10-07-2006, 08:04 PM

That must have been the 3-day server crash... late 2004 or early 2005. But I never heard of anyone anywhere loosing characters.

Actually, from what I'm seeing around, Massive Magazine might have a teeny bias.

Xede · 10-07-2006, 08:15 PM

i remember that weekend or w/e

befallen was down the longest....ironic thing was when nektulos came up i ended up making a toon there just to pass the time til befallen came back up.

Illuminator · 10-07-2006, 08:17 PM

No biggie, I was still a fucking n00b, one lost treasured item wouldn't have been the end of it all.

Sqee · 10-07-2006, 08:17 PM

Quote:

Originally Posted by Calthine

That must have been the 3-day server crash... late 2004 or early 2005. But I never heard of anyone anywhere loosing characters.

Actually, from what I'm seeing around, Massive Magazine might have a teeny bias.

Nobody remembers all the unannounced down time we had last christmas?

Kinda makes sence actually.

Darammer · 10-07-2006, 08:19 PM

Quote:

Originally Posted by Sqee

Nobody remembers all the unannounced down time we had last christmas?

Kinda makes sence actually.

If it was just last Christmas, I doubt the article would have said "Back then, there were systems for backing up and restoring data, but not as many as there are now. Hartsman says everything is triple redundant these days, but in the early days of EQ2 some of the back-up systems wernt as thorough."

Christmas 2k5 would not be "the early days of EQ2."

Nurta · 10-07-2006, 09:07 PM

I remember that, I was pissed b/c having that day off I wanted to play. That was back when I was like an energizer bunny on crack for EQ2. But, I've been in those situations of restores for my job. It's stressful to say the least but it's just a game imagine when patients' lives are at stake, yeah it's even more stressful. I'm sure others have been in similar situations too. You don't want to be testing your DR during a disaster that's for sure. I'm happy the servers' data is more protected now........haha

Snark · 10-07-2006, 09:20 PM

very cool, thanks for sharing this:smiley-happy:

LFG · 10-07-2006, 09:41 PM

If memory serves me correctly, that was December 15.