i bet this guy needed a new pair of shorts when he realized what he'd done

http://www.foxnews.com/story/0,2933,259842,00.html <-- i know, i know, but it's an AP article

Quote :

"Computer Tech Accidentally Erases Info on Alaska's $38 Billion Oil Fund Tuesday, March 20, 2007 JUNEAU, Alaska — Perhaps you know that sinking feeling when a single keystroke accidentally destroy hours of work. Now imagine wiping out a disc drive containing an account worth $38 billion. A computer technician at the Alaska Department of Revenue deleted applicant information for an oil-funded sales account — one of state residents' biggest perks. While reformatting the disk drive during a routine maintenance check, the technician mistakenly reformatted the backup drive as well and, suddenly, all the data disappeared. A third line of defense — backup tapes that are updated nightly — were unreadable. "Nobody panicked, but we instantly went into planning for the worst-case scenario," said Permanent Fund Dividend Division Director Amy Skow, about the computer foul-up in July that ended up costing the department more than $200,000. Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence. The only backup was the paperwork itself — stored in more than 300 cardboard boxes. "We had to bring that paper back to the scanning room, and send it through again, and quality control it, and then you have to have a way to link that paper to that person's file," said Skow. Staff working overtime and weekends re-entered the lost data into the system by the end of August. Last October and November, they met their obligation to the public and a majority of the estimated 600,000 payments for last year's $1,106.96 individual dividends went out on schedule. Former Revenue Commissioner Bill Corbus said no one was blamed in the incident. "Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," said Corbus. According to department staff, they now have a proven and regularly tested backup and restore procedure."

this is exactly why you verify your backups, retire tapes regularly, and above all, don't be a dumbass

3/20/2007 10:44:54 AM

ouch. sucks for him

3/20/2007 10:47:28 AM

Sounds like their IT dept needs an overhaul.

3/20/2007 10:47:43 AM

this is exactly why you verify your backups, retire tapes regularly, and above all, don't be a dumbass

... speaking of which... I'm going to go make sure my lab's backup tapes are readable. The tapes retired after 20 writes, but my thesis research lives on there in case our raid 6 user storage array fails.

[Edited on March 20, 2007 at 11:10 AM. Reason : need to check my backups]

3/20/2007 11:08:49 AM

wow

oh i bet there was a witch hunt
definitely

3/20/2007 11:31:12 AM

uh apparently they've never heard of data recovery in alaska.

it would have take a lab a weekend to recover a formatted drive.

3/20/2007 11:37:24 AM

Noen nailed this one.



but would they be able to send it to data recovery if the information was super sensitive?

3/20/2007 12:19:36 PM

daily back ups
off site rotations
and 4 day retention cycles

3/20/2007 12:23:40 PM

Wow. They kept all of two backups for a computer that was essentially worth 38 billion. How much does a tape for a tape drive/blank DVD cost?


[Edited on March 20, 2007 at 12:50 PM. Reason : -]

3/20/2007 12:49:22 PM

^ They had tape backup, but their backup tapes were corrupted. Read the whole thing before posting.

3/20/2007 12:53:58 PM

I did read it, the article specifically mentions two backups.

Backup one: on a separate disk
Backup two: on one tape

Potential backups:

One or more physically separate tapes
One or more DVDs.

[Edited on March 20, 2007 at 1:06 PM. Reason : -]

3/20/2007 12:59:50 PM

and people still use tapes..
fuck that.
optical media >>>> tape

3/20/2007 1:00:53 PM

Quote :

"uh apparently they've never heard of data recovery in alaska. it would have take a lab a weekend to recover a formatted drive."

I bet they considered that but didn't want to risk reading back corrupted data that could screw up someone's account.

3/20/2007 1:44:39 PM

Quote :

"There was no witch hunt,"

hahahahaha yeah right.... Im sure there were meetings, beatings, conference calls, emails, etc.

3/20/2007 1:54:28 PM

^ Alaskans are very easy going. You know marijuana is decriminalized there?

3/20/2007 1:56:51 PM

Quote :

"I bet they considered that but didn't want to risk reading back corrupted data that could screw up someone's account."

from a format, there's almost 0 risk of this happening, and it can be verified anyway. hell there's offtheshelf tools that will bring a drive back from a format.

3/20/2007 1:57:24 PM

Quote :

"and people still use tapes.. fuck that. optical media >>>> tape"

yea, because its easy to have TBs worth of optical media storage attached to your network.

3/20/2007 2:34:40 PM

hahah GG

3/20/2007 2:39:26 PM

Quote :

"uh apparently they've never heard of data recovery in alaska. it would have take a lab a weekend to recover a formatted drive."

That was my first thought too. I've used a [1.5MB] $30 program to fully recover data from a formatted drive. Either they're a bunch of dumbasses up there or there are details to this story we are missing.

And I think the only reason this made the news is because of the $38 billion number in the title (and everybody thinking the data was worth that much or something).

3/20/2007 3:36:42 PM

the fact that this happened in the first place proves that they are indeed a bunch of dumbasses up there

3/20/2007 4:07:52 PM

Quote :

"and people still use tapes.. fuck that. optical media >>>> tape"

Find me optical media where I can dump 400 GB at a time and I'll sign on.

3/20/2007 4:16:24 PM

Quote :

"Alaska Department of Revenue"

Sorry folks, this is a state agency. Throw most of your assumptions out the window. Mix minimal funding, a large % of underqualified staff, and bureaucracy and see what happens. Enough to get the job done as needed, but not really to go the extra mile to do things right.

Give them props for openly admitting fault when it went public. And, if nothing else, this may cause them to be more rigorous with their internal auditing to make sure stuff like this doesn't happen again. (Backup drive shouldn't have been accessible THAT easily by the same technician).

3/20/2007 5:22:37 PM

Quote :

"from a format, there's almost 0 risk of this happening, and it can be verified anyway. hell there's offtheshelf tools that will bring a drive back from a format. "

What if they did it with a zero-wipe? Or with a multipass random data wipe?

It does seem odd they wouldn't have tried this though, so i'm assuming there must be a reason.

3/20/2007 6:45:47 PM

If I dont get my PFD Ima be PISSED AS FUCK...I need that $1g supplement.

3/20/2007 10:05:23 PM

^^Unless you are physically destroying the drive by DOD standards, OR encrypting the drive before formatting, it can be recovered. It's pretty insane what can be pulled from hard drives.

3/20/2007 11:24:56 PM