Story 1, Only backups?
We had a client where we sold them a new shiny server solution. I don't remember the spec of the server but they had SBS 2011 and were using Symantec System Recovery for backups. Not too shabby so far. Unfortunately, however, they were sold an iOmega ix2 home NAS for their backups! That's right, the salesperson obviously tried to find something cheap and sold the customer a NAS designed for home use!
You may be thinking, so what? I've had an iOmega for years and it hasn't let me down. That's fine for you maybe, but this is for a backup device! It's writing gigs and gigs of data every night and a full backup every week. These boxes are designed for one off storage that rarely changes, not to have the data re-written every month!
Anyway, back on topic. The backups started failing on the NAS. They were getting stuck and wern't finishing. Engineer goes to site for the monthly checkup. They're advised that the backups are failing and to have a look at the NAS. Engineer looks at the NAS, decides to do a factory reset and start again. Fair enough, that's something to try at least, and they have offsite copies on USB drives so we're all covered right?
Few days later, everything seems to be going well. Backups are working again, looks like the reset did the trick. Then we get a call from the customer. One of the users, (one of the important ones) reports that one of their network drives is disconnected. Engineer investigates, looks at the network drive and determines it's pointing to the NAS box.
Engineer calls the guy who was at looking at the NAS to find out what happened. The conversion went something like this:
Eng1 - Hey, what's happening with the NAS for customer xyz?
Eng2 - The backups were failing, I had to perform a factory reset.
Eng1 - Did you check there was any data on there before you reset it?
Eng2 - It's only for backups, there's no user data on there.
Eng1 - But did YOU check before you reset it that there was no data on there?
Eng2 - No, there should only be backups on there.
The penny dropped at that point. It was clear there had been some data on that NAS, and it had been wiped. However, no one yet knew the scale of this loss.
The director took control of the case. He emailed the user explaining what had happened, and tried to shift some blame onto the contractor who was visiting the site a few months prior, as he suspected they had setup the network drive for the user.
The user was unimpressed to say the least. All they wanted was the data back. The director contacted the manufacturer and asked for advice on how to perform data recovery on the NAS. Hoping that there may be an slim chance of retrieving some of the data.
The data recovery was run, but it transpired that the user data consisted of huge video files, several gigs in size. This was a massive problem in itself. Big files are much more vulnerable to be damaged after a format than smaller ones. Also the backups had been running again which would have overwritten a lot of the data as well. 100% data loss in this case, all the files that were recovered came back corrupt.
Moral of the story? Never format something without checking the data first. You never know what someone has left on there. Even if they shouldn't have put it there, if the data belongs to someone important then you could be held accountable!
Story 2, Picture time!
This story has a happier ending. One of our clients was a private school. They had an extremely old server, with limited storage capacity. The same salesman had a lightbulb moment, server is full? Let's sell them a NAS and they can put some data on there. Much cheaper than buying server hard drives!
So that was that. They put all of their school photos on this NAS, yep another iOmega ix2. All was going well until one day, nobody could access the NAS. Phone call came in, can't access the X drive. Okay, checked the server, the X drive is stored on the NAS. I told them to reboot the NAS. Nothing happened. Can't ping it, can't browse to it, can't access over HTTP. Not good so far!
So we collect the NAS and bring it back to our offices. Plug it in and run the iOmega utility. Can see the NAS but that's about it, can't browse to it or see the data etc. Called iOmega support, that was hopeless, can't remember what their repsonse was but it wasn't helpful! Customer wants their data back badly by this point.
So we take out one of the hard drives, and try to power it up. If one of them has failed then the device should work fine with the other. Nope, same result, both drives are unusable. Okay let's plug one of the drives into a PC. Okay, partitions are showing as RAW. They must be formatted in Linux and Windows doesn't understand the file system.
Wait, we have a linux machine that someone built for testing. Let's hook up the hard drive to that. Boot up the machine, boom, the drive won't mount. It's part of a mirror and it's looking for it's other half. Okay, let's plug in the other drive. Boom, machine won't boot. The other drive is completely shot and the PC can't even see it. Fortunately after some searching, we found a command that allows you to force linux to mount the drive, even though the second one is missing.
I think the command was "mdadm
--assemble /dev/md1 /dev/sdm1 –force"
So I looked in the file browser, wo ho! All the data is there, copied it to a USB drive and took it back to the school. This was a lucky one, and just shows how domestic NAS's in a business enviroment are false economy. We effectively lost a whole days work (anyone in IT will know that a day rate for an engineer is not cheap) and of course, we sold the customer the NAS, and it was covered in the support contract. So we couldn't exactly charge them for the hours we put in recoving the data.
Moral of the story here? If you're going to use a NAS for storage, always have a backup. Also, if it's in a production enviroment, BUY A BUSINESS PRODUCT. QNap, Synology, Netgear all have business grade NAS boxes that can go for years without so much as a drive failure!
No comments:
Post a Comment