We sold a customer an entry level server (ML 150 G5 I think it was) to run SBS 2008 and Freight software with only 4GB of ram! There was no proper hardware RAID so they ended up with 2 SATA drives in RAID-1.
It was problematic from the start. First off, SBS 2008 runs Exchange and SQL out of the box. Both memory hungry and hard drive thrashers. As soon as they tried to introduce the freight software, which is also SQL based, the whole thing just ground to a halt.
We then had to donate a server to them to run the freight software as we had told them (or salesman rather) that the server would be powerful enough for their needs.
The customer still complained that their systems were very slow, in particular the internet. Then the salesman decided to sell them new computers to fix the problem. Of course, it made no difference as it was down to the server being so overworked and actually being too slow to respond to DNS queries!
I was a first level engineer at the time. So in my wisdom, I added some extra DNS servers into DHCP so that the clients could run their DNS queries against the internet as well as the server, this sped up the internet nicely. Only problem was they would randomly lose connection to the server as sometimes it would resolve the internal name of the server against the wrong DNS server.
So that was scrapped and it was back to forwarders. I did manage to speed things up though, by limited the SQL memory so the windows internal database basically had no power, and also limited the memory exchange could steal.
We did eventually put more RAM in the server (took a couple of visits as they forgot to bring the RAM the first time!). The server was better after that, but it was still never fit for purpose.
After 2-3 years, we decided to sell them some shiny new Symantec System Recovery software and a NAS to replace their failing tape drive backup solution. The engineer (not me!)installed the NAS and got it setup. Then began installing symantec, but it needs a reboot to finish off the installation.
I think the reboot did eventually happen a week later, but the software was never setup. Then the emails stopped working. I discovered that the exchange SMTP queue file was corrupt, so I recreated it. Then it was fine for the rest of the day.
Then at about 5pm. The server goes offline. Customer reboots and it starts to do a diskcheck, never a good sign. Rebooted it again to bypass the diskcheck and it refused to come back on. It went into a reboot loop. Engineer goes to site and collects the server as nothing more could be done.
So we get the server, no decent backup and not booting. Plugged the hard drives into our test machine to see if they were still usable. Both of them could be read, however, one of the drives only had data from 2 years ago! So it seems the RAID had stopped working ages ago and the server had been running on one hard drive for years, now it's finally starting to fail.
We were able to get the server booted, but only in Directory Services Restore mode. So Active Directory was corrupt. By this time I was a third level engineer, but I had never had to deal with this kind of problem before. There server had been there for 5 days now. So I rolled up my sleeves, got some Maccy D's breakfast and got to work. I found an article which gave a guide on how to repair the ntds.dit file. To my surprise this actually worked and the server booted. However, exchange was bolloxed, database would not mount so we still weren't in great shape.
We managed to source another hard drive to get the RAID started, so at least we would have a decent hard drive in there. Then with some amazing luck, we found that there was a tape backup of exchange, as they were still changing the tapes and there was a backup a few days before the crash.
So I restored the exchange backup from the tape, and the server was finally working again. I took it back to site, plugged it in and we were back online. Fortunately they had their email cached in outlook, so that filled in the gaps where the backup was a few days behind. Also they had hosted email security which held the emails for 7 days in the cloud and they didn't lose those either (server had been gone for 6 days).
So we just about got away with that, but it was pretty poor on our part. If the backup had been setup within a reasonable time then it would have been much more helpful and wouldn't have taken so long to get the server back. Just shows a decent RAID can make all the difference and a good backup is always essential.
