Server upgrade with no upside

17.10.2006
The problem was obvious: Data storage was maxed out, the network crawled, and half the time our backups were corrupt. The solution was simple, too: Upgrade the servers and increase network function -- without a loss of data. But sometimes "obvious" and "simple" add up to confusion and catastrophe.

I was head of IT for a small company in Tennessee, and I knew how business worked. So first I made a needs assessment, requested recommendations from experts, gathered proposals, reviewed them, and passed them on to my boss, the owner/CFO and Solitary-Decision-Maker (let's call him Keith). He told me to start the upgrade immediately while he looked over the fine print.

Fine print, indeed! While Keith stalled, started, stopped, changed his mind, and started over, nearly a year passed. Proposals went past their contract dates and had to be rebid. Vendors were frustrated, and I was stuck patching the outdated, jury-rigged system while I waited for it to self-destruct, taking our data with it.

One morning, when I came in to find smoke drifting out of a server (fortunately it was just a bad fan), I told Keith if he didn't make a decision, I was out of there. He paled and made the call. Unfortunately, it was the wrong call, at least in my opinion; he put the upgrade in the hands of a company (let's call it "Dubious Network Consulting") that I had warned him about. But he was the boss.

Dubious sent Joe -- a "professional" making ten times my salary. Joe was a nice guy, but I got the impression he was a bit out of his depth. I was about to leave for a one-week vacation, so I made him promise that he would not change anything until I got back and he could bring me an external backup that I could check to ensure our data was intact.

I returned from a week in Mexico to discover, to my horror, that Joe had already upgraded several terminals to prepare them for the new server and its new OS. Next thing I knew, a third of our workstations were inoperable, and (surprise!) there was no backup. Our salespeople had no data. Also no calendar, no contacts, no e-mail, no access to documents. All contracts written the previous week were gone. Joe even managed to delete my profile and all my information from the server.