That could have been avoided -- should have been, in fact. But once it looked like there was a security breach, the OIT did the right thing by going public with the news.
Now it's time to do something else right: Document everything.
Yes, that does sound pretty dull. But it would have avoided lots of the wrong kind of excitement. For those who haven't followed the story, on Feb. 15, the OIT announced that the Cain & Abel password-recovery tool had been discovered on a server during a security sweep. Cain & Abel has been used by attackers in the past, and OIT officials feared the worst. They notified the public, warned potential victims, called in the FBI and launched an investigation.
They also reportedly placed OIT employee Douglas A. Oliver on leave. He later told Computerworld that he'd installed Cain & Abel as part of a security test. Oliver said last week that he has been cleared to return to work April 25.
According to Oliver, in early February, OIT security testers using Cain & Abel and other tools discovered a slew of problems on state servers: DNS cache poisoning, unencrypted administrative password files, still-active accounts for ex-employees and a SQL Slammer worm infestation.
On Feb. 10, OIT workers began to patch all the state's SQL Server installations to block SQL Slammer. On Feb. 15, the state announced the Cain & Abel "breach," and Oliver was put on paid leave two days later.
Why was Oliver fingered? Probably because he installed Cain & Abel under his security credentials, just as he was supposed to. But why did the OIT identify Cain & Abel as the big problem rather than SQL Slammer, which posed a more direct threat?
It's hard to say for sure, because OIT officials aren't talking. It might have been, say, a clever ruse to mislead potential attackers. More likely, executive-level types were just confused over what kind of malware was involved, who might have put it on the state's systems, and how and where the real risks were.
No matter. Confused or not, those officials didn't try to cover up the problem. They took quick action and risked embarrassment by going public. Good for them.
But now that the security mess appears to be cleaned up, there's something they should do to prevent future, um, excitement.
They should make sure that everything on production servers is documented. That all changes are logged. And that those logs are kept secure, so they'll actually be useful next time there's something to investigate.
Full documentation of what went on the servers, and when, would have cleared Oliver by confirming when and why he installed Cain & Abel. Inventories would have allowed OIT staffers to identify anything else on the servers that wasn't an authorized installation. Checksums would have helped spot modules that were infected by malware or replaced by intruders.
And full documentation doesn't just help against old-style threats like SQL Slammer. Rootkits, the hot new problem, are designed to be hard to spot. Knowing what an uninfected system is supposed to look like, down to the details, gives IT people a fighting chance at catching and dealing with a rootkit.
Sure, keeping tight control over what's on servers will be more work. But the OIT already knows that procedures have to be tightened up -- remember those unencrypted passwords and ex-employee accounts? With today's security threats, such safeguards are no longer optional.
Think of them as IT's version of Sarbanes-Oxley, only a lot more useful than the real Sarb-Ox. Those controls are the price the OIT -- and all IT people -- will have to pay if we really want to ever get things back to normal again.
Frank Hayes, Computerworld's senior news columnist, has covered IT for more than 20 years. Contact him at frank_hayes@computerworld.com.