Our current network evolved from massive mainframes to early Windows machines, Unix boxes, dumb terminals, NT and now thousands of smart clients and rooms of powerful dedicated servers. We monitor every facet of their operation. Yet a phantom nearly took us down.
The onset was not ominous. A network printer overran its buffer and jobs stopped printing. We speculated it was a hardware failure, swapped out the old printer, configured the new one, rebooted the device, and then it was business as usual.
Then it happened again on another printer. Soon a master console winked out. Just for a moment, but it was definitely gone. Users reported sporadic data header corruption. Strange incidents became more frequent with no set pattern.
A computer virus on a server? A network worm on a shared drive? Not likely. We use a multilevel security approach guarding against blended threats.
NetOps caught some of the signals. They were random but definitely originated from inside the perimeter. We had a phantom server.
TIP: Document what comes in and what goes out
Some of our very old servers had been in operation for many years. They plodded along doing the mundane maintenance tasks for which they were originally assigned. New hardware came and went. Many of the older devices could no longer be traced except by anecdotal memory.
TIP: Scan your entire IP range periodically
Most modern operating systems monitor a wide range of activity. But beware! They cannot detect what they cannot see. Very old hardware can lie below the radar. Our phantom server with no name was undetectable by normal means.
Program consoles can display their clients, but not every machine runs every service. We suddenly realized that we did not have a single, simple comprehensive method to detect everything that was out there on our network, no matter what it was.
The humble ping command came to the rescue. It detects connectivity and lets you capture IP addresses and machine names as it traverses your network.
TIP: Use an informative standardized naming convention
Cute server names may be amusing, but cute names can be problematic when trying to locate a specific item in a hurry. Encoding the location and functionality into the device name saves both time and aggravation. You should label every server with a tag bearing its name and IP address in a conspicuous location when it enters service. This technique may be low-tech, but it saves valuable time when trying to find a box amongst its colleagues.
TIP: Read your logs
A log file can be quite informative. If the data content is overwhelming, then export it to a spreadsheet or database and view it in sorted order. Old machines pressed into service a long time ago probably have nonstandard names.
A ping, a log file and a sort revealed our phantom server. The network segment suggested its approximate location in the building. Systems people didn't remember the precise location, but lab people remembered some equipment being moved, and with their help we finally found it.
It was sequestered in a closet barricaded by buckets of spare parts ,and it ran a dialect of SCO Unix that was several operating system versions behind. No one knew the device was there. It had been moved when a former manager renovated the UPS, and he retired before taking it out of service. Staff from that time were long since gone.
The device was a clinical lab server monitoring communication with data acquisition equipment. It ran silently performing its routine task and periodically issued a LAM (Look-At-ME!) alert when it was in distress. The LAM messages passed through the network unheeded until they collided with other server message blocks. These collisions appeared as system aberrations.
The lessons learned? Know your network. Document your resources. Scan your range.
Sometimes a little low-tech common sense will resolve an unusual ghost in your network.
Dr. Lee Ratzan is a systems analyst at a health care agency in New Jersey and teaches technology at Rutgers University. Contact him at lratzan@scils.rutgers.edu