Last week, former Internet Security Systems researcher Michael Lynn presented at the Black Hat USA 2005 conference a reliable process that could be used to exploit Cisco routers running the Internetworking Operating System (IOS.)

Even though the exact exploit demonstrated during his presentation was not disclosed, Lynn showed enough details to prove the exploit is real and previous misconceptions that routers and switches are not exploitable are false.

Within days, there were more than a half dozen sites mirroring a copy of Lynn"s presentation detailing the IOS exploit process (see Cisco vulnerability posted to Internet). In addition, all major networking mailing lists, such as NANOG, and many blog sites, such as Schneier on Security by security expert Bruce Schneier, were hot with discussions over such topics as responsible and ethical disclosures, possibly exploits and dooms day speculations. A legal defense fund for Lynn has also been created to assist him with the legal battles.

It"s important to recognize that amid all the noise and arguments over the recent events, the specific vulnerability discussed in the presentation was not new. The flaw was patched by Cisco in April. All vulnerable versions of the IOS have been removed from the Cisco"s Web site. Cisco also allows upgrades even for non-contract customers as long as the call comes through their technical assistance center.

However, it is likely most of the routers on the Internet have not yet upgraded to the latest patched IOS images. In addition, although the new IOS images are no longer vulnerable to the presented exploit, any newly discovered buffer or heap overflow vulnerabilities on the IOS can still be exploited using this same process. Knowing that Cisco"s IOS software has been stolen and has been known to be in the wild, it is reasonable to assume that new vulnerabilities will be found and that worms exploiting the new vulnerabilities will probably appear short after. Given the widespread use of Cisco"s routers, any vulnerability and/or exploit running wild will cause a huge disaster to the Internet as a whole.

One thing that I have not seen discussed in the many forums is what network administrators should do to remediate the risks of the "Digital Pearl Harbor," as described by Lynn. Cisco, ISS and many network professionals have suggested that the administrators upgrade all the Cisco routers to the latest IOS image.

Although a valid suggestion, upgrading routers is not a simple task. In addition to network disruptions, the latest IOS images may introduce new bugs, since they don"t just fix this one single issue. The latest IOS image you choose will most likely fix many bugs and introduce many new features. Upgrading means you will have to deal with new bugs that may not have affected you before.

There is, unfortunately, no single best way to fix this problem. However, many experienced networking administrators, especially ones working at large Internet service providers, have probably been through this type of upgrade scenario before and have probably created best practices. It would be great to see the community share these best practices with each other.

Before the IOS disaster strikes, and to get the upgrade process rolling, network administrators should do the following:

1. Inventory all Cisco routers in your infrastructure ASAP, including model number, IOS version, amount of memory available, and the importance of the router for the network. It"s also necessary to figure out whether the router can have additional memory modules installed. Many older routers have only 16MB of memory and thus cannot be upgraded to the latest IOS images.

2. Identify all routers that can be upgraded to the latest version. Routers that don"t have enough memory but can accept more should have additional memory installed as soon as possible. Routers that cannot accept new IOS image upgrades must be replaced. Based on this information, network administrators can then identify all of the IOS images that are required to upgrade the whole infrastructure.

3. Create a testing lab for the new IOS images. As mentioned before, the new IOS images most likely include more bug fixes and new features than the ones currently installed in the infrastructure. The new images may also introduce new bugs, so it is absolutely critical to test the new images to ensure they work well in a real-world environment.

4. Create a plan to replace the old routers ASAP. This may be cost inhibitive since many corporations are still running old hardware. However, there should be a plan to replace the most critical ones first and build into the IT budget to replace the rest.

5. Create a plan to upgrade routers to the latest IOS images after the testing is complete. Again, upgrade the most critical ones first.

Until Cisco fixes the root issue, ongoing upgrades are likely to be required because new vulnerabilities will be found and new exploits will be created. Network administrators, and the IT community as a whole, should work together to create a repeatable and reliable process that can be used in the future.

-- Jian Zhen, CISM, CISSP, is the director of product management at LogLogic, a log management vendor in Sunnyvale, Calif. He has been in the information security industry for nine years. He can be reached at zhenjl@gmail.com or through his blog at Operational Intelligence.