IT upgrades slow trains in San Francisco

31.03.2006
Unsuccessful software upgrades made to San Francisco's Bay Area Rapid Transit (BART) train system last Sunday stranded thousands of commuters for up to two hours Monday and Tuesday when the trains had to be stopped for safety reasons while the IT system was repaired.

Linton Johnson, a BART spokesman, said the delays and shutdowns occurred after IT staffers did maintenance upgrades to the software that coordinates and runs the trains, tracks, operating signals and track switches. The attempted upgrades caused the system to crash twice during the following 48 hours, he said.

On Monday and Tuesday, the crashes caused transit delays when the trains were halted so they didn't run into each other, he said. After those software problems, BART IT workers on Wednesday decided to install backup software for redundancy in case of continuing problems, Johnson said. 'We were rushing to do the right thing,' he said. 'However, in the process of installing that backup system, we interfered with a [network switch] that crashed our system.'

That switch allows computers in BART's central operations center to communicate with other parts of the mass transit system to keep it running properly.

When the backup system was brought online, the network switch was overloaded, 'which wasn't anticipated by our computer technicians,' he said.

Normally, such maintenance isn't done during the week to avoid disrupting commuter service, Johnson said. But the backup system installation was done on Wednesday to try to stop the problems experienced earlier in the week. 'We, for some reason, decided to try and get this backup system installed' during normal commuting hours, he said. 'Technically, it shouldn't have affected anything, but in reality, it did.'