Wednesday's system failure stopped the trains at 5:27 p.m. PDT for about 70 minutes. The problems caused delays of up to two hours before all the trains were operating on schedule, he said.

'We're taking measures to make sure that this never happens again,' Johnson said. 'This is essentially a mistake on BART's part.'

About 15 of BART's approximately 100 IT workers usually perform software upgrades, he said. Software testing is done in advance on a separate 'virtual environment' so that it won't interfere with everyday operations, he said. The software upgrades done last Sunday had passed extensive testing and are designed to 'self-correct. [It] has proven to self-correct quickly when there is a problem.'

The work is part of a larger multiyear software upgrade being done in phases to avert commuter train slowdowns, he said. 'In the midst of this installation procedure, which has been going on for 14 months...things were going great until Monday,' Johnson said. The software upgrades are expected to take another five months to complete and BART officials are still trying to figure out what caused the problems this week.

The Integrated Control System is maintained by BART IT workers, but originally came from Logica, an England-based vendor, Johnson said.