Why was the reliability program developed? When a patient is submitting a prescription, we often have a more complete view of [his] history [than the pharmacy] because we see prescription data, medical data and lab data [that can be used] to apply better judgment in how medication is dispensed.
But if you don't have high stability and high quality of the core basic services, you can never get to those strategic discussions with the client.
Two years ago, we had a lot of reliability problems in delivering software into production. We were great at firefighting, but we tended not to spend the extra effort on prevention. We repaired and moved on.
We created this forum where we discuss every outage. What happened? Are we mobilized to get the repair done? How we can address prevention? That brings a heightened sense of accountability to key stakeholders.
About 18 months after we started, we had dropped defects that made it into production by 98 percent. Now we have good metrics around application failures.
How was the new program received? It was a bit of a shock across the company. Over time, a lot of accountability had been given to committees. They can't be accountable. We dissolved most of the committees.
In the past, a lot of this stuff that was creating problems wasn't even visible at the senior level unless it got so bad that the clients started calling the CEO. We now have a reporting system that tracks mean time between failures of all the applications.
When we first turned it on, it was scary, [but] my message [to employees] was, "We have to fix this together. The only thing that will get you in trouble is covering things up."
Give an example of a problem you have uncovered with the program. When we first started the reliability process, I would meet with my direct reports and ask about the status of operations. They would say, "We are green" [meaning all was going well]. I would walk from my office to the videoconference facility [to find] there was stuff going wrong all over the place.
[The reliability program] brings the people actually doing the work in with the senior executives of the company to talk about what happened. There is no spin control. It is live. The people who really helped us in the beginning -- who stepped up and were under fire -- they are some of the best managers we have today.