Questions about the project arose last week at about the same time Cliff Dodd, the company's CIO, resigned. Dodd stepped down last Monday after another Kaiser employee, Justen Deal, sent a memo to every company worker warning of technological and financial repercussions related to the rollout of the nearly US$4 billion system from Epic Systems Corp.
Deal said he also sent letters to Kaiser management expressing his concerns. But in internal memos, officials said they investigated those concerns and denied that the implementation of the HealthConnect system has been a failure.
"In the implementation of a new, large and complex system such as KP HealthConnect, various technical problems are likely to arise, but none that you mention are unknown to KP-IT nor were as insurmountable as you imply," Mark Zemelman, one of Kaiser's attorneys, said in a letter to Deal.
Kaiser declined to offer specifics about Dodd's resignation and whether questions about the project played a role in his departure. Deal, a publication project supervisor in the Health Education and Training Department at Kaiser Permanente's Los Angeles facility, was placed on paid administrative leave after sending the memo. Deal is not an IT employee, according to Kaiser.
A 722-page internal report obtained by Computerworld details hundreds of technical problems with the system -- some affecting patient care -- that appear to bear out the concerns of Deal and others in the organization that the system is a failure. The report also contains suggestions on how to fix the problems.
In his memo to employees, Deal pointed to reliability and scalability issues Kaiser faces with its HealthConnect system.
When fully implemented, it is supposed to give Kaiser's 12,000 physicians and other users instant access to the medical records of some 8.4 million patients, along with e-messaging capability, computerized order entry and electronic prescribing. In addition, the system is supposed to integrate appointment scheduling, registration and billing functions.
But according to the report, that doesn't seem to be happening with any degree of regularity.
For example, a power outage on May 9 at Kaiser's data center in Corona, Calif., lasted for 55 hours and 7 minutes and affected the ability of numerous health care facilities to access the HealthConnect system. "This outage affected every Kaiser Permanente Region. ... Providers/clinicians across KP would not have access to medical information to treat members, process lab and pharmacy requests, and in some cases were in full down-time modes compromising direct member treatment, with the inability to access current medical information for decision making," according to the report.
Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.
"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."
"Epic simply cannot scale to meet the size and needs of Kaiser Permanente," Deal said. "And we're wasting billions of dollars trying to make it. The big issues for me are the financial repercussions of trying to launch such an ineffective and inefficient and unreliable system across the organization."
According to Deal, Kaiser is wasting more than $1.5 billion a year, primarily on HealthConnect as well as on other inefficient and ineffective IT projects. Kaiser declined to provide financial details of the HealthConnect system, which it has been working on since 2003. Kaiser expects the outpatient portion of the project to be fully implemented in 2008, with the in-patient portion done in 2009.
Another issue is with the Epic software and its adaptability, according to Deal and the IT employee. They said the software was written in MUMPS (Massachusetts General Hospital Utility Multi-Programming System) -- a health care programming language originally developed in the 1960s.
"Basically, the problems really do follow the entire scope of the platform in that we have these issues with the adaptability of Epic and we have these issues with the information security of Epic -- so we implemented Citrix to try and protect that, and we're facing the scalability issues with that," Deal said. "Using Citrix is something that defies common sense. It would be like trying to use a dial up modem for thousands of users. It's just not going to work, and it's not something anyone would tell you a dial-up modem should work for.
"I don't think that Citrix really appreciates what we're trying to do with their software," Deal said. "I don't think ... we have any [service-level agreement] from them because they would not guarantee that this was going to work at all for this implementation."
"From what I've read, Epic [is] not very scalable and not very robust, and when you're putting Citrix in front of that -- we have most the most number of Citrix servers in the world -- you're going to have problems," said the IT employee.
Epic officials referred questions about the project to Kaiser.
Scott Herren, group vice president and general manager at Citrix Systems Inc.'s Virtualization System Group said the issue isn't scalability, it's really about the overall system architecture being set up to accommodate loads of that size. Kaiser is using Citrix's Presentation Server.
He said these issues do not stem from the Citrix product. "In fact, we have many very large successful Epic deployments around the world. However, in order to support large deployments, the Citrix implementation must be architected accordingly," Herren said.
"Kaiser has expanded its user base very quickly over the last year, and as such, Citrix is working with them, as well as other IT vendors, to make sure that their IT infrastructure is 100% ready for that expansion," he said. "Kaiser's IT infrastructure is robust with many applications, server farms and data centers running 24/7. As it is complex, we are confident in working with them to address any pain points so that their needs are being met and that they can consistently meet their customer and business demands."
The problems detailed in the Kaiser report include:
- On March 26, for 3 hours and 51 minutes, users in multiple locations were "intermittently unable to access [HealthConnect] -- receiving Citrix error message and therefore unable to access any patient info or update patient info in a timely manner," said the report.
On April 10, for 1 hour and 23 minutes, users in the Baldwin Park Medical Office were unable to place new orders for in-patients. "Drug Database information is not populating for pharmacist and technicians," the report said. "New orders will have to be manually done. In the manual process, the nurses cannot see patient updates for new orders for new medications or changes in meds, such as stopping orders. The manual process will create significant delays in patient care."
- On May 10, a power outage that lasted for 37 hours and 9 minutes affected multiple facilities. "Pharmacists must handle medication orders on these patients manually because one cannot enter medication orders into the system on patients that have not been admitted," the report said. "Also, if a patient were transferred during this time, they would need to track their location manually. In addition, the [users] are reporting that multiple patients are showing in the wrong beds. Patient Care Impact: The [users] reported that the potential for patients to not get the treatments they need and the possibility of patients receiving the incorrect medications."
- On May 23, for 1 hour and 57 minutes users in the Glendale Medical Office could not dispense medication because the billing application was down.
- On June 7, for 6 hours and 34 minutes, labs were unable to collect data, run tests and provide test results.
- On Sept. 15, for 48 minutes, users in multiple locations were unable to check patients in for their appointments. "As a result," the report said, "this causes delays at the front desk for the members to be checked in and seen by their providers. In addition, the departments are not made aware that the member has arrived for their appointment, which can cause delays in the workflow of the department."
- On Sept. 28, for 2 hours and 35 minutes, users in one facility were unable to release lab orders through HealthConnect. "This issue is a hindrance to the diagnosis," said the report.
- On Oct. 10, for 3 hours and 24 minutes, doctors and nurses in several facilities were unable to retrieve critical medical information to treat patients.
- On Oct. 11, for 1 hour and 47 minutes, multiple users in one facility were unable to access the network.
"The dedication of this organization is to quality outputs and system availability," said Henry Neidermeier, vice president and quality leader for KP-IT. "The complexity of the challenge here, with the links in the availability chain, is significant."
Kaiser spokesman Matthew Schiffgens said any issues that have come up should not be surprising, given the complexity of the project. He acknowledged that a report had been compiled to track problems.
"As you move out with a very large deployment like this, you encounter challenges along the way, and we have a process to systematically identify and address challenges as they arise," Schiffgens said. "The problem at the Corona data center was a good one. It came up, we addressed it, and we feel confident that we made the proper infrastructure to manage that. That is the fundamental practice of running a good business. Does that mean there are systematic and ongoing problems? No. You identify issues and address them as they go along. You're seeing a robust process in terms of those reports."
Neidermeier said Kaiser is focused on managing the problems detailed in the report and working to make sure it has backup procedures in place so that patient care isn't compromised.
But Kaiser workers are still concerned, according to the IT employee.
"People out in the field are frustrated, and the people in IT are just as frustrated because this was a solution forced upon us and was not an IT solution," said the IT employee. "I know in conversations I've had with my superiors there was a big push back in selecting Epic, and it was not a choice made by IT simply because of the large infrastructure needed to support it."