Issue StoriesNetworking
Solving System Crashesby James Noga, BS, MS; Patricia Volpe, BSBE, MBA; and Jeffrey B. Cooper, PhD
Our article in the November issue of 24x7 presented a case report of a system failure that illustrates a growing concern—the hazards associated with the integration of biomedical technologies into information systems. We described how an ECG archiving system crash—possibly initiated by a virus attack and exacerbated by several organizational and technical vulnerabilities-propagated into a system failure that impacted a hospital's ability to provide vital patient information to caregivers. The complexity of the crash was such that repair was lengthy and to this day incomplete. There were many root causes of this event. We will mention those briefly, but will concentrate primarily on a series of suggestions for how biomedical and information system organizations can collaborate better. The purpose of that collaboration is to create barriers to such failures and opportunities to better serve our health care systems by designing robust, integrated networks that effectively and safely move data, signals, and information; and provide the connectivity demanded in modern health care. We suggest that the reader review the previous article to appreciate the source of these suggestions.
We have not been able to identify the exact steps by which the system failure occurred. But, in reviewing the event, we ascertained weaknesses in system design, thorough documentation, and our own incomplete definition of communication pathways between the biomedical and information systems organizations. Lack of a backup system for the main archiving hardware hampered the recovery process. Capital budget constraints on hospital equipment can drive purchases without inclusion of such systems. Also contributing to the failure process was that this system was at the end of its life cycle yet still entirely functional. Funding for its replacement had been approved, but was delayed for several reasons, not the least of which was challenges in managing legacy ECG carts requiring connectivity. Before the replacement could be implemented, the system spiraled into its failure. The system was running on the Windows NT platform for which security patches were no longer available, leaving it vulnerable to attack. The manufacturer's slower than desired response time in validating urgent security patches frequently left the system at risk of virus infection and may have contributed to its initial vulnerability. Perhaps most critical was the custom interface that had been developed to enable improved workflow and billing: a turnover of personnel who had developed the interface combined with a lack of complete documentation contributed to the delayed resolution. Deeper root causes also existed. The lack of industry standards that define connectivity requirements for medical devices to health care information systems makes new interface development more challenging. And, the differing cultures within biomedical and information systems organizations likely contributed to the communication difficulties. In many ways, managing integrated biomedical technology and information technology systems is much more challenging than managing either alone. In our discussions, based on the responses from readers of the article and our own analysis, the following themes and suggestions emerged:
Creating a secure, available, and reliable biomedical equipment and systems environment will require intense and robust cooperation among BME, IS, vendors, and the FDA. We players on the field of device/network integration should not operate in silos. We must approach our challenges as a cohesive team whose goal is to ensure that integrated biomedical systems are designed, implemented, and supported with an understanding of the inherent risks of any new technology. It is only in that way that we will achieve the promise of connectivity without compromising safety. James Noga, BS, MS, is the chief information officer, Massachusetts General Hospital, Boston; Patricia Volpe, BSBE, MBA, is director of biomedical engineering, Massachusetts General Hospital; and Jeffrey B. Cooper, PhD, is director of biomedical engineering, Partners Healthcare System, and professor of anaesthesia, Harvard Medical School, Boston. For more information, contact . A Reader's ViewIn 24x7's November "Networking" article, "System Crashes: A Case Report," the authors—Jeffrey B. Cooper, Patricia Volpe, and James Noga—asked readers to send their recommendations regarding the topic. The following is a response by 24x7 reader Barry Kohler: That was a very relevant article. It highlights what will probably be a major responsibility of clinical engineering departments if they want to stay relevant. There are two issues that need to be resolved at the manufacturer level:
There has been some motion in the past few years concerning antivirus software compatibility. I hope that all your readers are pursuing this and asking their vendors to do likewise. Not all vendors share that information willingly, and some don't look into it at all. Our IT department recently changed antivirus vendors to one that most of our medical equipment manufacturers have not certified. As for planned obsolescence, it would be nice if we could have performance metrics for hardware that the FDA would accept as being justification for a comparable or superior component. As for the OS, I think that gorilla is way too big to expect any changes soon. In our hospital, we have had some major organizational changes in the last few years that acknowledge the overlap of skill sets and responsibilities. The most important was to have the biomed department answer to the CIO before the CEO. With fewer people to go through before a common administrator, it is much easier to balance department skills and priorities with those of the clinical staff. There have been successes and setbacks toward a better working relationship between clinical engineering and IT. I would recommend to anyone who can, try to rotate people between departments so they can become more conversant with each other and work more like a team toward a common goal. I recommend keeping all of our medical equipment on completely separate networks due to security lapses that cannot be easily overcome. The number of interfaces should be limited. Many vendors advertise that they can use the hospital backbone to transmit information between devices. It saves some upfront infrastructure cost and time to implement, but, unfortunately, many manufacturers tend to add options that need a separate interface like one for HL7 and another for a Web server. I also prefer to avoid generic PC-based equipment simply because of the issues above. Again, I would recommend that your readers take the time to emphasize their priorities the next time they see a salesperson in their building. Besides passing recommendations to salespeople, it would be nice if committees within AAMI, application vendors, and perhaps ECRI would assemble to work out the discrepancies between cost containment, high reliability expected by the FDA, and rapid deployment of innovation that pervades the general computing field. |
|
|
ADDITIONAL ONLINE RESOURCES |
|