Building Usability Knowledge for Health Information Technology: A Usability-Oriented Analysis of Incident Reports.

BACKGROUND: The contribution of usability flaws to patient safety issues is acknowledged but not well-investigated. Free-text descriptions of incident reports may provide useful data to identify the connection between health information technology (HIT) usability flaws and patient safety. OBJECTIVES: This article examines the feasibility of using incident reports about HIT to learn about the usability flaws that affect patient safety. We posed three questions: (1) To what extent can we gain knowledge about usability issues from incident reports? (2) What types of usability flaws, related usage problems, and negative outcomes are reported in incidents reports? (3) What are the reported usability issues that give rise to patient safety issues? METHODS: A sample of 359 reports from the U.S. Food and Drug Administration Manufacturer and User Facility Device Experience database was examined. Descriptions of usability flaws, usage problems, and negative outcomes were extracted and categorized. A supplementary analysis was performed on the incidents which contained the full chain going from a usability flaw up to a patient safety issue to identify the usability issues that gave rise to patient safety incidents. RESULTS: A total of 249 reports were included. We found that incident reports can provide knowledge about usability flaws, usage problems, and negative outcomes. Thirty-six incidents report how usability flaws affected patient safety (ranging from incidents without consequence, to death) involving electronic patient scales, imaging systems, and HIT for medication management. The most significant class of involved usability flaws is related to the reliability, the understandability, and the availability of the clinical information. CONCLUSION: Incidents reports involving HIT are an exploitable source of information to learn about usability flaws and their effects on patient safety. Results can be used to convince all stakeholders involved in the HIT system lifecycle that usability should be considered seriously to prevent patient safety incidents.

Insights and limits of usability evaluation methods along the health information technology lifecycle.

A great variety of usability evaluation methods exist but they do not provide the same kind of results and do not address the same stage of the Health Information Technology (HIT) lifecycle. This paper takes stock of the application of expert evaluation, usability testing, clinical simulation, clinical trials and post-implementation surveillance to provide an overview of their main similarities and differences. Results from this comparison will help in choosing methods that are best able to evaluate a HIT and improve its usability and ultimately its safety of use.

A framework for reporting on human factor/usability studies of health information technologies.

Increasingly, studies are being published on the potential negative effect of introducing poor designed Health Information Technology (HIT) into clinical settings, relating to technology-induced errors and adverse events. Academic research on HIT design and evaluation is an extremely important source of information in providing new insights into factors contributing to successful system (re)design efforts, system user-friendliness and usability issues and safety critical aspects of HIT design. However, these studies have been inconsistent and incomprehensive in their reporting, complicating the appraisal of outcomes, generalizability of study findings, meta-analysis and harmonization of the available evidence. To improve identification of type of use errors and safety related issues regarding design and implementation of HIT, consensus on issues to be reported on in scientific publications is a necessary step forward. This study presents the first approach to a framework providing a set of principles to follow for comprehensive and unambiguous reporting of HIT design and usability evaluation studies with the objective to reduce variation, improve on the publication reporting quality and proper indexation of these studies. This framework may be helpful in expanding the knowledge base not only concerning the application of Human Factors (HF)/Usability studies of HIT but also improve the knowledge base of how to (re)design and implement effective, efficient and safe HIT.

Next steps in evaluation and evidence - from generic to context-related.

INTRODUCTION: E-health systems are increasingly important and widespread, but their selection and implementation are still frequently based on belief, rather than scientific evidence, and adverse effects are not systematically addressed. Progress is being made in promoting generic evaluation methodologies as a source of scientific evidence, but effort is now needed to consider methods for special situations. METHOD: Review of five evaluation contexts - national e-health plans, telemedicine, Health Informatics 3.0, usability and economics. CONCLUSION: Identification of requirements for approaches to be developed in these five settings.

Example of a Human Factors Engineering approach to a medication administration work system: potential impact on patient safety.

OBJECTIVE: The objectives of this paper are: In this approach, the implementation of such a complex IT solution is considered a major redesign of the work system. The paper describes the Human Factor (HF) tasks embedded in the project lifecycle: (1) analysis and modelling of the current work system and usability assessment of the medication CPOE solution; (2) HF recommendations for work re-design and usability recommendations for IT system re-engineering both aiming at a safer and more efficient work situation. METHODS: Standard ethnographic methods were used to support the analysis of the current work system and work situations, coupled with cognitive task analysis methods and documents review. Usability inspection (heuristic evaluation) and both in-lab (simulated tasks) and on-site (real tasks) usability tests were performed for the evaluation of the CPOE candidate. Adapted software engineering models were used in combination with usual textual descriptions, tasks models and mock-ups to support the recommendations for work and product re-design. RESULTS: The analysis of the work situations identified different work organisations and procedures across the hospital's departments. The most important differences concerned the doctor-nurse communications and cooperation modes and the procedures for preparing and administering the medications. The assessment of the medication CPOE functions uncovered a number of usability problems including severe ones leading to impossible to detect or to catch errors. Models of the actual and possible distribution of tasks and roles were used to support decision making in the work design process. The results of the usability assessment were translated into requirements to support the necessary re-engineering of the IT application. CONCLUSION: The HFE approach to medication CPOE efficiently identifies and distinguishes currently unsafe or uncomfortable work situations that could obviously benefit from an IT solution from other work situations incorporating efficient work procedures that might be impaired by the implementation of the CPOE. In this context, a careful redesign of the work situation and of the entire work system is necessary to actually benefit from the installation of the product in terms of patient safety and human performances. In parallel, a usability assessment of the product to be implemented is mandatory to identify potentially dangerous usability flaws and to fix them before the installation.

Integrating the SE and HCI models in the human factors engineering cycle for re-engineering Computerized Physician Order Entry systems for medications: basic principles illustrated by a case study.

OBJECTIVES: The integration of Human Factors is still insufficient in the design and implementation phases of complex interactive systems such as Computerized Physician Order Entry (CPOE) systems. One of the problems is that human factors specialists have difficulties to communicate their data and to have them properly understood by the computer scientists in the design and implementation phases. This paper presents a solution to this problem based on the creation of common documentation supports using Software Engineering (SE) and Human-Computer Interaction (HCI) methods. METHOD: The integration of SE and HCI methods and models is an interesting means for modelling an organization's activities, with software applications being part of these activities. Integrating these SE and HCI methods and models allows case studies to be seen from the technical, organizational and ergonomic perspectives, and also makes it easier to compare current and future work situations. RESULTS: The exploitation of these techniques allows the creation of common work supports that can be easily understandable by computer scientists and relevant for re-engineering or design. In this paper, the basic principles behind such communication supports are described and illustrated by a real case study.

Human factors methods to support the experts' review of automatically detected adverse drug events.

The European project Patient Safety through Intelligent Procedures in Medication (PSIP) aims at semi-automatically identifying and preventing Adverse Drug Events (ADEs). Data mining of the structured hospital data bases provides a list of potential ADEs, along with their frequencies and probabilities. Once a set of ADEs has been detected by data mining techniques, it is necessary to have them validated by human experts. This paper presents the methods used to support the review by clinicians and pharmacologists of these automatically detected ADEs. We use think-aloud methods and portable labs to track and record the experts reasoning and their reviewing cognitive procedures. We present preliminary results obtained with this method, which allows identifying the key data and information used to characterize the ADEs.

User-centred, multidimensional assessment method of Clinical Information Systems: a case-study in anaesthesiology.

This paper describes a multi-dimensional assessment method to support the choice and acquisition process for Clinical Information Systems. The method addresses three different dimensions: (1) Quality management, which evaluates the fulfillment of Users Requirements, and the Users' satisfaction with the existing functions; (2) Usability assessment, which includes a Usability inspection, and a Usability test; (3) Performance evaluation, which assesses the exhaustiveness and quality of documentation. The method is illustrated using the case study of a Clinical Information System (CIS) acquisition for anaesthesiology. It proved efficient and promising as a support for the decision process, for enhancing users' involvement in the project, and for initiating the necessary re-engineering of the Human Computer Interface.

A usability study of CPOE's medication administration functions: impact on physician-nurse cooperation.

Implementation of CPOE systems in Healthcare Institutions has proven efficient in reducing medication errors but it also induces hidden side-effects on Doctor-Nurse cooperation. We propose a usability engineering approach to this problem. An extensive activity analysis of the medication ordering and administration process was performed in several departments of 3 different hospitals. Two of these hospitals are still using paper-based orders, while the 3rd one is in the roll-out phase of medication functions of its CPOE system. We performed a usability assessment of this CPOE system. The usability assessment uncovered usability problems for the entry of medication administration time scheduling by the physician and revealed that the information can be ambiguous for the nurse. The comparison of cooperation models in both situation shows that users tend to adopt a distributed decision making paradigm in the paper-based situation, while the CPOE system supports a centralized decision making process. This analysis can support recommendation for the re-engineering of the Human-Computer Interface.

Multidimensional evaluation of a Clinical Information System for anaesthesiology: quality management, usability, and performances.

This paper describes a multi-dimensional assessment methodology to support the choice and selection process of Clinical Information Systems. The methodology includes 4 evaluation components: Fulfilment of Users Requirements, Usability Inspection, Usability Testing and Performance Evaluation. The methodology is illustrated with the case study of a selection of a CIS for anaesthesiology. It proved to be very efficient and powerful to support the selection process, to enhance users' involvement in the project, and to initiate the necessary re-engineering of the HCI.

Comparative assessment of two interfaces for delivering a multimedia medical course in the French-speaking Virtual Medical University (UMVF).

The UMVF aims at helping medical students during their normal curriculum via the facilities provided by Internet based techniques. This paper describes a comparative assessment of two interfaces delivering a multimedia course: a conventional web server (WS) and an integrated e-learning platform in the form of a Virtual Campus (VC). Eleven students were arbitrarily divided into two groups. We used a qualitative method for comparing their acceptance of the on line course provided by the two different interfaces. The two groups were globally satisfied. However, a decrease in satisfaction was noted at the end of the experimentation in the VC group. This may be explained by a more complex Graphical User Interface (GUI) of the VC and some constraints which do not exist with the WS. The current e-learning platforms are probably not optimised for working conditions where presential and virtual activities are mixed. We think that a new type of "light" platforms should be developed for these specific working conditions. Students of the two groups also had limitations about the multimedia environment. They may change their opinion if they get more accustomed with the multimedia environment and if their teachers make a more adequate use of the multimedia techniques.

Usability assessment study of a web site displaying medical resources on line: the CISMeF.

We present here the usability assessment study of a medical web site dedicated to the Cataloguing and Indexing of French speaking Medical web Sites (CISMeF). We performed a usability inspection using heuristic evaluation and an empirical usability test using a portable lab. The results of the heuristic evaluation show usability flaws along guidance, prompting and legibility ergonomic criteria. The usability test confirms this result and gives some more hints about the severity rating of the problems. From these results, we draw up a set of recommendations for the re-engineering of the Human Computer Interface (HCI). We conclude on the necessity to integrate usability engineering early enough in the projects lifecycle.