Usability Evaluation Ecological Validity: Is More Always Better?

BACKGROUND: The ecological validity associated with usability testing of health information technologies (HITs) can affect test results and the predictability of real-world performance. It is, therefore, necessary to identify conditions with the greatest effect on validity. METHOD: We conducted a comparative analysis of two usability testing conditions. We tested a HIT designed for anesthesiologists to detect pain signals and compared two fidelity levels of ecological validity. We measured the difference in the number and type of use errors identified between high and low-fidelity experimental conditions. RESULTS: We identified the same error types in both test conditions, although the number of errors varied as a function of the condition. The difference in total error counts was relatively modest and not consistent across levels of severity. CONCLUSIONS: Increasing ecological validity does not invariably increase the ability to detect use errors. Our findings suggest that low-fidelity tests are an efficient way to identify and mitigate usability issues affecting ease of use, effectiveness, and safety. We believe early low-fidelity testing is an efficient but underused way to maximize the value of usability testing.

Tabular, Annotated, Visual, or Trends + Contextual Information? Preferences for Online Laboratory Results Displays.

People are increasingly offered access to their personal health information (e.g., laboratory results, clinical notes, diagnostic imaging results). However, this information is the same as that used by health care providers with clinical expertise and training in medical terminology, which citizens typically do not have. In this study, we examined participants (N = 24) preferences for four different types of displays for online laboratory (lab) results: Tabular, Annotated, Visual, and Trends + Contextual Information. The Friedman test of difference comparing participants' ratings of the four displays was significant, χ2(3)=10.8, P=.013, and the Wilcoxon signed rank pairwise comparison tests revealed that participants rated the visual lab results display significantly more favourably than the traditional display (Z=-2.746, P=.006). These findings indicate that many people prefer lab results displayed using more visual cues and some perceived this format as easier to understand than the other display formats. Given the importance of people accessing, understanding, and using their own health information, it is crucial for displays and systems to provide a better user experience. Displaying data (e.g., lab results) visually is one possible way to improve interpretability of personal health information provided to the public.

Overcoming Challenges to Inclusive User-based Testing of Health Information Technology with Vulnerable Older Adults: Recommendations from a Human Factors Engineering Expert Inquiry.

OBJECTIVES: Involving representative users in usability testing of health information technology (HIT) is central to user-centered design. However, (vulnerable) older adults as representative users have unique requirements. Aging processes may affect physical capabilities and cognitive skills, which can hamper testing with this demographic and may require special attention and revised protocols. This study was performed to provide expert-based recommendations for HIT user-testing with (vulnerable) older adults to support inclusive HIT design and evaluation. METHODS: First, we conducted a structured workshop with ten experts in HIT implementation and research, recruited through purposeful sampling, to generate insights into how characteristics of older adults may influence user-testing. Next, five Human Factor researchers experienced in HIT user-testing with (vulnerable) older adults validated the results and provided additional textual insights to gain consensus on the most important recommendations. A thematic analysis was performed on the resulting inquiries. Applied codes were based on the User-Centered Design framework. RESULTS: The analysis resulted in nine recommendations for user-testing of HIT with older adults, divided into three main themes: (1) empathetic approach and trust-building, (2) new requirements for testing and study design, and (3) adjustments to usability evaluation methods. For each theme a checklist of relevant items to follow-up on the recommendation is provided. CONCLUSIONS: The recommendations generated through expert inquiry contribute to more effective usability testing of HIT with older adults. This provides an important step towards improved accessibility of HIT amongst older adults through inclusive user-centered design.

Hidden in Plain Sight: Overlooked Results and Other Errors in Evaluating Online Laboratory Results.

People are increasingly accessing their own laboratory (lab) results online. However, Canadians may be expected to use different systems to access their results, depending upon where they are tested (e.g., community lab vs. hospital), and these results may be displayed differently. This study examined the extent to which participants without medical expertise (N = 25) made errors identifying lab results (i.e., missing or mis-identifying abnormal results) in a mock report. Six participants overlooked each of the flagged values, 20 participants missed an abnormal result that was not flagged, and 2 participants mis-identified a normal value as out of range. We describe potential causes of these errors and the implications for the design of consumer-facing lab results.

Using Simulations to Train Medical Students for Unanticipated Technology Failures in Telemedicine.

Simulations offer a safe environment for health professional training and the opportunity to predictably and consistently introduce events or variables that may be rare or dangerous in a live setting. Exposing trainees to unanticipated events during simulations can improve their ability to adapt and improvise. The COVID-19 pandemic accelerated the adoption of telehealth worldwide and highlighted the need for better training in health professional schools. In the United States, the Association of American Medical Colleges (AAMC) published new telehealth competency standards in 2021. The AAMC stated that health care providers should be aware of the risks of technology failures, capable of troubleshooting them, and lead systems interventions to improve safety. However, the AAMC does not provide guidance on the specific failures or solutions. In this study, we developed a set of technology failures that can be simulated in a telehealth curriculum. We incorporated one technology failure into a simulated telehealth encounter and gathered students' (N = 53) feedback on the exercise. Students' feedback was overwhelmingly positive. They agreed that integrating technology failures into telehealth simulations provides important practice managing these events during clinical encounters. While telehealth is an important healthcare delivery modality that can improve access-to-care, it is imperative to train medical students to navigate technology failures so that can adeptly manage these issues in clinical practice.

Opportunities to Improve COVID-19 Dashboard Designs for the Public.

Many organizations created COVID-19 dashboards to communicate epidemiologic statistics or community health capabilities with the public. In this paper we used dashboard heuristics to identify common violations observed in COVID-19 dashboards targeted to citizens. Many of the faults we identified likely stem from failing to include users in the design of these dashboards. We urge health information dashboard designers to implement design principles and test dashboards with representative users to ensure that their tools are satisfying user needs.

Information Chaos: An Adapted Framework Describing Citizens' Experiences with Information During COVID-19.

With the onset of the coronavirus pandemic, clinicians, public officials, and citizens alike struggled to stay abreast of the constant and evolving stream of information about the clinical manifestations of illness, epidemiology of the disease, and the public health response. In this paper, we adapted (i.e., added and modified elements) Beasley and colleagues' information chaos framework to understand the context of citizens' experiences with information during the COVID-19 pandemic. We will show how our adapted framework can be used to characterize information associated challenges observed during this time and the possible impact of information chaos on peoples' cognition and behaviours. Ultimately, we believe that research will benefit by adopting a more holistic perspective using the information chaos framework than strictly studying the independent factors in isolation.

Evaluation of Multimedia Medication Reconciliation Software: A Randomized Controlled, Single-Blind Trial to Measure Diagnostic Accuracy for Discrepancy Detection.

BACKGROUND: The Veterans Affairs Portland Healthcare System developed a medication history collection software that displays prescription names and medication images. OBJECTIVE: This article measures the frequency of medication discrepancy reporting using the medication history collection software and compares with the frequency of reporting using a paper-based process. This article also determines the accuracy of each method by comparing both strategies to a best possible medication history. STUDY DESIGN: Randomized, controlled, single-blind trial. SETTING: Three community-based primary care clinics associated with the Veterans Affairs Portland Healthcare System: a 300-bed teaching facility and ambulatory care network serving Veteran soldiers in the Pacific Northwest United States. PARTICIPANTS: Of 212 patients with primary care appointments, 209 patients fulfilled the study requirements. INTERVENTION: Patients randomized to a software-directed medication history or a paper-based medication history. Randomization and allocation to treatment groups were performed using a computer-based random number generator. Assignments were placed in a sealed envelope and opened after participant consent. The research coordinator did not know or have access to the treatment assignment until the time of presentation. MAIN OUTCOME MEASURES: The primary analysis compared the discrepancy detection rates between groups with respect to the health record and a best possible medication history. RESULTS: Of 3,500 medications reviewed, we detected 1,435 discrepancies. Forty-six percent of those discrepancies were potentially high risk for causing an adverse drug event. There was no difference in detection rates between treatment arms. Software sensitivity was 83% and specificity was 91%; paper sensitivity was 81% and specificity was 94%. No participants were lost to follow-up. CONCLUSION: The medication history collection software is an efficient and scalable method for gathering a medication history and detecting high-risk discrepancies. Although it included medication images, the technology did not improve accuracy over a paper list when compared with a best possible medication history. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT02135731.

Implementation of a Medication Reconciliation Assistive Technology: A Qualitative Analysis.

Objective: To aid the implementation of a medication reconciliation process within a hybrid primary-specialty care setting by using qualitative techniques to describe the climate of implementation and provide guidance for future projects. Methods: Guided by McMullen et al's Rapid Assessment Process1, we performed semi-structured interviews prior to and iteratively throughout the implementation. Interviews were coded and analyzed using grounded theory2 and cross-examined for validity. Results: We identified five barriers and five facilitators that impacted the implementation. Facilitators identified were process alignment with user values, and motivation and clinical champions fostered by the implementation team rather than the administration. Barriers included a perceived limited capacity for change, diverging priorities, and inconsistencies in process standards and role definitions. Discussion: A more complete, qualitative understanding of existing barriers and facilitators helps to guide critical decisions on the design and implementation of a successful medication reconciliation process.

Beyond technology acceptance to effective technology use: a parsimonious and actionable model.

OBJECTIVE: To develop and test a parsimonious and actionable model of effective technology use (ETU). DESIGN: Cross-sectional survey of primary care providers (n = 53) in a large integrated health care organization that recently implemented new medication reconciliation technology. METHODS: Surveys assessed 5 technology-related perceptions (compatibility with work values, implementation climate, compatibility with work processes, perceived usefulness, and ease of use) and 1 outcome variable, ETU. ETU was measured as both consistency and quality of technology use. RESULTS: Compatibility with work values and implementation climate were found to have differential effects on consistency and quality of use. When implementation climate was strong, consistency of technology use was high. However, quality of technology use was high only when implementation climate was strong and values compatibility was high. This is an important finding and highlights the importance of users' workplace values as a key determinant of quality of use. CONCLUSIONS: To extend our effectiveness in implementing new health care information technology, we need parsimonious models that include actionable determinants of ETU and account for the differential effects of these determinants on the multiple dimensions of ETU.

Design of admission medication reconciliation technology: a human factors approach to requirements and prototyping.

OBJECTIVE: Our objectives were to (1) develop an in-depth understanding of the workflow and information flow in medication reconciliation, and (2) design medication reconciliation support technology using a combination of rapid-cycle prototyping and human-centered design. BACKGROUND: Although medication reconciliation is a national patient safety goal, limitations both of physical environment and in workflow can make it challenging to implement durable systems. We used several human factors techniques to gather requirements and develop a new process to collect a medication history at hospital admission. METHODS: We completed an ethnography and time and motion analysis of pharmacists in order to illustrate the processes used to reconcile medications. We then used the requirements to design prototype multimedia software for collecting a bedside medication history. We observed how pharmacists incorporated the technology into their physical environment and documented usability issues. RESULTS: Admissions occurred in three phases: (1) list compilation, (2) order processing, and (3) team coordination. Current medication reconciliation processes at the hospital average 19 minutes to complete and do not include a bedside interview. Use of our technology during a bedside interview required an average of 29 minutes. The software represents a viable proof-of-concept to automate parts of history collection and enhance patient communication. However, we discovered several usability issues that require attention. CONCLUSIONS: We designed a patient-centered technology to enhance how clinicians collect a patient's medication history. By using multiple human factors methods, our research team identified system themes and design constraints that influence the quality of the medication reconciliation process and implementation effectiveness of new technology. KEYWORDS: Evidence-based design, human factors, patient-centered care, safety, technology.

Addressing the sociotechnical drivers of quality improvement: a case study of post-operative DVT prophylaxis computerised decision support.

BACKGROUND: Quality improvement (QI) initiatives characterised by iterative cycles of quantitative data analysis do not readily explain the organisational determinants of change. However, the integration of sociotechnical theory can inform more effective strategies. Our specific aims were to (1) describe a computerised decision support intervention intended to improve adherence with deep venous thrombosis (DVT) prophylaxis recommendations; and (2) show how sociotechnical theory expressed in 'Fit between Individuals, Task and Technology' framework (FITT) can identify and clarify the facilitators and barriers to QI work. METHODS: A multidisciplinary team developed and implemented electronic menus with DVT prophylaxis recommendations. Stakeholders were interviewed and human factors were analysed to optimise integration. Menu exposure, order placement and clinical performance were measured. Vista tool extraction and chart review were used. Performance compliance pre-implementation was 77%. RESULTS: There were 80-110 eligible cases per month. Initial menu use rate was 20%. After barriers were classified and addressed using the FITT framework, use improved 50% to 90%. Tasks, users and technology issues in the FITT model and their interfaces were identified and addressed. Workflow styles, concerns about validity of guidelines, cycle times and perceived ambiguity of risk were issues identified. CONCLUSIONS: DVT prophylaxis in a surgical setting is fraught with socio-political agendas, cognitive dissonance and misaligned expectations. These must be sought and articulated if organisations are to respond to internal resistance to change. This case study demonstrates that QI teams using information technology must understand the clinical context, even in mature electronic health record environments, in order to implement sustainable systems.

Primary care provider perceptions and use of a novel medication reconciliation technology.

BACKGROUND: Although medication reconciliation (MR) can reduce medication discrepancies, it is challenging to operationalise. Consequently, we developed a health information technology (HIT) to collect a patient medication history and make it available to the primary care (PC) provider. We deployed a self-service kiosk in a PC clinic that permits patients to indicate a medication adherence history. Patient responses are immediately viewable in the legacy electronic health record. This paper describes a survey developed to assess PC provider perceptions of our HIT and HIT implementation effectiveness. METHODS: We developed and administered a survey to all PC providers to assess technology implementation effectiveness. The survey included scales measuring (1) user attitudes towards MR, (2) perceptions of our HIT and (3) the local organisational climate for implementation. We also assessed the consistency and quality of tool use. RESULTS: Nearly 90% of PC providers responded to the survey and 58% indicated that they were familiar with the technology and had seen the tool output. Most providers believed that MR represented an important safety intervention, although 43% did not believe that they had the necessary resources to manage discrepancies. Composite scale scores for the 58% of respondents familiar with the HIT indicate that the majority favoured our tool over usual care. However, composite scale scores suggest that the climate for implementation at our facility was suboptimal. Overall, the quality and consistency of tool use among providers was very heterogeneous. CONCLUSIONS: A patient self-service kiosk offers an efficient mechanism to collect a medication adherence history; provider survey responses indicate that they appreciated and used the MR kiosk output. Nonetheless, opportunities exist to improve data displays and embed decision support to facilitate discrepancy management.

Design and implementation of a medication reconciliation kiosk: the Automated Patient History Intake Device (APHID).

Errors associated with medication documentation account for a substantial fraction of preventable medical errors. Hence, the Joint Commission has called for the adoption of reconciliation strategies at all United States healthcare institutions. Although studies suggest that reconciliation tools can reduce errors, it remains unclear how best to implement systems and processes that are reliable and sensitive to clinical workflow. The authors designed a primary care process that supported reconciliation without compromising clinic efficiency. This manuscript describes the design and implementation of Automated Patient History Intake Device (APHID): ambulatory check-in kiosks that allow patients to review the names, dosage, frequency, and pictures of their medications before their appointment. Medication lists are retrieved from the electronic health record and patient updates are captured and reviewed by providers during the clinic session. Results from the roll-in phase indicate the device is easy for patients to use and integrates well with clinic workflow.