The differences between adults and adolescents using a mobile health application for menstrual complaints: A usability and qualitative study.

BACKGROUND: A "Menstruatie Educatie Kalender" application (Menstruation Education Calendar, (MEK-APP)) was developed for adults to evaluate menstrual complaints. The future aim of this app is to use it as a self-diagnostic instrument for menstrual abnormalities for both adults and adolescents. Early identification of the potential of an application for future use by both user groups would increase implementation success and adoption of the application. OBJECTIVE: To compare differences in experienced usability by adults versus adolescents and to identify factors influencing future use for both age groups in one mHealth application (in this study the MEK-APP). METHODS: This study consisted of three phases: (1) usability testing of the MEK-APP for iOS and Android by think-aloud method, (2) two-month daily use and (3) in-depth individual interview. During the think-aloud sessions, twelve tasks were performed in the application while they were thinking aloud. Usability problems were rated for their severity with Nielsen' Severity Scale. Both the think-aloud sessions and in-depth interviews were verbatim transcribed and thematically analyzed to determine the factors influencing future use for both groups. In addition, the System Usability Scale (SUS) and Intrinsic Motivation Inventory (IMI) questionnaires were filled out during the interviews. RESULTS: Seven adults (>18 years) and seven adolescents (14-18 years) evaluated the MEK-APP. There were 14 usability issues and 16 bugs in both groups. There were no differences between adults and adolescents. In the thematic qualitative analysis, the following future use factors were identified: user-expectation, motivation, privacy, understandability, and user-experience. The user-expectation, motivation and privacy differed between both groups but did not influence usage. No differences were observed in SUS and IMI scores between both groups. CONCLUSIONS: There are five factors influencing the future use of a menstrual-related mHealth application for both adults and adolescents. It is possible to serve different age groups with a single application.

General Practitioners' Perspectives About Remote Dermatology Care During the COVID-19 Pandemic in the Netherlands: Questionnaire-Based Study.

BACKGROUND: The COVID-19 pandemic affected the delivery of primary care and stimulated the use of digital health solutions such as remote digital dermatology care. In the Netherlands, remote store-and-forward dermatology care was already integrated into Dutch general practice before the COVID-19 pandemic. However, it is unclear how general practitioners (GPs) experienced this existing digital dermatology care during the pandemic period. OBJECTIVE: We investigated GPs' perspectives about facilitators and barriers related to store-and-forward digital dermatology care during the COVID-19 pandemic in the Netherlands, using a sociotechnical approach. METHODS: In December 2021, a web-based questionnaire was distributed via email to approximately 3257 GPs who could perform a digital dermatology consultation and who had started a digital consultation (not necessarily dermatology) in the previous 2 years. The questionnaire consisted of general background questions, questions from a previously validated telemedicine service user satisfaction questionnaire, and newly added questions related to the pandemic and use of the digital dermatology service in general practice. The open-ended and free-text responses were analyzed for facilitators and barriers using content analysis, guided by an 8-dimensional sociotechnical model. RESULTS: In total, 71 GPs completed the entire questionnaire, and 66 (93%) questionnaires were included in the data analysis. During the questionnaire distribution period, another national lockdown, social distancing, and stay-at-home mandates were announced; thus, GPs may have had increased workload and limited time to complete the questionnaire. Of the 66 responding GPs, 36 (55%) were female, 25 (38%) were aged 35-44 years, 33 (50%) were weekly platform users, 34 (52%) were working with the telemedicine organization for >5 years, 42 (64%) reported that they used the store-and-forward platform as often during as before the pandemic, 61 (92%) would use the platform again, 53 (80%) would recommend the platform to a colleague, and 10 (15%) used digital dermatology home consultation. Although GPs were generally satisfied with the digital dermatology service, platform, and telemedicine organization, they also experienced crucial barriers to the use of the service during the pandemic. These barriers were GPs' and patients' limited digital photography skills, costs and the lack of appropriate equipment, human-computer interface and interoperability issues on the telemedicine platform, and different use procedures of the digital dermatology service. CONCLUSIONS: Although remote dermatology care was already integrated into Dutch GP practice before the pandemic, which may have facilitated the positive responses of GPs about the use of the service, barriers impeded the full potential of its use during the pandemic. Training is needed to improve the use of equipment and quality of (dermoscopy) images taken by GPs and to inform GPs in which circumstances they can or cannot use digital dermatology. Furthermore, the dermatology platform should be improved to also guide patients in taking photographs with sufficient quality.

Proposing a Novel Hybrid Short-Term Exchange Program in Biomedical and Health Informatics Education.

International student exchange is a valuable opportunity for Biomedical and Health Informatics students to gain new perspectives and experiences. In the past, such exchanges have been made possible through international partnerships between universities. Unfortunately, numerous obstacles such as housing, financial concerns, and environmental implications related to travel, have made it difficult to continue international exchange. Experiences with hybrid and online education during covid-19 paved the way for a new approach that allows for short international exchange with a hybrid online-offline supervision model. This will be initiated with an exploration project between two international universities , each related to their respective institute's research focus.

Medical informatics and climate change: a framework for modeling green healthcare solutions.

OBJECTIVE: The aim of this study was to develop a theory-based framework to enhance and accelerate development, selection, and implementation of solutions mitigating the climate impact of healthcare organizations. MATERIALS AND METHODS: Existing frameworks were combined to develop the Green-MIssion (Medical Informatics Solutions) framework. It was further developed and refined by mapping solutions from project plans and reviewing it with an expert panel. RESULTS: The framework classifies solutions into three categories: (1) monitor and measure environmental impact of a healthcare setting; (2) help create and increase awareness among employees and patients; and (3) interventions to reduce environmental impacts. DISCUSSION AND CONCLUSION: The framework combines concepts from healthcare information technology and environmental sciences and can be used to structure green medical informatics solutions for different healthcare settings. Furthermore, research should evaluate its application for measuring and assessing the impact of green medical informatics solutions on environmental sustainability and climate resilience.

Expert appraisal and prioritization of barriers to mHealth use for older adults living with Alzheimer's disease and related Dementias: A Delphi study.

INTRODUCTION: Older adults living with Alzheimer's disease and related dementias (ADRD) can benefit from mHealth innovations in (daily) care. However, successful implementation and adoption of such innovations can be hindered by a lack of inclusive design. Inclusive design can be challenging, due to the variety of ADRD- and aging-related symptoms that can pose barriers to using mHealth. Previously, a literature-based model with 53 barriers to mHealth use for this population has been developed ("MHealth for OLder adults living with DEMentia - USability" or MOLDEM-US). In this study, we aim to prioritize these through a Delphi study with ADRD experts (case managers, informal caregivers, hospital healthcare professionals, district nurses, and researchers). METHODS: In the first round, participant characteristics and potentially new insights into barriers to mHealth use for older adults living with ADRD were gathered. The consensus questionnaire was submitted in the second round, containing barriers to mHealth use for this population (from MOLDEM-US) with questions inquiring its impact and frequency. In the third round, participants rejudged those barriers for which no consensus (<51 %) or minor consensus (51 % - 60 %) was reached. RESULTS: Thirty-seven participants completed the three rounds of the study. Consensus was reached for eleven barriers after the second round, all having major impact and frequency: integration of functions during daily activities, perceived complexity, efficiency in seeing benefits, trust in own ability, restlessness and agitation, computer literacy, self confidence in using wearables, learnability, working memory, and visual acuity. CONCLUSION: After round three, consensus was achieved for all 53 barriers. Twenty-six barriers are considered to majorly affect mHealth use, most of which relate to cognition and frame of mind. This study contributes to the development of mHealth design guidelines that take into account the progressive and diverse ADRD- and aging-related symptoms negatively affecting mHealth implementation and adoption.

Overcoming Challenges to Inclusive User-based Testing of Health Information Technology with Vulnerable Older Adults: Recommendations from a Human Factors Engineering Expert Inquiry.

OBJECTIVES: Involving representative users in usability testing of health information technology (HIT) is central to user-centered design. However, (vulnerable) older adults as representative users have unique requirements. Aging processes may affect physical capabilities and cognitive skills, which can hamper testing with this demographic and may require special attention and revised protocols. This study was performed to provide expert-based recommendations for HIT user-testing with (vulnerable) older adults to support inclusive HIT design and evaluation. METHODS: First, we conducted a structured workshop with ten experts in HIT implementation and research, recruited through purposeful sampling, to generate insights into how characteristics of older adults may influence user-testing. Next, five Human Factor researchers experienced in HIT user-testing with (vulnerable) older adults validated the results and provided additional textual insights to gain consensus on the most important recommendations. A thematic analysis was performed on the resulting inquiries. Applied codes were based on the User-Centered Design framework. RESULTS: The analysis resulted in nine recommendations for user-testing of HIT with older adults, divided into three main themes: (1) empathetic approach and trust-building, (2) new requirements for testing and study design, and (3) adjustments to usability evaluation methods. For each theme a checklist of relevant items to follow-up on the recommendation is provided. CONCLUSIONS: The recommendations generated through expert inquiry contribute to more effective usability testing of HIT with older adults. This provides an important step towards improved accessibility of HIT amongst older adults through inclusive user-centered design.

Evaluating mHealth Design for People with Dementia: Preliminary Results.

The availability of mHealth for people with dementia is increasing. Various mHealth design guidelines for this population have been proposed. In this study, we developed a binary checklist with evaluation statements to assess the implementation of twenty design suggestions in seven currently available mHealth apps for people with dementia. Between 17%-65% of the evaluation statements in the checklist were implemented in these apps. Not all statements were considered applicable for each assessed mHealth app, which resulted in dividing the criteria in two groups as either key evaluation statements or optional evaluation statements. In future work we want to augment this checklist to contribute to the future design of mHealth for people with dementia.

Dementia-Related Barriers to mHealth Use: Validation by Expert Opinion.

mHealth use for people with dementia is fraught with factors influencing its implementation in care and daily life. A better understanding of these factors may provide guidelines to inclusive design. This study aimed to assess whether factors gathered in a literature-based model could be validated by opinions of experts. On basis of a questionnaire as part of a larger study, experts identified barriers that they considered to be related to aging and dementia influencing mHealth use. Nineteen barriers that were mentioned by the dementia experts were covered in our literature-based model. No adaptions to the model were required. The dementia experts acclaimed three barriers to mHealth use that could not be mapped onto the framework: the unavailability of (informal) caregivers to support the mHealth use, the stage and type of dementia of an mHealth user, and the fear of the unknown. These should be considered as prerequisites in the implementation phase of mHealth and explored more in future research.

To Refer or Not to Refer in Teledermoscopy: Retrospective Study.

BACKGROUND: Challenges remain for general practitioners (GPs) in diagnosing (pre)malignant and benign skin lesions. Teledermoscopy (TDsc) supports GPs in diagnosing these skin lesions guided by teledermatologists' (TDs) diagnosis and advice and prevents unnecessary referrals to dermatology care. However, the impact of the availability of TDsc on GPs' self-reported referral decisions to dermatology care before and after the TDsc consultation is unknown. OBJECTIVE: The objective of this study is to assess and compare the initial self-reported referral decisions of GPs before TDsc versus their final self-reported referral decisions after TDsc for skin lesions diagnosed by the TD as (pre)malignant or benign. METHODS: TDsc consultations requested by GPs in daily practice between July 2015 and June 2020 with a TD assessment and diagnosis were extracted from a nationwide Dutch telemedicine database. Based on GP self-administered questions, the GPs' referral decisions before and their final referral decision after TDsc consultation were assessed for (pre)malignant and benign TD diagnoses. RESULTS: GP self-administered questions and TD diagnoses were evaluated for 6364 TDsc consultations (9.3% malignant, 8.8% premalignant, and 81.9% benign skin lesions). In half of the TDsc consultations, GPs adjusted their initial referral decision after TD advice and TD diagnosis. Initially, GPs did not have the intention to refer 67 (56.8%) of 118 patients with a malignant TD diagnosis and 26 (16.0%) of 162 patients with a premalignant TD diagnosis but then decided to refer these patients after the TDsc consultation. Furthermore, GPs adjusted their decision from referral to nonreferral for 2534 (74.9%) benign skin lesions (including 676 seborrheic keratosis and 131 vascular lesions). CONCLUSIONS: GPs adjusted their referral decision in 52% (n=3306) of the TDsc consultations after the TD assessment. The availability of TDsc is thus of added value and assists GPs in their (non)referral for patients with skin lesions to dermatology care. TDsc resulted in referrals of patients with (pre)malignant skin lesions that GPs would not have referred directly to the dermatologist. TDsc also led to a reduction of unnecessary referrals of patients with low complex benign skin lesions (eg, seborrheic keratosis and vascular lesions).

Considerate mHealth design for older adults with Alzheimer's disease and related dementias (ADRD): A scoping review on usability barriers and design suggestions.

BACKGROUND: The number of older adults with Alzheimer's disease and related dementias (ADRD) is increasing worldwide. This offers ample opportunities for mobile health (mHealth) apps, for example to support them in performing daily activities or monitoring their health status. However, how to design these apps taking into consideration ADRD related barriers remains a challenge. OBJECTIVE: To provide a synthesis of mHealth usability barriers of older adults with ADRD and design suggestions to enhance user-friendliness of mHealth apps. METHODS: We performed a scoping review of mHealth studies focused on older adults with ADRD and: (1) usability barriers experienced, and/or (2) implementation or design suggestions. PubMed, Medline, EMBASE, PsycINFO, and Web of Science were searched. Clinical and grey literature on ADRD diminishments was explored to identify potential barriers to using mHealth. Detected usability barriers caused by ADRD impairments were mapped onto the MOLD-US framework through validation, calibration, and adaption. MOLD-US was initially developed to provide an overview of barriers influencing mHealth usability for the general aging population. RESULTS: Title and abstract of 792 unique citations were scanned of which 69 were included for full text review. Fifteen studies matched inclusion criteria. In total 42 barriers were identified to influence mobile health use for older adults with ADRD. Twenty design suggestions were extracted from the studies. CONCLUSIONS: The identified usability barriers were classified in five categories: cognition, perception, physical ability, frame of mind, and speech- and language. In addition, the design suggestions were categorized as evidence- or expert-based. Evidence-based design suggestions include showing limited information, repeating instructions multiple times and breaking instructions into simple steps given one at a time. This research provides a first step for further collaboration between ADRD experts and designers to support the development of effective mHealth apps with high user-friendliness.

Challenges and Best Practices in Ethical Review of Human and Organizational Factors Studies in Health Technology: a Synthesis of Testimonies.

OBJECTIVE: Human and Organizational Factors (HOF) studies in health technology involve human beings and thus require Institutional Review Board (IRB) approval. Yet HOF studies have specific constraints and methods that may not fit standard regulations and IRB practices. Gaining IRB approval may pose difficulties for HOF researchers. This paper aims to provide a first overview of HOF study challenges to get IRB review by exploring differences and best practices across different countries. METHODS: HOF researchers were contacted by email to provide a testimony about their experience with IRB review and approval. Testimonies were thematically analyzed and synthesized to identify and discuss shared themes. RESULTS: Researchers from seven European countries, Argentina, Canada, Australia, and the United States answered the call. Four themes emerged that indicate shared challenges in legislation, IRB inefficiencies and inconsistencies, general regulation and costs, and lack of HOF study knowledge by IRB members. We propose a model for IRB review of HOF studies based on best practices. CONCLUSION: International criteria are needed that define low and high-risk HOF studies, to allow identification of studies that can undergo an expedited (or exempted) process from those that need full IRB review. Enhancing IRB processes in such a way would be beneficial to the conduct of HOF studies. Greater knowledge and promotion of HOF methods and evidence-based HOF study designs may support the evolving discipline. Based on these insights, training and guidance to IRB members may be developed to support them in ensuring that appropriate ethical issues for HOF studies are considered.

Evidence-Based Health Informatics as the Foundation for the COVID-19 Response: A Joint Call for Action.

BACKGROUND: As a major public health crisis, the novel coronavirus disease 2019 (COVID-19) pandemic demonstrates the urgent need for safe, effective, and evidence-based implementations of digital health. The urgency stems from the frequent tendency to focus attention on seemingly high promising digital health interventions despite being poorly validated in times of crisis. AIM: In this paper, we describe a joint call for action to use and leverage evidence-based health informatics as the foundation for the COVID-19 response and public health interventions. Tangible examples are provided for how the working groups and special interest groups of the International Medical Informatics Association (IMIA) are helping to build an evidence-based response to this crisis. METHODS: Leaders of working and special interest groups of the IMIA, a total of 26 groups, were contacted via e-mail to provide a summary of the scientific-based efforts taken to combat COVID-19 pandemic and participate in the discussion toward the creation of this manuscript. A total of 13 groups participated in this manuscript. RESULTS: Various efforts were exerted by members of IMIA including (1) developing evidence-based guidelines for the design and deployment of digital health solutions during COVID-19; (2) surveying clinical informaticians internationally about key digital solutions deployed to combat COVID-19 and the challenges faced when implementing and using them; and (3) offering necessary resources for clinicians about the use of digital tools in clinical practice, education, and research during COVID-19. DISCUSSION: Rigor and evidence need to be taken into consideration when designing, implementing, and using digital tools to combat COVID-19 to avoid delays and unforeseen negative consequences. It is paramount to employ a multidisciplinary approach for the development and implementation of digital health tools that have been rapidly deployed in response to the pandemic bearing in mind human factors, ethics, data privacy, and the diversity of context at the local, national, and international levels. The training and capacity building of front-line workers is crucial and must be linked to a clear strategy for evaluation of ongoing experiences.

The Value of Teledermoscopy to the Expertise of General Practitioners Diagnosing Skin Disorders Based on ICD-10 Coding.

Early recognition of skin cancer is vital to enhance patient outcomes. Teledermoscopy (TDsc), a telemedicine service, supports general practitioners (GPs) in gaining fast access to dermatologists' feedback to detect skin cancer. This study aimed to assess if GPs gain expertise in diagnosing skin disorders after continued use of TDsc, based on diagnosis classification by the International Statistical Classification of Diseases and Related Health Problems (ICD-10). A retrospective study was conducted on TDsc consultations sent by GPs to teledermatologists in the Netherlands (July 2015 - June 2018). GP sensitivity and confirmed cases in diagnosing skin disorders slightly increased over time. However, the total positive predictive value showed a decrease. In three years, 43 melanomas were diagnosed by the TD for which the GP did not provide a (correct) pre-diagnose. Though GPs appear to improve their expertise in skin disorder detection after continued TDsc use, TDsc remains imperative to early melanoma detection.

Is Teledermoscopy Improving General Practitioner Skin Cancer Care?

Skin cancer incidences have tripled in the Netherlands for the last twenty years and are expected to increase even more in the coming years. Teledermoscopy (TDsc) is implemented in Dutch practice to support and enhance early skin cancer detection by general practitioners (GPs) through remote consultation with dermatologists. This study assesses the effect of TDsc consultation on the quality and efficiency of skin cancer care in the primary setting by analyzing 10,184 TDsc consultations.

The Value of Radio Frequency Identification in Quality Management of the Blood Transfusion Chain in an Academic Hospital Setting.

BACKGROUND: A complex process like the blood transfusion chain could benefit from modern technologies such as radio frequency identification (RFID). RFID could, for example, play an important role in generating logistic and temperature data of blood products, which are important in assessing the quality of the logistic process of blood transfusions and the product itself. OBJECTIVE: This study aimed to evaluate whether location, time stamp, and temperature data generated in real time by an active RFID system containing temperature sensors attached to red blood cell (RBC) products can be used to assess the compliance of the management of RBCs to 4 intrahospital European and Dutch guidelines prescribing logistic and temperature constraints in an academic hospital setting. METHODS: An RFID infrastructure supported the tracking and tracing of 243 tagged RBCs in a clinical setting inside the hospital at the blood transfusion laboratory, the operating room complex, and the intensive care unit within the Academic Medical Center, a large academic hospital in Amsterdam, the Netherlands. The compliance of the management of 182 out of the 243 tagged RBCs could be assessed on their adherence to the following guidelines on intrahospital storage, transport, and distribution: (1) RBCs must be preserved within an environment with a temperature between 2°C and 6°C; (2) RBCs have to be transfused within 1 hour after they have left a validated cooling system; (3) RBCs that have reached a temperature above 10°C must not be restored or must be transfused within 24 hours or else be destroyed; (4) unused RBCs are to be returned to the BTL within 24 hours after they left the transfusion laboratory. RESULTS: In total, 4 blood products (4/182 compliant; 2.2%) complied to all applicable guidelines. Moreover, 15 blood products (15/182 not compliant to 1 out of several guidelines; 8.2%) were not compliant to one of the guidelines of either 2 or 3 relevant guidelines. Finally, 148 blood products (148/182 not compliant to 2 guidelines; 81.3%) were not compliant to 2 out of the 3 relevant guidelines. CONCLUSIONS: The results point out the possibilities of using RFID technology to assess the quality of the blood transfusion chain itself inside a hospital setting in reference to intrahospital guidelines concerning the storage, transport, and distribution conditions of RBCs. This study shows the potentials of RFID in identifying potential bottlenecks in hospital organizations' processes by use of objective data, which are to be tackled in process redesign efforts. The effect of these efforts can subsequently be evaluated by the use of RFID again. As such, RFID can play a significant role in optimization of the quality of the blood transfusion chain.

Obstacles to Successful Implementation of eHealth Applications into Clinical Practice.

eHealth can improve healthcare worldwide, and scientific research should provide evidence on the efficacy, safety and added value of such interventions. For successful implementation of eHealth interventions into clinical practice, barriers need to be anticipated. We identified seven barriers by interviewing health professionals in the Dutch healthcare system. These barriers covered three topics: financing, human factors and organizational factors. This paper discusses their potential impact on eHealth uptake. Bridging the gap between studies to assess effective eHealth interventions and their value-based implementation in healthcare is much needed.

Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: A mixed method usability approach.

INTRODUCTION: Cumbersome electronic patient record (EPR) interfaces may complicate data-entry in clinical practice. Completeness of data entered in the EPR determines, among other things, the value of computerized clinical decision support (CCDS). Quantitative usability evaluations can provide insight into mismatches between the system design model of data entry and users' data entry behavior, but not into the underlying causes for these mismatches. Mixed method usability evaluation studies may provide these insights, and thus support generating redesign recommendations for improving an EPR system's data entry interface. AIM: To improve the usability of the data entry interface of an EPR system with CCDS in the field of cardiac rehabilitation (CR), and additionally, to assess the value of a mixed method usability approach in this context. METHODS: Seven CR professionals performed a think-aloud usability evaluation both before (beta-version) and after the redesign of the system. Observed usability problems from both evaluations were analyzed and categorized using Zhang et al.'s heuristic principles of good interface design. We combined the think-aloud usability evaluation of the system's beta-version with the measurement of a new usability construct: users' deviations in action sequence from the system's predefined data entry order sequence. Recommendations for redesign were implemented. We assessed whether the redesign improved CR professionals' (1) task efficacy (with respect to the completeness of data they collected), and (2) task efficiency (with respect to the average number of mouse clicks they needed to complete data entry subtasks). RESULTS: With the system's beta version, 40% of health care professionals' navigation actions through the system deviated from the predefined next system action. The causes for these deviations as revealed by the think-aloud method mostly concerned mismatches between the system design model for data entry action sequences and users expectations of these action sequences, based on their paper-based daily routines. This caused non completion of data entry tasks (31% of main tasks completed), and more navigation actions than minimally required (146% of the minimum required). In the redesigned system the data entry navigational structure was organized in a flexible way around an overview screen to better mimic users' paper-based daily routines of collecting patient data. This redesign resulted in an increased number of completed main tasks (70%) and a decrease in navigation actions (133% of the minimum required). The think-aloud usability evaluation of the redesigned system showed that remaining problems concerned flexibility (e.g., lack of customization options) and consistency (mainly with layout and position of items on the screen). CONCLUSION: The mixed method usability evaluation was supportive in revealing the magnitude and causes of mismatches between the system design model of data-entry with users' data entry behavior. However, as both task efficacy and efficiency were still not optimal with the redesigned EPR, we advise to perform a cognitive analysis on end users' mental processes and behavior patterns in daily work processes specifically during the requirements analysis phase of development of interactive healthcare information systems.

Towards Evidence Based Usability in Health Informatics?

In a Health Information Technology (HIT) regulatory context in which the usability of this technology is more and more a critical issue, there is an increasing need for evidence based usability practice. However, a clear definition of evidence based usability practice and how to achieve it is still lacking. This paper underlines the need for evidence based HIT design and provides a definition of evidence based usability practice as the conscientious, explicit and judicious use of current best evidence in making decisions in design of interactive systems in health by applying usability engineering and usability design principles that have proven their value in practice. Current issues that hamper evidence based usability practice are highlighted and steps needed to achieve evidence are presented.

The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool.

OBJECTIVE: To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes. METHODS: Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method. RESULTS: The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool's screens. RTA detected unique issues concerning system match with subjects' language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians' query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57). CONCLUSION: In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design.

A framework for reporting on human factor/usability studies of health information technologies.

Increasingly, studies are being published on the potential negative effect of introducing poor designed Health Information Technology (HIT) into clinical settings, relating to technology-induced errors and adverse events. Academic research on HIT design and evaluation is an extremely important source of information in providing new insights into factors contributing to successful system (re)design efforts, system user-friendliness and usability issues and safety critical aspects of HIT design. However, these studies have been inconsistent and incomprehensive in their reporting, complicating the appraisal of outcomes, generalizability of study findings, meta-analysis and harmonization of the available evidence. To improve identification of type of use errors and safety related issues regarding design and implementation of HIT, consensus on issues to be reported on in scientific publications is a necessary step forward. This study presents the first approach to a framework providing a set of principles to follow for comprehensive and unambiguous reporting of HIT design and usability evaluation studies with the objective to reduce variation, improve on the publication reporting quality and proper indexation of these studies. This framework may be helpful in expanding the knowledge base not only concerning the application of Human Factors (HF)/Usability studies of HIT but also improve the knowledge base of how to (re)design and implement effective, efficient and safe HIT.