Detectability of use errors in summative usability tests of medical devices: Impact of the test environment.

The regulations on summative usability evaluations of medical devices (MDs) emphasize that the test environment must have sufficient ecological validity for generalization to real-life use. Here, we examined the influence of environmental fidelity (a component of ecological validity) on the detectability of MD use errors. A total of 140 participants participated in a summative usability evaluation of an anaphylactic shock auto-injector device under either a high-fidelity condition or a condition acceptable from the manufacturer's perspective, lower-fidelity condition. The numbers of errors detected in each condition were compared by applying descriptive statistics and logistic and Poisson multivariate regressions. We found that the level of fidelity did not influence the overall number of use errors detected but did influence the detection of certain use errors. To optimize the test environment and increase the detection of use errors, each environmental feature's role in the test task should first be examined.

Teaching Module to Train Students in the Participatory Design of Digital Technologies for Disability Assistance.

Teaching the principles of participatory design to students mainly interested in digital skills is important because user-centered approaches have become essential in the field of new technologies when we want to guarantee that a product will meet the needs of its end-users. Working with technologies dedicated to the disability assistance was considered to be the right application domain. In 2021, ESIEE Paris, a school training students to become engineers with digital skills, created and opened a new teaching module to learn how to follow the principles of participatory design to improve the quality of a project. This paper describes the organization and the conclusions of this experience after two editions of the module.

Comparison of the validity, perceived usefulness, and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability.

OBJECTIVE: Two tools are currently available in the literature to evaluate the usability of medication alert systems, the instrument for evaluating human factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aimed to compare their convergent validity, perceived usability, usefulness, strengths, and weaknesses, as well as users' preferences. METHOD: To evaluate convergent validity, two experts mapped TEMAS' items against I-MeDeSA's items with respect to the usability dimensions they target. To assess perceived usability, usefulness, strengths, and weaknesses of both tools, staff with expertise in their medication alerting system were asked to use French versions of the TEMAS and I-MeDeSA. After the use of each tool, participants were asked to complete the System Usability Scale (SUS) and answer questions about the understandability and usefulness of each tool. Finally, participants were asked to name their preferred tool. Numeric scores were statistically compared. Free-text responses were analyzed using an inductive approach. RESULTS: Forty-five participants from 10 hospitals took part in the study. In terms of convergent validity, I-MeDeSA focuses more on the usability of the graphical user interface while TEMAS considers a wider range of usability principles. Both tools have a fair level of perceived usability (I-MeDeSA' SUS score = 61.85 and TEMAS' SUS score = 62.87), but results highlight that revisions are necessary to both tools to improve their usability. Participants found TEMAS more useful than I-MeDeSA (t = -3.63, p =.005) and had a clear preference for TEMAS to identify problems in formative evaluation (39 of 45; 0.867, p <.001) and to compare the usability of alert systems during the procurement process (36 of 45; 0.8, p <.001). CONCLUSIONS: The TEMAS is perceived as more useful and is preferred by participants. The I-MeDeSA seems more relevant for quick evaluations that focus on the graphical user interface. The TEMAS seems to be more suitable for in-depth usability evaluations of alert systems. Even if both tools are perceived to be equally usable, they suffer from wording, instructional, and organizational problems that hinder their use. The results of this study will be used to improve the design of I-MeDeSA and TEMAS.

Integration of a new technology into a work system: a case study of a smart drinking glass in French nursing homes.

PURPOSE: To reduce the risk of dehydration in older adults, the French company Auxivia has developed a smart drinking glass (SDG) that can measure the amount of water drunk. The present study looked at the various work systems (WSs) designed for use of the SDG in a nursing home. The study's objectives were to (i) determine the WSs' impact on the staff's ability to comply with the device's prerequisites and ensure the device's effective use and (ii) draw up guidelines on designing work systems. MATERIALS AND METHODS: At three nursing homes in France, two independent observers performed 9 h of observations at each site and a total of 29 interviews. RESULTS: Decisions concerning implementation and the resulting WSs have an impact on the tasks to be performed, the tasks' inherent constraints and the use of the SDG. It is essential to take account of the sociotechnical system as a whole before integrating a technology. Ideally, the introduction of an SDG will go unnoticed by staff and residents; however, our results emphasize the value of highlighting work constraints via a human factors analysis. CONCLUSIONS: It is essential to take account of sociotechnical WSs as a whole when integrating a technology.Implications for rehabilitationFor caregivers, better measurement of the amount of water drunk by older adults might help to prevent dehydration.For service providers, better knowledge of how work systems influence the nursing home staff's ability to comply with the smart drinking glass's prerequisites might improve the device implementation process.For managers, a nursing home's work systems should take account of the smart drinking glass's prerequisites on one hand and the home's environment and constraints on the other.

Improving the usability and usefulness of computerized decision support systems for medication review by clinical pharmacists: A convergent, parallel evaluation.

BACKGROUND: Computerized decision support systems (CDSSs) help hospital-based clinical pharmacists to perform medication reviews and so are promising tools for improving medication safety. However, their poor usability can reduce effectiveness and acceptability. OBJECTIVES: To evaluate the usability and perceived usefulness of a CDSS for medication review by hospital-based pharmacists and to draw up guidelines on improving its usability. METHODS: We performed a convergent, parallel evaluation. Firstly, three researchers conducted a heuristic evaluation of the CDSS. Secondly, clinical pharmacists who use the CDSS filled out the Usefulness, Satisfaction and Ease of Use (USE) questionnaire. Lastly, semi-structured interviews with the pharmacists enabled us to understand their opinions and experiences. The results of the heuristic evaluation were used to identify potential improvements in the CDSS. We performed a statistical analysis of the USE questionnaire data. Interviews were analyzed based on the unified theory of acceptance and use of technology (UTAUT), together with a task-technology fit model. The results generated by these three approaches were compared in order to determine convergences and divergences, identify challenges related to the usability and usefulness of the CDSS, and draw up guidelines for its improvement. RESULTS: Forty-seven usability problems were discovered; they variously concerned the graphical user interface, the pharmacists' needs, and the medication review model implemented in the CDSS. Only the "usefulness" dimension of the USE was not scored positively. All the UTAUT dimensions and the task-technology fit dimension emerged in the interviews. Cross-comparisons of the results from the three approaches led to the identification of four challenges and the corresponding formulation of 23 guidelines. CONCLUSIONS: The guidelines developed here should help to improve the design and acceptability of CDSSs. Hence, CDSSs will be able to assist clinical pharmacists more fully with their medication reviews and help to further improve patient safety.

Why and how can we improve patient and caregiver information for the proper use of their medical devices throughout the care pathway?

Information is generally a major pillar in the relationship of trust between the patient and their healthcare team. In the case of medical devices, which are operator-dependent and for which the proper "handling" of the device will directly affect its performance, information on the proper use becomes a major therapeutic issue. To do this, patients must have access to quality information about health products. Twenty years after the Kouchner Act introduced a right to information for patients, our panel questioned the quality of information provided to patients about the medical devices they use during their care pathway. We mapped off the information process throughout the life of the medical device (manufacturer, prescription, dispensing, use and monitoring) to identify deficiencies and potential corrective solutions. Thus, the device could respond to the 5B as for the drug (right device, right patient, right indication, right time, and right use). Seven recommendations have emerged and are proposed in this article.

The rehabilitation robot: factors influencing its use, advantages and limitations in clinical rehabilitation.

PURPOSE: Despite the proven effectiveness of rehabilitation robots (RR) in the literature, they are still little used in clinical rehabilitation. The aim of this study was to analyse the factors influencing the use of RR and the perception of therapists who used RR. METHOD: In order to characterize the factors influencing the use of RR by therapists, a semi-structured interview was conducted with 18 therapists. These interviews are based on an interview guide inspired by the Unified Theory of Acceptance and Use of Technology model. The interviews were recorded and then transcribed, summarized and finally synthesized cross-sectionally. In addition and in parallel, the System Usability Scale (SUS) was also proposed to clinicians in order to collect quantitative data. RESULTS: The interviews highlight the facilitators perceived by the therapists, such as the intensity of the movement, the complementarity with conventional rehabilitation. The results also showed the possible barriers perceived, these can be sometimes inconclusive (e.g., bugs). The SUS results show no effect, either on the gender of the users, their therapists, or the duration of use of the tool. CONCLUSION: Better communication on the functionality of the robot and the construction of achievable goals would lead to more results that are conclusive but also better patient care. To date, and despite the evidence for the effectiveness of RRs, therapists believe that there are still many barriers to their use. They agree, however, that if changes are made, RRs will become an integral part of therapy.IMPLICATIONS FOR REHABILITATIONThe study idenfied and highlighted the factors influencing the use of the rehabilitation robot in the clinics through metric and ergonomic evaluations.The study allowed to quantify the level of acceptance of the Lokomat among therapists.This study allowed to identify negative factors that could be resolved through the implementation of a structured and generalized protocol for patients and thus improve their care.

Jupyter Notebooks for Introducing Data Science to Novice Users.

Data science is a bridge discipline involving computer science, statistics, and knowledge of the health field. We developed a Jupyter Notebook to enable novice users to easily and autonomously analyze data from social networks. We conducted an experimentation with non-programmer students. They had to adapt a R Notebook and complete 14 questions and to perform descriptive analyses. The average rate of correct answers was 90.7. Jupyter Notebook enabled novice users to easily and autonomously analyze data from Twitter.

Designing Formulae for Ranking Search Results: Mixed Methods Evaluation Study.

BACKGROUND: A major factor in the success of any search engine is the relevance of the search results; a tool should sort the search results to present the most relevant documents first. Assessing the performance of the ranking formula is an important part of search engine evaluation. However, the methods currently used to evaluate ranking formulae mainly collect quantitative data and do not gather qualitative data, which help to understand what needs to be improved to tailor the formulae to their end users. OBJECTIVE: This study aims to evaluate 2 different parameter settings of the ranking formula of LiSSa (the French acronym for scientific literature in health care; Department of Medical Informatics and Information), a tool that provides access to health scientific literature in French, to adapt the formula to the needs of the end users. METHODS: To collect quantitative and qualitative data, user tests were carried out with representative end users of LiSSa: 10 general practitioners and 10 registrars. Participants first assessed the relevance of the search results and then rated the ranking criteria used in the 2 formulae. Verbalizations were analyzed to characterize each criterion. RESULTS: A formula that prioritized articles representing a consensus in the field was preferred. When users assess an article's relevance, they judge its topic, methods, and value in clinical practice. CONCLUSIONS: Following the evaluation, several improvements were implemented to give more weight to articles that match the search topic and to downgrade articles that have less informative or scientific value for the reader. Applying a qualitative methodology generates valuable user inputs to improve the ranking formula and move toward a highly usable search engine.

Aero-Sim: A User-Centred Testbed for the Evaluation of Devices as Protection from Droplets and/or Aerosol Contaminations.

Concerns over high transmission risk of SARS-CoV-2 have led to innovation and usage of an aerosol box to protect healthcare workers during airway intubation in patients with COVID-19. However, only a few studies have examined the impact of these aerosol boxes on the dispersion of droplets and aerosols, which are both thought to be significant contributors to the spread of COVID-19. In addition, to our knowledge, only a few studies have included in the concerned testbeds elements of the work context, which nevertheless have an impact on the use of the device.

Context-Sensitive Implementation of Clinical Pharmacy Missions Within an Academic Hospital.

Clinical pharmacy activities contribute to improve patient safety. Yet, the work system's characteristics influence how clinical pharmacy activities are performed and conversely clinical pharmacy causes that work system to evolve. This exploratory study aims to identify the different ways in which clinical pharmacy activities are performed in different units of a large academic hospital. Interviews and observations have been performed to identify in each ward the clinical pharmacy activities implemented and how they are carried out.

Competitive Usability Evaluation of Electronic Health Records: Preliminary Results of a Case Study.

It is necessary for hospitals to be able to compare the usability of electronic health records (EHR) before acquisition. Adding usability as a critical element of the procurement process is therefore crucial. During the competitive usability evaluation of several EHRs, the usability walkthrough method has the potential of making end-users more active in the procurement process than demonstrations. This case study presents first results of a comparison of three EHRs performed by nine representative end-users. All users uncovered usability problems while performing their scenarios. The results show that none of the EHRs evaluated is without major usability problems. These problems have been well-known to human factors researchers for a long time.

Medication review and reconciliation in older adults.

Older people are frequently exposed to polypharmacy, inappropriate prescribing, and adverse drug events. Two clinical processes can help geriatricians to optimize and increase the safety of drug prescriptions for older adults: medication reconciliation and medication review. Medication reconciliation provides the best possible medication history and identifies and resolves discrepancies in drug prescriptions. During the medication review, the best possible medication history is crosschecked against other data, including morbidities, patient's preferences, or geriatric syndromes, to produce a personalized medication strategy. Alignment of treatment recommendations with patient preferences and goals through shared decision-making is particularly important in medication review. Medication reconciliation and medication review have proven to be effective, but their broad implementation remains difficult. Indeed, these procedures are time-consuming and require specific skills, coordination between different healthcare professionals, organizations and dedicated means. The involvement of geriatricians therefore remains essential for the successful implementation of medication reconciliation and medication review in geriatric settings and among frail older people.

Human factors engineering for medical devices: European regulation and current issues.

A large proportion of the patient injuries or deaths attributable to medical device (MD) misuse can be eliminated and/or mitigated by adopting an effective human factors and ergonomics (HFE) approach. The implementation of a usability engineering process is now mandatory for MD manufacturers seeking to obtain the European Union's CE Mark. Here, we describe the European Union's HFE regulation and highlight the challenges faced by (i) manufacturers implementing this regulation and (ii) regulatory bodies charged with assessing the compliance of usability files. In Europe, 95% of MD manufacturers are small- and medium-sized enterprises; compliance with the CE Mark regulations is a real challenge to their competitiveness. Levels of knowledge about HFE vary greatly from one regulatory organization to another, which can sometimes lead to very different expectations. We also present the specific use-related risk management approach required by the HFE regulation. Lastly, we focus on the limitations of the HFE regulation for MDs and on future HFE challenges in further reducing and/or eliminating MD use errors. The main challenge is the need to go beyond technology design and the premarket assessment and to look at the postproduction stage; the coupling between an MD and a sociotechnical system can lead to consequences that were not predicted during the design process. This implies the need to consider the emerging properties of technologies in use by involving all the stakeholders.

Estimating the number of usability problems affecting medical devices: modelling the discovery matrix.

BACKGROUND: Usability testing of medical devices are mandatory for market access. The testings' goal is to identify usability problems that could cause harm to the user or limit the device's effectiveness. In practice, human factor engineers study participants under actual conditions of use and list the problems encountered. This results in a binary discovery matrix in which each row corresponds to a participant, and each column corresponds to a usability problem. One of the main challenges in usability testing is estimating the total number of problems, in order to assess the completeness of the discovery process. Today's margin-based methods fit the column sums to a binomial model of problem detection. However, the discovery matrix actually observed is truncated because of undiscovered problems, which corresponds to fitting the marginal sums without the zeros. Margin-based methods fail to overcome the bias related to truncation of the matrix. The objective of the present study was to develop and test a matrix-based method for estimating the total number of usability problems. METHODS: The matrix-based model was based on the full discovery matrix (including unobserved columns) and not solely on a summary of the data (e.g. the margins). This model also circumvents a drawback of margin-based methods by simultaneously estimating the model's parameters and the total number of problems. Furthermore, the matrix-based method takes account of a heterogeneous probability of detection, which reflects a real-life setting. As suggested in the usability literature, we assumed that the probability of detection had a logit-normal distribution. RESULTS: We assessed the matrix-based method's performance in a range of settings reflecting real-life usability testing and with heterogeneous probabilities of problem detection. In our simulations, the matrix-based method improved the estimation of the number of problems (in terms of bias, consistency, and coverage probability) in a wide range of settings. We also applied our method to five real datasets from usability testing. CONCLUSIONS: Estimation models (and particularly matrix-based models) are of value in estimating and monitoring the detection process during usability testing. Matrix-based models have a solid mathematical grounding and, with a view to facilitating the decision-making process for both regulators and device manufacturers, should be incorporated into current standards.

Implementation of an ontological reasoning to support the guideline-based management of primary breast cancer patients in the DESIREE project.

The DESIREE project has developed a platform offering several complementary therapeutic decision support modules to improve the quality of care for breast cancer patients. All modules are operating consistently with a common breast cancer knowledge model (BCKM) following the generic entity-attribute-value model. The BCKM is formalized as an ontology including both the data model to represent clinical patient information and the termino-ontological model to represent the application domain concepts. This ontological model is used to describe data semantics and to allow for reasoning at different levels of abstraction. We present the guideline-based decision support module (GL-DSS). Three breast cancer clinical practice guidelines have been formalized as decision rules including evidence levels, conformance levels, and two types of dependency, "refinement" and "complement", used to build complete care plans from the reconciliation of atomic recommendations. The system has been assessed on 138 decisions previously made without the system and re-played with the system after a washout period on simulated tumor boards (TBs) in three pilot sites. When TB clinicians changed their decision after using the GL-DSS, it was for a better decision than the decision made without the system in 75 % of the cases.

Applying a Human-Centered Design to Develop a Patient Prioritization Tool for a Pediatric Emergency Department: Detailed Case Study of First Iterations.

BACKGROUND: Overcrowding in the emergency departments has become an increasingly significant problem. Patient triage strategies are acknowledged to help clinicians manage patient flow and reduce patients' waiting time. However, electronic patient triage systems are not developed so that they comply with clinicians' workflow. OBJECTIVE: This case study presents the development of a patient prioritization tool (PPT) and of the related patient prioritization algorithm (PPA) for a pediatric emergency department (PED), relying on a human-centered design process. METHODS: We followed a human-centered design process, wherein we (1) performed a work system analysis through observations and interviews in an academic hospital's PED; (2) deduced design specifications; (3) designed a mock PPT and the related PPA; and (4) performed user testing to assess the intuitiveness of the icons, the effectiveness in communicating patient priority, the fit between the prioritization model implemented and the participants' prioritization rules, and the participants' satisfaction. RESULTS: The workflow analysis identified that the PPT interface should meet the needs of physicians and nurses, represent the stages of patient care, and contain patient information such as waiting time, test status (eg, prescribed, in progress), age, and a suggestion for prioritization. The mock-up developed gives the status of patients progressing through the PED; a strip represents the patient and the patient's characteristics, including a delay indicator that compares the patient's waiting time to the average waiting time of patients with a comparable reason for emergency. User tests revealed issues with icon intuitiveness, information gaps, and possible refinements in the prioritization algorithm. CONCLUSIONS: The results of the user tests have led to modifications to improve the usability and usefulness of the PPT and its PPA. We discuss the value of integrating human factors into the design process for a PPT for PED. The PPT/PPA has been developed and installed in Lille University Hospital's PED. Studies are carried out to evaluate the use and impact of this tool on clinicians' situation awareness and prioritization-related cognitive load, prioritization of patients, waiting time, and patients' experience.

Human Factors and Organizational Issues.

OBJECTIVE: To select the best papers that made original and high impact contributions in the area of human factors and organizational issues in biomedical informatics in 2019. METHODS: A rigorous extraction process based on queries from Web of Science® and PubMed/Medline was conducted to identify the scientific contributions published in 2019 that address human factors and organizational issues in biomedical informatics. The screening of papers on titles and abstracts independently by the two editors led to a total of 30 papers. These papers were discussed for a selection of 15 finalist papers, which were then reviewed by the two editors and by three external reviewers from internationally renowned research teams. RESULTS: The query process resulted in 626 papers that reveal interesting and rigorous methods and important studies in human factors that move the field forward, particularly in clinical informatics and emerging technologies such as brain-computer interfaces. This year three papers were clearly outstanding and help advance the field. They provide examples of applying existing frameworks together in novel and highly illuminating ways, showing the value of theory development in human factors. CONCLUSION: The selected papers make important contributions to human factors and organizational issues, expanding and deepening our knowledge of how to apply theory and applications of new technologies in health.