Testing Usability of a Medical Device Using Virtual Reality.

Before commercialization of a medical device, it is important to evaluate its usability. Traditional methods such as user testing to evaluate usability of medical device encountered difficulties to put participants in simulation conditions that are sufficiently realistic to be representative of real life. Virtual reality can be used to immerse participants in a high-fidelity simulation at a lower cost, but is not widely used today. This feasibility study aims to compare the results obtained between user tests in a real simulated environment and in a virtual reality environment, with feedback on the advantages and disadvantages of both conditions.

Evaluations of medical device usability during clinical investigations: a scoping review of clinical study protocols.

OBJECTIVE: Combining clinical investigations with usability studies provides valuable information for medical devices evaluation. But both types of study are very different in terms of objectives and methodologies. How are usability studies integrated into clinical investigations in practice? METHODS: We searched the ClinicalTrials.gov database for clinical investigation protocols that included usability outcome(s) and analyzed them. RESULTS: 77 study protocols were identified for the analysis, including 102 outcomes related to usability in total. The most frequently assessed outcomes were satisfaction (53/102) and ease of use (33/102). The questionnaire was the most frequently planned technique (85/102) followed by interviews (24/102). Other methods were used, such as observation (9/102), mostly when the end users was a healthcare professional, and diary (6/102), mostly with patients. CONCLUSION: Our study results showed that the collection of usability data can be included in a clinical investigation, with various levels of investment. Resource-light, rapid integration via a questionnaire will enable the collection of subjective data on the users' perceptions. When more resources are available, observation in accessible environments can be set up (especially during use by healthcare professionals in hospital) or interviews and/or diaries for home-based environments (especially by patients).

Designing Formulae for Ranking Search Results: Mixed Methods Evaluation Study.

BACKGROUND: A major factor in the success of any search engine is the relevance of the search results; a tool should sort the search results to present the most relevant documents first. Assessing the performance of the ranking formula is an important part of search engine evaluation. However, the methods currently used to evaluate ranking formulae mainly collect quantitative data and do not gather qualitative data, which help to understand what needs to be improved to tailor the formulae to their end users. OBJECTIVE: This study aims to evaluate 2 different parameter settings of the ranking formula of LiSSa (the French acronym for scientific literature in health care; Department of Medical Informatics and Information), a tool that provides access to health scientific literature in French, to adapt the formula to the needs of the end users. METHODS: To collect quantitative and qualitative data, user tests were carried out with representative end users of LiSSa: 10 general practitioners and 10 registrars. Participants first assessed the relevance of the search results and then rated the ranking criteria used in the 2 formulae. Verbalizations were analyzed to characterize each criterion. RESULTS: A formula that prioritized articles representing a consensus in the field was preferred. When users assess an article's relevance, they judge its topic, methods, and value in clinical practice. CONCLUSIONS: Following the evaluation, several improvements were implemented to give more weight to articles that match the search topic and to downgrade articles that have less informative or scientific value for the reader. Applying a qualitative methodology generates valuable user inputs to improve the ranking formula and move toward a highly usable search engine.

How to interact with medical terminologies? Formative usability evaluations comparing three approaches for supporting the use of MedDRA by pharmacovigilance specialists.

BACKGROUND: Medical terminologies are commonly used in medicine. For instance, to answer a pharmacovigilance question, pharmacovigilance specialists (PVS) search in a pharmacovigilance database for reports in relation to a given drug. To do that, they first need to identify all MedDRA terms that might have been used to code an adverse reaction in the database, but terms may be numerous and difficult to select as they may belong to different parts of the hierarchy. In previous studies, three tools have been developed to help PVS identify and group all relevant MedDRA terms using three different approaches: forms, structured query-builder, and icons. Yet, a poor usability of the tools may increase PVS' workload and reduce their performance. This study aims to evaluate, compare and improve the three tools during two rounds of formative usability evaluation. METHODS: First, a cognitive walkthrough was performed. Based on the design recommendations obtained from this evaluation, designers made modifications to their tools to improve usability. Once this re-engineering phase completed, six PVS took part in a usability test: difficulties, errors and verbalizations during their interaction with the three tools were collected. Their satisfaction was measured through the System Usability Scale. The design recommendations issued from the tests were used to adapt the tools. RESULTS: All tools had usability problems related to the lack of guidance in the graphical user interface (e.g., unintuitive labels). In two tools, the use of the SNOMED CT to find MedDRA terms hampered their use because French PVS were not used to it. For the most obvious and common terms, the icons-based interface would appear to be more useful. For the less frequently used MedDRA terms or those distributed in different parts of the hierarchy, the structured query-builder would be preferable thanks to its great power and flexibility. The form-based tool seems to be a compromise. CONCLUSION: These evaluations made it possible to identify the strengths of each tool but also their weaknesses to address them before further evaluation. Next step is to assess the acceptability of tools and the expressiveness of their results to help identify and group MedDRA terms.

Modeling Keyword Search Strategy: Analysis of Pharmacovigilance Specialists' Search of MedDRA Terms.

In the information retrieval task, searching and choosing keywords to form the query is crucial. The present study analyzes and describes the keywords' search strategy into a thesaurus in the field of pharmacovigilance. Two ergonomics experts shadowed 22 pharmacovigilance specialists during their daily work. They focus on the strategies for searching and choosing MedDRA terms to build pharmacovigilance queries. Interviews of four pharmacovigilance specialists completed the observations. Results highlight that, for unusual or complex searches, pharmacovigilance specialists proceed iteratively in three main phases: (i) preparation of a list of terms and of evaluation criteria, (ii) exploration of the MedDRA hierarchy and choice of a term, and (iii) evaluation of the results against the criteria. Overall, the search and the choice of keywords within a thesaurus shares similarity with the information retrieval task and is closely interwoven with the query building process. Based on the results, the paper proposes design specifications for new interfaces supporting the identification of MedDRA terms so that pharmacovigilance reports searches achieve a good level of expressiveness.

Medication Review: Human Factors Study Aiming at Helping an Acute Geriatric Unit to Sustain and Systematize the Process.

BACKGROUND: Medication Review (MRev) has been implemented in many hospitals to improve patient safety and well-being. However, it seems sometimes difficult to implement, maintain and systematize this process, especially when key-elements are absent. This study focuses on the analysis of a MRev process implemented in an Acute Geriatric Unit (AGU) which, at the time of the study, had no Computerized Physician Order Entry (CPOE) and no sufficient staff to - normally - support the process. OBJECTIVE: This study describes the MRev process as existing in the AGU with a particular focus on the preparatory MRev meeting phase and presents our recommendations to maintain and optimize it. METHODS: Human Factor experts have collected and analyzed data during MRev process by interviews, shadowing observations and video recording from April to October 2014 at Lille University Hospital. RESULTS: MRev process consists of three phases (meeting preparation, MRev meeting and patient discharge) and includes seven main tasks for which actors, documented supports, outcomes and difficulties are identified. Although allocating a fulltime pharmacist for the AGU would solve several problems, the main realistic recommendations concern training for junior and senior actors according to their roles and the improvement of some tasks processes. CONCLUSION: Despite less than optimal conditions as compared to those recommended by the literature, the observed AGU performs an efficient review based on well designed tools and processes.

Work System Characteristics Impacting the Performance and Quality of the Discharge Letter Process.

Studies on the impact of a Health Information Technology seldom consider socio-technical characteristics of the work system in which the technology is implemented. Yet those dimensions may act as hidden variables that could explain the inconsistency of impact studies' results in terms of performance, quality and satisfaction. This paper reports on the identification of those variables in the discharge letter (DL) process. Human Factors experts performed an analysis of the work system of the DL process in 17 medical units. The DL process is composed of three sub-processes running with work system differing according to the distribution of tasks, the technology implemented and the work organization. Hidden variables identified are: verification by the physician, technology's integration, number of editing cycles, physicians' preferences etc. Those variables can be collected automatically or by questionnaire. Statistical analyses will have to be performed to know which variable explain impact indicators.