IMIA Accreditation of Biomedical and Health Informatics Education: Current State and Future Directions.

Objectives: The educational activities initiated by the International Medical Informatics Association (IMIA) have had global impacts and influenced national societies and local academic programs in the field of Biomedical and Health Informatics (BMHI). After the successful publication and dissemination of its educational recommendations, IMIA launched an accreditation procedure for educational programs in BMHI. The accreditation procedure was pilot tested by several BMHI academic programs in different countries and continents to obtain a global perspective. Methods: This paper presents an overview of IMIA quality assurance and accreditation procedures along with feedback on issues and problems which emerged during the pilot. Results: It appears that IMIA quality assurance and procedures worked quite well in different countries of Europe, the Middle East, South America, and Asia. These first experiences provided adequate information for adapting, modifying, and optimizing the procedures and finally for the planning of future activities. Conclusions: IMIA accreditation framework comprises a single set of standards that apply at various levels to both academic and professional BMHI programs. The pilot phase confirmed the robustness and generalizability of quality assurance standards and associated procedures on which IMIA accreditation is based at an international level.

Commentaries on the IMIA Award of Excellence Lecture by Reinhold Haux.

The IMIA Yearbook editorial team asked five internationally renowned biomedical informaticians to respond to Prof. Haux's editorial. This paper summarizes their thoughts and responses. Contributions are ordered alphabetically by the contributor's last name. All authors provided an equal contribution to this manuscript.

Discussion of "Biomedical informatics: we are what we publish".

This article is part of a For-Discussion-Section of Methods of Information in Medicine about the paper "Biomedical Informatics: We Are What We Publish", written by Peter L. Elkin, Steven H. Brown, and Graham Wright. It is introduced by an editorial. This article contains the combined commentaries invited to independently comment on the Elkin et al. paper. In subsequent issues the discussion can continue through letters to the editor.

Appropriateness of ICD-coded diagnostic inpatient hospital discharge data for medical practice assessment. A systematic review.

OBJECTIVES: We performed a systematic review to investigate the quality of diagnostic hospital discharge data (DHDD) in order to gain insight in the usefulness of these data for medical practice assessment. We investigated the methods used to evaluate data quality, factors that determine data quality and its consequences for medical practice assessment. METHODS: We selected studies in which both completeness (or sensitivity: SENS) and correctness (or positive predictive value: PPV) were measured. We used the random-effects model to calculate mean SENS and PPV and to explore the effect of a number of covariates. RESULTS: The 101 included studies were very heterogeneous. We distinguished six typical study designs. We found a mean SENS of 0.67 (95%CI: 0.62-0.73) and PPV of 0.76 (95%CI: 0.73-0.79); SENS was significantly lower for comorbidity and complication studies than for some single disease studies. PPV was significantly higher for Scandinavian countries than for other countries. Recoding compared to re-abstracting of the medical record as a gold standard gave a significantly lower PPV. Diagnostic data were considered appropriate by the authors of the studies for quality of care purposes when both SENS and PPV were at least 0.85. Only 13% of the studies fulfilled this criterion. CONCLUSIONS: Variability in quality of care between settings can easily be overshadowed by variability in data quality. However, the use of DHDD by physicians to evaluate their own medical practice may be useful. But only if physicians are willing to critically interpret the meaning of the information for their medical practice assessment.

IMIA Accreditation of Health Informatics Programs.

OBJECTIVE: To develop a procedure for accrediting health informatics programs. METHODS: Development of a procedure for accreditation. Test of the accreditation procedure via a trial including four or five health informatics programs. A site visit committee consisting of three members evaluates the program based on a self-assessment report written by the program and the experiences and observations of the site visit committee during the site visit. RESULTS: A procedure for accreditation has been developed. The instructions for health informatics programs have been written and a checklist for the site visit committee members is available. In total six subjects are considered, each one consisting of one or more facets. Each facet is judged using its corresponding criterion. Five health informatics programs volunteered. One health informatics program in Finland has already been visited and a report has been produced by the site visit committee. The next site visits are in June and July 2012. The site visit in Finland showed that English summaries of master theses are not enough to get a first impression of the methods used in the thesis. A table of contents is also needed. This information then can be used to select theses written in a language other than English for discussion. CONCLUSIONS: The accreditation procedure document with instructions about writing the self-assessment report was very well structured and the instructions were clear according to the Finnish program. The site visit team could work well with the checklist. Self-assessment report model was very well structured and the instructions were clear.

Biomedical informatics--a confluence of disciplines?

BACKGROUND: Biomedical informatics is a broad discipline that borrows many methods and techniques from other disciplines. OBJECTIVE: To reflect a) on the character of biomedical informatics and to determine whether it is multi-disciplinary or inter-disciplinary; b) on the question whether biomedical informatics is more than the sum of its supporting disciplines and c) on the position of biomedical informatics with respect to related disciplines. METHOD: Inviting an international group of experts in biomedical informatics and related disciplines on the occasion of the 50th anniversary of Methods of Information in Medicine to present their viewpoints. RESULTS AND CONCLUSIONS: This paper contains the reflections of a number of the invited experts on the character of biomedical informatics. Most of the authors agree that biomedical informatics is an interdisciplinary field of study where researchers with different scientific backgrounds alone or in combination carry out research. Biomedical informatics is a very broad scientific field and still expanding, yet comprised of a constructive aspect (designing and building systems). One author expressed that the essence of biomedical informatics, as opposed to related disciplines, lies in the modelling of the biomedical content. Interdisciplinarity also has consequences for education. Maintaining rigid disciplinary structures does not allow for sufficient adaptability to capitalize on important trends nor to leverage the influences these trends may have on biomedical informatics. It is therefore important for students to become aware of research findings in related disciplines. In this respect, it was also noted that the fact that many scientific fields use different languages and that the research findings are stored in separate bibliographic databases makes it possible that potentially connected findings will never be linked, despite the fact that these findings were published. Bridges between the sciences are needed for the success of biomedical informatics.

Classification and prioritization of usability problems using an augmented classification scheme.

Various methods exist for conducting usability evaluation studies in health care. But although the methodology is clear, no usability evaluation method provides a framework by which the usability reporting activities are fully standardized. Despite the frequent use of forms to report the usability problems and their context-information, this reporting is often hindered by information losses. This is due to the fact that evaluators' problem descriptions are based on individual judgments of what they find salient about a usability problem at a certain moment in time. Moreover, usability problems are typically classified in terms of their type, number, and severity. These classes are usually devised by the evaluator for the purpose at hand and the used problem types often are not mutually exclusive, complete and distinct. Also the impact of usability problems on the task outcome is usually not taken into account. Consequently, problem descriptions are often vague and even when combined with their classification in type or severity leave room for multiple interpretations when discussed with system designers afterwards. Correct interpretation of these problem descriptions is then highly dependent upon the extent to which the evaluators can retrieve relevant details from memory. To remedy this situation a framework is needed guiding usability evaluators in high quality reporting and unique classification of usability problems. Such a framework should allow the disclosure of the underlying essence of problem causes, the severity rating and the classification of the impact of usability problems on the task outcome. The User Action Framework (UAF) is an existing validated classification framework that allows the unique classification of usability problems, but it does not include a severity rating nor does it contain an assessment of the potential impact of usability flaws on the final task outcomes. We therefore augmented the UAF with a severity rating based on Nielsen's classification and added a classification for expressing the potential impact of usability problems on final task outcomes. Such an augmented scheme will provide the necessary information to system developers to understand the essence of usability problems, to prioritize problems and to tackle them in a system redesign. To investigate the feasibility of such an augmented scheme, it was applied to the results of usability studies of a computerized physician order entry system (CPOE). The evaluators classified the majority of the usability problems identically by use of the augmented UAF. In addition it helped in differentiating problems that looked similar but yet affect the user-system interaction and the task results differently and vice versa. This work is of value not only for system developers but also for researchers who want to study the results of other usability evaluation studies, because this scheme makes the results of usability studies comparable and easily retrievable.

Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system.

OBJECTIVES: To assess the effectiveness of two usability evaluation methods, cognitive walkthrough (CW) and think aloud (TA), for identifying usability problems and to compare the performance of CW and TA in identifying different types of usability problems. METHODS: A CW was performed by two usability evaluators and 10 physicians were recruited to perform a TA usability testing of a CPOE system (Medicator). The severity of identified usability problems was determined and the usability problems were categorized based on the User Action Framework (UAF). The potential of usability problems to cause medication errors was also determined. The thoroughness, validity and effectiveness of the two methods were compared. RESULTS: Fifty seven unique usability problems of different severity, spread over the four phases of interaction as defined by the UAF, were identified. The effectiveness of the TA method for identifying usability problems was 0.08 higher than that of the CW (0.70 vs. 0.62). The thoroughness (the extent to which a method can identify existing usability problems) of the TA was higher for the "Planning" and "Assessment" phases and lower for the "Translation" phase (as defined by UAF). The thoroughness of TA for identifying problems that may potentially result in medication errors was higher than that of CW (0.81 vs. 0.68). The number of usability problems identified by each of the methods was significantly less than the total number of detected real usability problems in Medicator (p<0.001). The observed differences between the number of real usability problems identified by CW and TA (38 vs. 41), the difference between the average severity of the detected problems by CW and TA (2.37 vs. 2.41), and the difference for identifying problems potentially resulting in medication errors (15 vs. 18) were not statistically significant (p>0.4). CONCLUSIONS: This study shows that although TA showed a slightly better effectiveness, there is no significant difference between the performance of the CW and the TA methods in terms of number of usability problems identified and the mean severity of these problems. Since no single evaluation method will uncover all of the usability problems a combination of methods is advised as the most appropriate approach, especially if usability problems can lead to potentially fatal outcomes.

Clinicians satisfaction with CPOE ease of use and effect on clinicians' workflow, efficiency and medication safety.

OBJECTIVES: To study the satisfaction of end-users of a computerized physician order entry (CPOE) system concerning ease of use and the effect on users' workflow, efficiency, and medication safety and to seek users' opinions regarding required improvements of the system. Usability evaluation had shown that this system, which was in use for almost a decade, contained a number of severe usability problems. So another objective of the study was to determine whether there was a direct relation between user satisfaction and the results of a usability evaluation of the system. METHODS: Two survey questionnaires were distributed to CPOE system users (physicians and nurses) working in inpatient departments of a university hospital. Questionnaires included items that were rated using a five point Likert scale. Multiple choice questions with space for free text additions also were used to collect qualitative data concerning the use of the CPOE system and the users' opinion concerning system requirements for improvement. Data were analyzed using descriptive statistics and by the use of Mann-Whitney U and Kruskal Wallis tests. RESULTS: Two hundred seventeen physicians and 587 nurses were eligible to participate in this study (response rate 49% and 56% respectively). Physicians were satisfied with the CPOE ease of use (median 3.8, interquartile range [IQR] 3.3-4), and the effect on workflow (median 3.7, IQR 3.3-4), medication safety (median 3.75, IQR 3-4), and efficiency (median 4, IQR 3-4). Nurses also had a positive attitude towards CPOE ease of use (median 3.6, IQR 3-4), and its effect on workflow (median 3, IQR 3-3.6), medication safety (median 3, IQR 2.5-3.5), and efficiency (median 3.5, IQR 3-4). Users mainly indicated that the system needs: supplementary functionalities (e.g. alerts for allergies), improvement of current functionalities, integration with other hospital information systems and improvement of information presentation (e.g. a clear medication overview). Users did not use some current functionalities because of lack of awareness of the functionalities or having difficulty in using them. CONCLUSIONS: Users of this CPOE system, which was used for almost a decade, were satisfied with the system's ease of use and its effect on efficiency, workflow and medication safety although the system showed many usability problems and lacked some functionalities. In this case study, therefore, there seems no direct relation between the results of the earlier performed usability evaluation and user satisfaction as determined in the current study.

Long-term impact of physician encoding on detail and number of recorded diagnoses. Support by a speciality specific list of diseases.

OBJECTIVES: To improve the recording of diagnostic discharge data, pediatricians encoded diagnostic information as part of discharge letter writing supported by a pediatric list of ICD-9-CM-based codes. We evaluated the effect of this new policy on level of detail and number of recorded diagnoses. METHODS: We compared proportions of specific principal diagnoses and numbers of secondary diagnoses of the four years before with the eight years after introduction. RESULTS: Immediately after introduction, half of the diagnoses for which both generic and specific codes existed was coded specific. In later years this proportion remained stable at 0.35 (p <0.05). Diagnoses that fall under the pediatrician's own subspecialty had more often a specific code than diagnoses that do not. The mean number of secondary diagnoses per admission increased from 0.7 before introduction to 1.4 in the third year after introduction (p <0.05) but gradually fell back to 0.7. This increase and decrease was mainly due to diagnoses that did not fall under the pediatrician's own subspecialty. The extra codes in individual discharge summaries had added informational value. CONCLUSIONS: Discharge letter-linked encoding by pediatricians supported by a pediatric list of diseases leads initially to increased detail and number of diagnoses with added informational value. When attention diminishes, especially the level of detail and number of secondary diagnoses that do not fall under one's own subspecialty decrease. The level of detail of principal diagnoses remains stable because of the advantage for pediatricians of having specific diagnostic codes falling under their own subspecialty.

From clinical practice guidelines to computer-interpretable guidelines. A literature overview.

BACKGROUND: Guidelines are among us for over 30 years. Initially they were used as algorithmic protocols by nurses and other ancillary personnel. Many physicians regarded the use of guidelines as cookbook medicine. However, quality and patient safety issues have changed the attitude towards guidelines. Implementing formalized guidelines in a decision support system with an interface to an electronic patient record (EPR) makes the application of guidelines more personal and therefore acceptable at the moment of care. OBJECTIVE: To obtain, via a literature review, an insight into factors that influence the design and implementation of guidelines. METHODS: An extensive search of the scientific literature in PubMed was carried out with a focus on guideline characteristics, guideline development and implementation, and guideline dissemination. RESULTS: We present studies that enable us to explain the characteristics of high-quality guidelines, and new advanced methods for guideline formalization, computerization, and implementation. We show how the guidelines affect processes of care and the patient outcome. We discuss the reasons of low guideline adherence as presented in the literature and comment upon them. CONCLUSIONS: Developing high-quality guidelines requires a skilled team of people and sufficient budget. The guidelines should give personalized advice. Computer-interpretable guidelines (CIGs) that have access to the patient's EPR are able to give personal advice. Because of the costs, sharing of CIGs is a critical requirement for guideline development, dissemination, and implementation. Until now this is hardly possible, because of the many models in use. However, some solutions have been proposed. For instance, a standardized terminology should be imposed so that the terms in guidelines can be matched with terms in an EPR. Also, a dissemination model for easy updating of guidelines should be established. The recommendations should be based on evidence instead of on consensus. To test the quality of the guideline, appraisal instruments should be used to assess the guideline as a whole, as well as checking the quality of the recommendations individually. Only in this way optimal guideline advice can be given on an individual basis at a reasonable cost.

Intention-based expressions in GASTINE.

OBJECTIVES: 1) To evaluate the design of the framework for computerized intention-based clinical practice guidelines; 2) to implement runtime features such as plan recognition and backtracking. METHOD: To evaluate the design, we implemented the heart failure guideline into GASTINE, a tool for representing and executing intention-based clinical guidelines. RESULT: Description of the current implementation of intention-based expressions in GASTINE and analysis of some generic shortcomings. Explanation of how these shortcomings are addressed. Presentation of how plan recognition and backtracking work and how they improve the system. CONCLUSION: The improved guideline system is rather flexible, i.e., it allows deviations from the guideline as long as they are in the spirit of the guideline. The recognition of actions as intended by the users facilitates a flexible decision support system. The intentions are used to explain why certain actions were suggested. Therefore it is assumed that showing the intention behind suggested actions provides a better insight into why these actions are advised.

What reasons do those with practical experience use in deciding on priorities for healthcare resources? A qualitative study.

BACKGROUND: Priority setting is necessary in current healthcare services. Discussion of fair process has highlighted the value of developing reasons for allocation decisions on the basis of experience gained from real cases. AIM: To identify the reasons that those with experience of real decision-making concerning resource allocation think relevant in deciding on the priority of a new but expensive drug treatment. METHODS: Semistructured interviews with members of committees with responsibility for making resource allocation decisions at a local level in the British National Health Service, analysed using modified grounded theory. RESULTS: 22 interviews were carried out. 14 reasons were identified. Four reasons were almost universally considered most important: cost effectiveness; clinical effectiveness; equality and gross cost. No one reason was considered dominant. Some considerations, such as political directives and fear of litigation, were thought by many participants to distort decision-making. There was a substantial lack of agreement over the relevance of some reasons, such as the absence of alternative treatment for the condition. CONCLUSIONS: There is a clear consensus on the importance and role of a limited number of reasons in allocation decisions among participants. A focus on the process of decision-making, however, does not obviate the need for those involved in the process to engage with problematical ethical issues, nor for the importance of further ethical analysis.

Healthcare system design and parttime working doctors.

More doctors would like to work parttime. Since research on fitting healthcare system design with the structure of parttime jobs is lacking, we studied how parttime work for medical doctors could be enabled from a system design perspective. A theoretical analysis was performed, illustrated by two case studies. We conclude that introducing parttime work can provide the opportunity for improving system design and, therewith, performance. From the case studies it seems that work redesign can enable parttime work, and at the same time improve system performance. Better managing variability in the system contributed to this. The case studies results also showed that systems characterized by different levels of variability fit with different work contracts.

Maturity of teledermatology evaluation research: a systematic literature review.

BACKGROUND: There is a growing interest in teledermatology in today's clinical practice, but the maturity of the evaluation research of this technology is still unclear. OBJECTIVES: This systematic review describes the maturity of teledermatology evaluation research over time and explores what kind of teledermatology outcome measures have been evaluated. METHODS: Systematic review of literature found in Medline database (1966 up to April 2006). A telemedicine evaluation strategy consisting of four consecutive research phases (parallel to drug and diagnostics evaluation research) extended with a fifth postimplementation phase was used to classify all included studies by two independent reviewers. In addition, main characteristics (store-and-forward or real-time, study design, outcome measures) were registered. RESULTS: Three hundred and forty-five papers were systematically selected from Medline, and 244 papers were excluded. For two randomized controlled trials (RCTs), multiple papers in phase III were found. After correcting for this, 99 studies remained included (11 phase I, 72 phase II, two phase III, six phase IV, eight postimplementation phase). The number of phase II studies is the largest and still growing, while other phases are much less represented. Diagnostic accuracy was the most often used outcome measure and was found in phase I, II and IV. Store-and-forward teledermatology has been evaluated more since 2001, but most phase IV studies (RCTs, including cost aspects) are on real-time teledermatology. CONCLUSIONS: Most teledermatology evaluation studies are classified as feasibility studies (phase II). The number of phase III and IV studies remains low through the years. Compared with other specialties in telemedicine (i.e. telesurgery, telepaediatrics), teledermatology seems to be a mature application. However, more evaluation studies with a focus on clinical outcomes such as preventable referrals or time to recovery are needed to prove that teledermatology indeed is a promising and cost-saving technology.

Evaluation of a course on patient records.

OBJECTIVES: To determine whether self-directed learning about (electronic) patient records during a PBL (problem-based learning) block, dealing with the content of disciplines concerned with the diagnosis and therapy of diseases of the abdomen, increased the knowledge of the students with respect to the patient records. METHODS: At the beginning and at the end of the ten-week block the same questionnaire was offered to the students (180). Cohen's d for effect size was used to determine the increase in knowledge. RESULTS: For those students that answered the questionnaire twice (53), a Cohen's d of 0.94 was obtained. CONCLUSIONS: The knowledge of the students concerning the advantages and limitations of (electronic) patient records increased significantly. The corresponding effect size was large.

Modeling in biomedical informatics: an exploratory analysis part 2.

OBJECTIVE: Modeling is a significant part of research, education and practice in biomedical and health informatics. Our objective was to explore which types of models of processes are used in current biomedical/health informatics research, as reflected in publications of scientific journals in this field. Also, the implications for medical informatics curricula were investigated. METHODS: Retrospective, prolective observational study on recent publications of the two official journals of the International Medical Informatics Association (IMIA), the International Journal of Medical Informatics (IJMI) and Methods of Information in Medicine (MIM). All publications of the years 2004 and 2005 from these journals were indexed according to a given list of model types. Random samples out of these publications were analysed in more depth. RESULTS: Three hundred and eighty-four publications have been analysed, 190 of IJMI and 194 of MIM. For publications in special issues (121 in IJMI) and special topics (132 in MIM) we found differences between theme-centered and conference-centered special issues/special topics (SIT) publications. In particular, we could observe a high variation between modeling in publications of theme-centered SITs. It became obvious that often sound formal knowledge as well as a strong engineering background is needed for carrying out this type of research. Usually, this knowledge and the related skills can be best provided in consecutive B.Sc. and M.Sc. programs in medical informatics (respectively, health informatics, biomedical informatics). If the focus should be primarily on health information systems and evaluation this can be offered in a M.Sc. program in medical informatics. CONCLUSIONS: In analysing the 384 publications it became obvious that modeling continues to be a major task in research, education and practice in biomedical and health informatics. Knowledge and skills on a broad range of model types are needed in biomedical/health informatics.

Modeling in biomedical informatics--an exploratory analysis (part 1).

OBJECTIVES: Modeling is a significant part of research, education and practice in biomedical and health informatics. Our objective was to explore, which types of models of processes are used in current biomedical/health informatics research, as reflected in publications of scientific journals in this field. Also the implications for medical informatics curricula were investigated. METHODS: Retrospective, prolective observational study on recent publications of the two official journals of the International Medical Informatics Association (IMIA), the International Journal of Medical Informatics (IJMI) and Methods of Information in Medicine (MIM). RESULTS: 384 publications have been analyzed, 190 of IJMI and 194 of MIM. In regular papers (69 in IJMI, 62 in MIM), analyzed here in part 1, all of these model types could be found. In many publications we observed a mixture of models, being used to solve the 'core' research questions and also to systematically evaluate the research done. Knowledge of (and models for) software engineering and project management are also often needed. IJMI seems to have a closer focus on research concerning health information systems and electronic patient records, with a strong emphasis on evaluation. MIM seems to have a broader range of research approaches, including also statistical modeling and computational intensive approaches. The aim to provide solutions for problems related to data, information and knowledge processing and to study the general principles of processing data, information and knowledge in medicine and health care in order to contribute to improve the quality of health care, and of research and education in medicine and the health sciences was given in all publications. CONCLUSIONS: Modeling continues to be a major task in research, education and practice in biomedical and health informatics. Knowledge and skills concerning a broad range of model types is needed.