Classification and prioritization of usability problems using an augmented classification scheme.

Various methods exist for conducting usability evaluation studies in health care. But although the methodology is clear, no usability evaluation method provides a framework by which the usability reporting activities are fully standardized. Despite the frequent use of forms to report the usability problems and their context-information, this reporting is often hindered by information losses. This is due to the fact that evaluators' problem descriptions are based on individual judgments of what they find salient about a usability problem at a certain moment in time. Moreover, usability problems are typically classified in terms of their type, number, and severity. These classes are usually devised by the evaluator for the purpose at hand and the used problem types often are not mutually exclusive, complete and distinct. Also the impact of usability problems on the task outcome is usually not taken into account. Consequently, problem descriptions are often vague and even when combined with their classification in type or severity leave room for multiple interpretations when discussed with system designers afterwards. Correct interpretation of these problem descriptions is then highly dependent upon the extent to which the evaluators can retrieve relevant details from memory. To remedy this situation a framework is needed guiding usability evaluators in high quality reporting and unique classification of usability problems. Such a framework should allow the disclosure of the underlying essence of problem causes, the severity rating and the classification of the impact of usability problems on the task outcome. The User Action Framework (UAF) is an existing validated classification framework that allows the unique classification of usability problems, but it does not include a severity rating nor does it contain an assessment of the potential impact of usability flaws on the final task outcomes. We therefore augmented the UAF with a severity rating based on Nielsen's classification and added a classification for expressing the potential impact of usability problems on final task outcomes. Such an augmented scheme will provide the necessary information to system developers to understand the essence of usability problems, to prioritize problems and to tackle them in a system redesign. To investigate the feasibility of such an augmented scheme, it was applied to the results of usability studies of a computerized physician order entry system (CPOE). The evaluators classified the majority of the usability problems identically by use of the augmented UAF. In addition it helped in differentiating problems that looked similar but yet affect the user-system interaction and the task results differently and vice versa. This work is of value not only for system developers but also for researchers who want to study the results of other usability evaluation studies, because this scheme makes the results of usability studies comparable and easily retrievable.

Intention-based expressions in GASTINE.

OBJECTIVES: 1) To evaluate the design of the framework for computerized intention-based clinical practice guidelines; 2) to implement runtime features such as plan recognition and backtracking. METHOD: To evaluate the design, we implemented the heart failure guideline into GASTINE, a tool for representing and executing intention-based clinical guidelines. RESULT: Description of the current implementation of intention-based expressions in GASTINE and analysis of some generic shortcomings. Explanation of how these shortcomings are addressed. Presentation of how plan recognition and backtracking work and how they improve the system. CONCLUSION: The improved guideline system is rather flexible, i.e., it allows deviations from the guideline as long as they are in the spirit of the guideline. The recognition of actions as intended by the users facilitates a flexible decision support system. The intentions are used to explain why certain actions were suggested. Therefore it is assumed that showing the intention behind suggested actions provides a better insight into why these actions are advised.

What reasons do those with practical experience use in deciding on priorities for healthcare resources? A qualitative study.

BACKGROUND: Priority setting is necessary in current healthcare services. Discussion of fair process has highlighted the value of developing reasons for allocation decisions on the basis of experience gained from real cases. AIM: To identify the reasons that those with experience of real decision-making concerning resource allocation think relevant in deciding on the priority of a new but expensive drug treatment. METHODS: Semistructured interviews with members of committees with responsibility for making resource allocation decisions at a local level in the British National Health Service, analysed using modified grounded theory. RESULTS: 22 interviews were carried out. 14 reasons were identified. Four reasons were almost universally considered most important: cost effectiveness; clinical effectiveness; equality and gross cost. No one reason was considered dominant. Some considerations, such as political directives and fear of litigation, were thought by many participants to distort decision-making. There was a substantial lack of agreement over the relevance of some reasons, such as the absence of alternative treatment for the condition. CONCLUSIONS: There is a clear consensus on the importance and role of a limited number of reasons in allocation decisions among participants. A focus on the process of decision-making, however, does not obviate the need for those involved in the process to engage with problematical ethical issues, nor for the importance of further ethical analysis.

Evaluation of a course on patient records.

OBJECTIVES: To determine whether self-directed learning about (electronic) patient records during a PBL (problem-based learning) block, dealing with the content of disciplines concerned with the diagnosis and therapy of diseases of the abdomen, increased the knowledge of the students with respect to the patient records. METHODS: At the beginning and at the end of the ten-week block the same questionnaire was offered to the students (180). Cohen's d for effect size was used to determine the increase in knowledge. RESULTS: For those students that answered the questionnaire twice (53), a Cohen's d of 0.94 was obtained. CONCLUSIONS: The knowledge of the students concerning the advantages and limitations of (electronic) patient records increased significantly. The corresponding effect size was large.

Modeling in biomedical informatics: an exploratory analysis part 2.

OBJECTIVE: Modeling is a significant part of research, education and practice in biomedical and health informatics. Our objective was to explore which types of models of processes are used in current biomedical/health informatics research, as reflected in publications of scientific journals in this field. Also, the implications for medical informatics curricula were investigated. METHODS: Retrospective, prolective observational study on recent publications of the two official journals of the International Medical Informatics Association (IMIA), the International Journal of Medical Informatics (IJMI) and Methods of Information in Medicine (MIM). All publications of the years 2004 and 2005 from these journals were indexed according to a given list of model types. Random samples out of these publications were analysed in more depth. RESULTS: Three hundred and eighty-four publications have been analysed, 190 of IJMI and 194 of MIM. For publications in special issues (121 in IJMI) and special topics (132 in MIM) we found differences between theme-centered and conference-centered special issues/special topics (SIT) publications. In particular, we could observe a high variation between modeling in publications of theme-centered SITs. It became obvious that often sound formal knowledge as well as a strong engineering background is needed for carrying out this type of research. Usually, this knowledge and the related skills can be best provided in consecutive B.Sc. and M.Sc. programs in medical informatics (respectively, health informatics, biomedical informatics). If the focus should be primarily on health information systems and evaluation this can be offered in a M.Sc. program in medical informatics. CONCLUSIONS: In analysing the 384 publications it became obvious that modeling continues to be a major task in research, education and practice in biomedical and health informatics. Knowledge and skills on a broad range of model types are needed in biomedical/health informatics.

IMIA Accreditation of Biomedical and Health Informatics Education: Current State and Future Directions.

Objectives: The educational activities initiated by the International Medical Informatics Association (IMIA) have had global impacts and influenced national societies and local academic programs in the field of Biomedical and Health Informatics (BMHI). After the successful publication and dissemination of its educational recommendations, IMIA launched an accreditation procedure for educational programs in BMHI. The accreditation procedure was pilot tested by several BMHI academic programs in different countries and continents to obtain a global perspective. Methods: This paper presents an overview of IMIA quality assurance and accreditation procedures along with feedback on issues and problems which emerged during the pilot. Results: It appears that IMIA quality assurance and procedures worked quite well in different countries of Europe, the Middle East, South America, and Asia. These first experiences provided adequate information for adapting, modifying, and optimizing the procedures and finally for the planning of future activities. Conclusions: IMIA accreditation framework comprises a single set of standards that apply at various levels to both academic and professional BMHI programs. The pilot phase confirmed the robustness and generalizability of quality assurance standards and associated procedures on which IMIA accreditation is based at an international level.

Diagnostic reference frames for seizures: A validation study.

INTRODUCTION: We developed structured descriptions of signs and symptoms for specific seizure types (called Diagnostic Reference Frames-DRFs-by us) that can serve as a frame of reference in the process of classifying patients with epileptic seizures. In this study the validity of the DRFs for clinical use is evaluated and described. MATERIAL AND METHODS: In this study we use a decision support system based on the DRFs and using Bayes's rule for the validation of the DRFs. Patient's manifestations are entered in the decision support system and by successively applying Bayes's rule posterior probabilities are calculated. The DRFs with the highest posterior probability gives an indication of the classification of the seizure. The validation of the DRFs was performed by comparing the seizure type with the highest posterior probability with the classification of experienced epileptologists on a series of test cases with known epileptic seizures. In this way we assessed the accuracy of the DRFs in classifying patients with epileptic seizures. RESULTS: We included sixty-six patients in this efficacy study. The patients and/or their relatives described the manifestations occurring during a seizure. Sixty cases (91%) were correctly classified using the decision support system. DISCUSSION: The accuracy of 91 % indicates that the knowledge encoded in the DRFs for the included seizure types is valid. The next step is to test the DRFs in a clinical setting to evaluate the applicability in daily practice.

Modeling in biomedical informatics--an exploratory analysis (part 1).

OBJECTIVES: Modeling is a significant part of research, education and practice in biomedical and health informatics. Our objective was to explore, which types of models of processes are used in current biomedical/health informatics research, as reflected in publications of scientific journals in this field. Also the implications for medical informatics curricula were investigated. METHODS: Retrospective, prolective observational study on recent publications of the two official journals of the International Medical Informatics Association (IMIA), the International Journal of Medical Informatics (IJMI) and Methods of Information in Medicine (MIM). RESULTS: 384 publications have been analyzed, 190 of IJMI and 194 of MIM. In regular papers (69 in IJMI, 62 in MIM), analyzed here in part 1, all of these model types could be found. In many publications we observed a mixture of models, being used to solve the 'core' research questions and also to systematically evaluate the research done. Knowledge of (and models for) software engineering and project management are also often needed. IJMI seems to have a closer focus on research concerning health information systems and electronic patient records, with a strong emphasis on evaluation. MIM seems to have a broader range of research approaches, including also statistical modeling and computational intensive approaches. The aim to provide solutions for problems related to data, information and knowledge processing and to study the general principles of processing data, information and knowledge in medicine and health care in order to contribute to improve the quality of health care, and of research and education in medicine and the health sciences was given in all publications. CONCLUSIONS: Modeling continues to be a major task in research, education and practice in biomedical and health informatics. Knowledge and skills concerning a broad range of model types is needed.

Triangulation applied to Jan H. van Bemmel.

OBJECTIVE: To describe the person of Jan H. van Bemmel from different points of view. METHOD: Triangulation. RESULTS AND CONCLUSIONS: Jan H. van Bemmel successfully contributed to research and education in medical informatics. He inspired a lot of people in The Netherlands and internationally.

On exemplary scientific conduct regarding submission of manuscripts to biomedical informatics journals.

As the Editors of leading international biomedical informatics journals, the authors report on a recent pattern of improper manuscript submissions to journals in our field. As a guide for future authors, we describe ethical and pragmatic issues related to submitting work for peer-reviewed journal publication. We propose a coordinated approach to the problem that our respective journals will follow. This Editorial is being jointly published in the following journals represented by the authors: Computer Methods and Programs in Biomedicine, International Journal of Medical Informatics, Journal of Biomedical Informatics, Journal of the American Medical Informatics Association, and Methods of Information in Medicine.

EGOOZ: specifying the components of electronic patient record-related education.

OBJECTIVE: To determine whether educators consider electronic patient record (EPR)-related education necessary and if so, what subjects have to be taught more extensively in the future. METHODS: A list of possibly relevant subjects was determined from the literature. A questionnaire was designed which contained those subjects and the respondents were asked to indicate, for each subject, its competency level and required competency level in current teaching. Since the response rate was low a second questionnaire was developed to have the results of the analysis of the first questionnaire validated by a larger group of educators. RESULTS: In total 45 learning goals were identified from the literature. The questionnaire was sent to representatives of several disciplines: basic medical education, medical specializations, pharmacy, dentistry and nursing. The analysis of the first questionnaire resulted in nine subjects that needed more attention in the future. Because of the low response the needs could not be specified for the individual disciplines. This insight was obtained from a second questionnaire. The response to this questionnaire was high. From the analysis of the second questionnaire differences between views of educators involved in the training of GPs and educators involved in the training of other specializations were observed. CONCLUSION: Educators find EPR-related education important. There are different opinions about the phase in which EPR-related education should be given.

[Good medical education is impossible without computers]. / Zonder computers geen goed onderwijs in de geneeskunde.

Computers are medical instruments, for which reason they must be included in the medical curriculum. As educational tools, electronic teaching environments, multimedia applications and computers stimulate self-study. Moreover, computers make the medical literature readily accessible and facilitate the administration of the training programme. They are thus an indispensable part of medical education, and of academic and student administration.

Three-Layer Model for the design of a Protocol Support System.

OBJECTIVE: The aim of the PropeR project is to investigate the impact of Active Computerized Protocol Support (ACPS) on daily care processes in different settings (home care and hospital care). ACPS consists of an active Protocol Support System (PSS) that is linked to an Electronic Patient Record system. The aim of this paper is to describe how we have taken the organizational and social aspects into account in the hospital setting and the consequences of this approach for the design of the PSS. METHODS: Socio-technical approaches have been applied. Observations and interviews with various health care providers were performed at the hematology and oncology department of the University Hospital Maastricht. Ten extensive sessions with a specialist physician and research nurse took place to further elaborate a study protocol and to discuss how it is integrated in daily practice. The knowledge editor component of Gaston was used to build a computer interpretable version of the selected protocol. RESULTS AND CONCLUSIONS: To support the representation of a study protocol integrated in routine clinical care, a Three-Layer Model was developed. This model distinguishes the protocol description, local adaptations to the protocol and communication as three separate layers. These layers have been incorporated into the knowledge acquisition tool Gaston. The Three-Layer Model makes easy updating possible, and also supports transferability of computerized (study) protocols to other organizations.

Radiation therapy for advanced (T3T4N0-N3M0) laryngeal carcinoma: the need for a change of strategy: a radiotherapeutic viewpoint.

Since 1974 through 1984, 137 selected patients with loco-regionally advanced carcinoma of the larynx (T3T4 N0-N3) were treated primarily with quality controlled high dose megavoltage radiation therapy with salvage surgery in reserve. This policy of treatment has yielded 67% loco-regional control probability with primary radiotherapy and 85% with salvage surgery at 3 years. Fourteen patients (14/38) had moderate to severe complications after salvage surgery. The quality of life of the patients successfully treated by radiation therapy is obviously and unquestionably better. Primary radiotherapy with salvage surgery in reserve should be accepted as the strategy of choice in at least a selected group of patients with advanced laryngeal carcinoma.

Heterogeneity of stage II glottic carcinoma and its therapeutic implications.

The heterogeneity of glottic carcinoma results in variable loco-regional control probabilities from 40 to 80%. These widely different results may be due to two prognostic parameters of the tumor: the mobility of the vocal cord and the volume of the tumor. On the other hand radiation dose and technique may influence the treatment results. Stringent work-up has been prospectively undertaken in 156 patients with Stage II glottic carcinoma since 1974 through 1983 in the Academic Hospital of the Free University in Amsterdam. Several subgroups were defined from the beginning to indicate one or both of these prognostic factors. All were treated with a high quality accurate radiation therapy program delivering high dose to all patients with worse prognostic factors and even higher dosages to voluminous tumors with impaired mobility of a cord or slowly regressing tumors. The latter was individually judged for each patient towards the end of the treatment period. The loco-regional control probability of around 80% for all the subgroups irrespective of the prognostic factors indicate that the heterogeneity of Stage II glottic cancer may be influenced by high dosages of quality controlled radiotherapy. Thus results hitherto reported in literature may be further improved by the dose-schedules outlined here.

The granularity of medical narratives and its effect on the speed and completeness of information retrieval.

OBJECTIVE: Using electronic rather than paper-based record systems improves clinicians' information retrieval from patient narratives. However, few studies address how data should be organized for this purpose. Information retrieval from clinical narratives containing free text involves two steps: searching for a labeled segment and reading its content. The authors hypothesized that physicians can retrieve information better when clinical narratives are divided into many small, labeled segments ("high granularity"). DESIGN: The study tested the ability of 24 internists and 12 residents at a teaching hospital to retrieve information from an electronic medical record--in terms of speed and completeness--when using different granularities of clinical narratives. Participants solved, without time pressure, predefined problems concerning three voluminous, inpatient case records. To mitigate confounding factors, participants were randomly allocated to a sequence that was balanced by patient case and learning effect. RESULTS: Compared with retrieval from undivided notes, information retrieval from problem-partitioned notes was 22 percent faster (statistically significant), whereas retrieval from notes divided into organ systems was only 11 percent faster (not statistically significant). Subdividing segments beyond organ systems was 13 percent slower (statistically significant) than not subdividing. Granularity of medical narratives affected the speed but not the completeness of information retrieval. CONCLUSION: Dividing voluminous free-text clinical narratives into labeled segments makes patient-related information retrieval easier. However, too much subdivision slows retrieval. Study results suggest that a coarser granularity is required for optimal information retrieval than for structured data entry. Validation of these conclusions in real-life clinical practice is recommended.

Determinants of success of inpatient clinical information systems: a literature review.

We reviewed the English and Dutch literature on evaluations of patient care information systems that require data entry by health care professionals published from 1991 to 2001. Our objectives were to identify attributes that were used to assess the success of such systems and to test the ability of a framework developed by Delone and McLean for management information systems(1) to categorize these attributes correctly. The framework includes six dimensions or success factors: system quality, information quality, usage, user satisfaction, individual impact, and organizational impact. Thirty-three papers were selected for complete review. Types of study design included descriptive, correlational, comparative, and case studies. A variety of relevant attributes could be assigned to the six dimensions in the Delone and McLean framework, but some attributes, predominantly in cases of failure, did not fit any of the categories. They related to contingent factors, such as organizational culture. Our review points out the need for more thorough evaluations of patient care information systems that look at a wide range of factors that can affect the relative success or failure of these systems.

A comparison of nursing minimal data sets.

It is often argued that Nursing Minimal Data Sets (NMDSs) have advantages for the nursing profession. The NMDSs that have been developed and applied in some countries have many features in common, but there are differences in purpose, content, sampling, collection approach, and developmental stage as well. This paper examines the advantages and disadvantages of data sets of nursing practice, and the differences and similarities of five national and international NMDS systems. The purpose is to apply this information toward an NMDS initiative in the Netherlands. Future initiatives in NMDS development should include international coordination.

Maximum likelihood estimation in calibrating a stereo camera setup.

Motion and deformation of the cardiac wall may be measured by following the positions of implanted radiopaque markers in three dimensions, using two x-ray cameras simultaneously. Regularly, calibration of the position measurement system is obtained by registration of the images of a calibration object, containing 10-20 radiopaque markers at known positions. Unfortunately, an accidental change of the position of a camera after calibration requires complete recalibration. Alternatively, redundant information in the measured image positions of stereo pairs can be used for calibration. Thus, a separate calibration procedure can be avoided. In the current study a model is developed that describes the geometry of the camera setup by five dimensionless parameters. Maximum Likelihood (ML) estimates of these parameters were obtained in an error analysis. It is shown that the ML estimates can be found by application of a nonlinear least squares procedure. Compared to the standard unweighted least squares procedure, the ML method resulted in more accurate estimates without noticeable bias. The accuracy of the ML method was investigated in relation to the object aperture. The reconstruction problem appeared well conditioned as long as the object aperture is larger than 0.1 rad. The angle between the two viewing directions appeared to be the parameter that was most likely to cause major inaccuracies in the reconstruction of the 3-D positions of the markers. Hence, attempts to improve the robustness of the method should primarily focus on reduction of the error in this parameter.

To test or not to test, that is the question.

This paper describes a system that is used in Maastricht, the Netherlands, to provide general practitioners with feedback about the adequateness of their test requesting behaviour. The system has proven to be effective. Since the screening of test requests and additional patient data is work-intensive a project was started in the framework of the AIM programme OPENLABS to design a computer-based feedback system. In this system the general practitioner sends test requests to the laboratory decision support system by means of electronic data interchange (EDI). The authors were involved in the development of the EDI messages, which are now accepted standards in the Netherlands. The requests are stored in a database that is part of the laboratory decision support system. The other part of the decision support system, an expert system, obtains the request data (including patient data) from the database, screens the requests for adequateness and provides feedback when necessary.