Forward Outlook: The Need for Evidence and for Action in Health Informatics.

Evidence-based health informatics (EBHI) is a key concept in the development and deployment of IT systems and applications in an ethical, efficient and effective health system. There is an ever growing body of knowledge to guide IT-related decision making, but further growth of this body of knowledge is required as the health IT domain and technologies are continuously evolving, leading to new functionalities and applications. However, EBHI should not wait until retrospective evidence is available - increasingly policy makers should draw on available prior and external evidence to influence design and development processes so as to ensure that health IT is devised in response to a delivery process need and not as an IT driven goal. EBHI should thereby seek to move forward significantly the metric that only some 20% of IT investment is truly successful.

A comparative review of patient safety initiatives for national health information technology.

OBJECTIVE: To collect and critically review patient safety initiatives for health information technology (HIT). METHOD: Publicly promulgated set of advisories, recommendations, guidelines, or standards potentially addressing safe system design, build, implementation or use were identified by searching the websites of regional and national agencies and programmes in a non-exhaustive set of exemplar countries including England, Denmark, the Netherlands, the USA, Canada and Australia. Initiatives were categorised by type and software systems covered. RESULTS: We found 27 patient safety initiatives for HIT predominantly dealing with software systems for health professionals. Three initiatives addressed consumer systems. Seven of the initiatives specifically dealt with software for diagnosis and treatment, which are regulated as medical devices in England, Denmark and Canada. Four initiatives dealt with blood bank and image management software which is regulated in the USA. Of the 16 initiatives directed at unregulated software, 11 were aimed at increasing standardisation using guidelines and standards for safe system design, build, implementation and use. Three initiatives for unregulated software were aimed at certification in the USA, Canada and Australia. Safety is addressed alongside interoperability in the Australian certification programme but it is not explicitly addressed in the US and Canadian programmes, though conformance with specific functionality, interoperability, security and privacy requirements may lead to safer systems. England appears to have the most comprehensive safety management programme for unregulated software, incorporating safety assurance at a local healthcare organisation level based on standards for risk management and user interface design, with national incident monitoring and a response function. CONCLUSIONS: There are significant gaps in the safety initiatives for HIT systems. Current initiatives are largely focussed on software. With the exception of diagnostic, prognostic, monitoring and treatment software, which are subject to medical device regulations in some countries, the safety of the most common types of HIT systems such as EHRs and CPOE without decision support is not being explicitly addressed in most nations. Appropriate mechanisms for safety assurance are required for the full range of HIT systems for health professionals and consumers including all software and hardware throughout the system lifecycle. In addition to greater standardisation and oversight to ensure safe system design and build, appropriate implementation and use of HIT is critical to ensure patient safety.

Applying GEP-HI for the planning of an evaluation study: a case study walkthrough (workshop).

A Good evaluation practice in Health Informatics (GEP-HI) Evaluation Practice guideline has been developed through a consensus making process. The guideline lists a set of 60 issues that are relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover the phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. In this seminar we walk through a case study to present how to plan a health information system evaluation study applying the good evaluation practice guideline.

Guideline for good evaluation practice in health informatics (GEP-HI).

OBJECTIVE: Development of a good practice guideline to plan and perform scientifically robust evaluation studies in health informatics. METHODS: Issues to be addressed in evaluation studies were identified and guidance drafted based on the evaluation literature and on experiences by key players. Successive drafts of the guideline were discussed in several rounds by an increasing number of experts during conferences and by e-mail. At a fairly early point the guideline was put up for comments on the web. RESULTS: Sixty issues were identified that are of potential relevance for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover all phases of an evaluation study: Preliminary outline, study design, operationalization of methods, project planning, execution and completion of the evaluation study. Issues of risk management and project control as well as reporting and publication of the evaluation results are also addressed. CONCLUSION: A comprehensive list of issues is presented as a guideline for good evaluation practice in health informatics (GEP-HI). The strengths and weaknesses of the guideline are discussed. Application of this guideline will support better handling of an evaluation study, potentially leading to a higher quality of evaluation studies. This guideline is an important step towards building stronger evidence and thus to progress towards evidence-based health informatics.

Next steps in evaluation and evidence - from generic to context-related.

INTRODUCTION: E-health systems are increasingly important and widespread, but their selection and implementation are still frequently based on belief, rather than scientific evidence, and adverse effects are not systematically addressed. Progress is being made in promoting generic evaluation methodologies as a source of scientific evidence, but effort is now needed to consider methods for special situations. METHOD: Review of five evaluation contexts - national e-health plans, telemedicine, Health Informatics 3.0, usability and economics. CONCLUSION: Identification of requirements for approaches to be developed in these five settings.

Mini Stare-HI: guidelines for reporting health informatics evaluations in conference papers.

BACKGROUND: To improve the quality of reports of health informatics evaluations we recently devised and published a guideline named STARE-HI, now formally endorsed by IMIA. OBJECTIVE: To develop a prioritization framework of ranked items (a mini-STARE-HI) to assist authors when reporting health informatics evaluation studies in a restricted space conference paper. METHOD: We invited 111 editors of health informatics conference proceedings and reviewers and authors of health informatics evaluation studies to score 106 reporting items on a scale ranging from "0 - not necessary" through to "10 - essential" by a web-based survey. RESULTS: The response rate for the survey was 63% (70 out of 111). The most important items (score >9) were "Interpret the data and give an answer to the study question", "Whether it is a laboratory, simulation or field study" and "Description of the outcome measure/evaluation criteria". Four items had a mean score <6. CONCLUSION: It has been possible to produce a ranking of reporting items from STARE-HI according to their prioritized relevance for inclusion in space-limited conference papers. We believe that this prioritization framework can improve quality and utility of conference papers on health informatics evaluation studies.

Linking informaticians and end users - using the STARE-HI evaluation reporting framework as a unifying design approach.

There is understandable concern about low uptake and sub-optimal use of health informatics systems, which is often caused by a lack of shared objectives and values by the different stakeholders. Moreover, all parties work to different ethical codes. For future success, all need to work to the same values and objectives, measured by agreed outcomes data, creating robust evidence. The Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI), by being recently endorsed by IMIA, EFMI and the EQUATOR Network, may therefore provide a generic objectives framework to help achieve common goals.

Introducing guidelines for good evaluation practice in health informatics.

Good evaluation practice guidelines have been developed through a consensus making process by a core team and the health informatics community. A set of 60 issues has been identified that is relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover all phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. Issues of risk management and project control are also addressed in the guidelines. Through application of these guidelines the general validity and generalization of evaluation studies are likely to be increased, since these guidelines aim at avoiding a number of omissions, pitfalls and risks.

Generic screen representations for future-proof systems, is it possible? There is more to a GUI than meets the eye.

BACKGROUND: Future-proof EHR systems must be capable of interpreting information structures for medical concepts that were not available at the build-time of the system. The two-model approach of CEN 13606/openEHR using archetypes achieves this by separating generic clinical knowledge from domain-related knowledge. The presentation of this information can either itself be generic, or require design time awareness of the domain knowledge being employed. OBJECTIVE: To develop a Graphical User Interface (GUI) that would be capable of displaying previously unencountered clinical data structures in a meaningful way. METHODS: Through "reasoning by analogy" we defined an approach for the representation and implementation of "presentational knowledge". A proof-of-concept implementation was built to validate its implementability and to test for unanticipated issues. RESULTS: A two-model approach to specifying and generating a screen representation for archetype-based information, inspired by the two-model approach of archetypes, was developed. There is a separation between software-related display knowledge and domain-related display knowledge and the toolkit is designed with the reuse of components in mind. CONCLUSIONS: The approach leads to a flexible GUI that can adapt not only to information structures that had not been predefined within the receiving system, but also to novel ways of displaying the information. We also found that, ideally, the openEHR Archetype Definition Language should receive minor adjustments to allow for generic binding.

Inter-organizational future proof EHR systems. A review of the security and privacy related issues.

OBJECTIVES: Identification and analysis of privacy and security related issues that occur when health information is exchanged between health care organizations. METHODS: Based on a generic scenario questions were formulated to reveal the occurring issues. Possible answers were verified in literature. RESULTS: Ensuring secure health information exchange across organizations requires a standardization of security measures that goes beyond organizational boundaries, such as global definitions of professional roles, global standards for patient consent and semantic interoperable audit logs. CONCLUSION: As to be able to fully address the privacy and security issues in interoperable EHRs and the long-life virtual EHR it is necessary to realize a paradigm shift from storing all incoming information in a local system to retrieving information from external systems whenever that information is deemed necessary for the care of the patient.

STARE-HI--Statement on reporting of evaluation studies in Health Informatics.

OBJECTIVE: Development of guidelines for publication of evaluation studies of Health Informatics applications. METHODS: An initial list of issues to be addressed in reports on evaluation studies was drafted based on experiences as editors and reviewers of journals in Health Informatics and as authors of systematic reviews of Health Informatics studies, taking into account guidelines for reporting of medical research. This list has been discussed in several rounds by an increasing number of experts in Health Informatics evaluation during conferences and by using e-mail and has been put up for comments on the web. RESULTS: A set of STARE-HI principles to be addressed in papers describing evaluations of Health Informatics interventions is presented. These principles include formulation of title and abstract, of introduction (e.g. scientific background, study objectives), study context (e.g. organizational setting, system details), methods (e.g. study design, outcome measures), results (e.g. study findings, unexpected observations) and discussion and conclusion of an IT evaluation paper. CONCLUSION: A comprehensive list of principles relevant for properly describing Health Informatics evaluations has been developed. When manuscripts submitted to Health Informatics journals and general medical journals adhere to these aspects, readers will be better positioned to place the studies in a proper context and judge their validity and generalisability. It will also be possible to judge better whether papers will fit in the scope of meta-analyses of Health Informatics interventions. STARE-HI may also be used for study planning and hence positively influence the quality of evaluation studies in Health Informatics. We believe that better publication of both quantitative and qualitative evaluation studies is an important step toward the vision of evidence-based Health Informatics. LIMITATIONS: This study is based on experiences from editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the principles has not been evaluated in real practice. Only when authors start to use these principles for reporting, shortcomings in the principles will emerge.

Improving the quality of the evidence base of health informatics.

Evaluation of health informatics technology has had attention from quite a few researchers in health informatics in the last few decades. In the early nineties of the past century several working groups and research projects have discussed evaluation methods and methodologies. Despite these activities, evaluation of health informatics has not received the recognition it deserves. In this presentation we will reiterate the arguments put forward in the Declaration of Innsbruck to consider evaluation an essential element of the evidence base of health informatics. Not only are evaluation studies essential, it is also required that such studies are properly reported. A joint effort of the IMIA, EFMI and AMIA working groups on evaluation has resulted in a guideline for reporting the results of evaluation studies of health informatics applications (STARE-HI). STARE-HI is currently endorsed by EFMI. The general assembly of IMIA has adopted STARE-HI as an official IMIA document. Endorsement from AMIA is being sought. A pilot study in which STARE-HI was applied to assess the quality of current reporting clearly indicates that there is quite some room for improvement. Application of guidelines such as STARE-HI would contribute to a further improvement of the evidence base of health informatics and would open the road for high quality reviews and meta-analyses.

Writing for publication in biomedical informatics.

Writing for publication can be a rewarding activity for researchers at all levels of experience. However, many students and researchers are less familiar with the various aspects of the publication process. The purpose of this workshop is to provide participants with the knowledge, skills, and practical advice that can lead to successful scientific publications.

PSI: The Dutch Academic Infrastructure for shared biobanks for translational research.

Translational research requires large patient populations. A single research institute is not able to build up such a population in a short period of time. The String of Pearls Initiative (in Dutch "Parelsnoer Initiatief", PSI) is a joint effort by the eight academic medical centers in the Netherlands to built an infrastructure for joint biobanking as to meet this challenge of establishing large collections of data and samples in relevant medical domains.

The quality of reporting of health informatics evaluation studies: a pilot study.

The quality of reporting of evaluations in Health Informatics is considered to be open for improvement. STARE-HI provides guidance for improved reporting. This study aims at establishing the quality of reporting prior to dissemination of STARE-HI. All evaluation papers in three leading Medical Informatics Journals published in 2005 have been assessed with the aid of STARE-HI. A total of 48 papers have been analysed with respect to title, abstract, keywords and study context. Main results indicate that the type of study is seldom reported in the title, that the structured abstract contains more information than an unstructured abstract, that important keywords are often forgotten. Also important information about the system assessed is often not mentioned explicitly. Our conclusion is that there is room for improvement in the reporting of evaluations in health informatics. STARE-HI can be helpful in suggesting a structure for evaluation papers in Health Informatics and issues that should be included in such papers.

Generic screen representations for future proof systems--is it possible? Two-model approach to a generic GUI.

Semantic interoperability should not only cover system interpretation of incoming information, but should be extended to include screen representation. This article describes a two-model approach to generate a screen representation for archetype-based information, which is inspired by the two-model approach used by openEHR for their archetypes. It provides a separation between software-related display knowledge and domain-related display knowledge and is designed with reuse of components in mind. This approach leads to a flexible GUI that can adapt not only to information structures that are not predefined within the receiving system and display them in a meaningful way, but also to novel ways of displaying the information. We are working on a proof of concept implementation to validate the approach.

With good intentions.

OBJECTIVE: To develop a framework for clinical practice guidelines that not only allows the representation of best practices, but also facilitates reasoning about acceptable alternatives for those best practices. METHOD: Design of an explicit representation formalism of intentions of guidelines and guideline steps. Implementation of this representation and reasoning mechanisms in GASTON, a tool for representing and executing computerized clinical guidelines. RESULTS: The developed formalism is used to represent a heart failure clinical guideline. It is demonstrated that the representation of intentions provides the needed flexibility to avoid unnecessary errors and warnings. CONCLUSION: By using an explicit representation of the intentions of guidelines and guideline steps, a flexible decision support system can be built that does not check only the adherence to the formal guideline but evaluates clinical activities in light of the intention of the guidelines.

With good Intentions.

The use of intentions in computer-based guidelines may help to make them more flexible, easier to adapt to local standards, easier to evaluate and to improve. We see possibilities of using intentions in several areas: as a generator of actions which could be performed and then compared with the decision of the physician, as a method for pruning proposed actions based on current and previous patient data. In this contribution we present our approach to intention-based guidelines.