The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Applying GEP-HI for the planning of an evaluation study: a case study walkthrough (workshop).

A Good evaluation practice in Health Informatics (GEP-HI) Evaluation Practice guideline has been developed through a consensus making process. The guideline lists a set of 60 issues that are relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover the phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. In this seminar we walk through a case study to present how to plan a health information system evaluation study applying the good evaluation practice guideline.

Guideline for good evaluation practice in health informatics (GEP-HI).

OBJECTIVE: Development of a good practice guideline to plan and perform scientifically robust evaluation studies in health informatics. METHODS: Issues to be addressed in evaluation studies were identified and guidance drafted based on the evaluation literature and on experiences by key players. Successive drafts of the guideline were discussed in several rounds by an increasing number of experts during conferences and by e-mail. At a fairly early point the guideline was put up for comments on the web. RESULTS: Sixty issues were identified that are of potential relevance for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover all phases of an evaluation study: Preliminary outline, study design, operationalization of methods, project planning, execution and completion of the evaluation study. Issues of risk management and project control as well as reporting and publication of the evaluation results are also addressed. CONCLUSION: A comprehensive list of issues is presented as a guideline for good evaluation practice in health informatics (GEP-HI). The strengths and weaknesses of the guideline are discussed. Application of this guideline will support better handling of an evaluation study, potentially leading to a higher quality of evaluation studies. This guideline is an important step towards building stronger evidence and thus to progress towards evidence-based health informatics.

Next steps in evaluation and evidence - from generic to context-related.

INTRODUCTION: E-health systems are increasingly important and widespread, but their selection and implementation are still frequently based on belief, rather than scientific evidence, and adverse effects are not systematically addressed. Progress is being made in promoting generic evaluation methodologies as a source of scientific evidence, but effort is now needed to consider methods for special situations. METHOD: Review of five evaluation contexts - national e-health plans, telemedicine, Health Informatics 3.0, usability and economics. CONCLUSION: Identification of requirements for approaches to be developed in these five settings.

Integrated care in Norway: the state of affairs years after regulation by law.

INTRODUCTION: A mandatory multidisciplinary plan for individual care, the 'Individual care Plan', was introduced by law in Norway in 2001. The regulation was established to meet the need for improved efficiency and quality of health and social services, and to increase patient involvement. The plan was intended for patients with long-term and complex needs for coordinated care. The aim of this study was to elaborate on knowledge of such planning processes in Norwegian municipalities. METHOD: A piloted questionnaire was sent to 92 randomly selected municipalities in 2005-2006, addressing local organization and participation in the work with individual care plans. Local political governance, size of the population, funds available for health care, and problems related to living conditions were indicators for analysing the extent to which the individual care plan was used five years after the regulation was introduced. RESULTS: Our results showed that 0.5% as opposed to an expected 3% of the population had an individual care plan. This was independent of the political, social and financial situation in the municipalities or the way the planning process had been carried out. The planning process was mostly taken care of by local health and social care professionals, rather than by hospital staff and general practitioners. DISCUSSION AND CONCLUSION: The low number of care plans and the oblique responsibility among professionals for planning showed that the objectives of the national initiative had not been achieved. More research is needed to determine the reasons for this lack of success and to contribute to solutions for improved multidisciplinary cooperation.

Theories, models and frameworks for diagnosing technology-induced error.

Information technology has the potential to greatly streamline healthcare and reduce the chance of human error. However, there is a growing literature indicating that if systems are not designed adequately they may actually increase the possibility of error in the complex interaction between clinician and machine in healthcare (i.e. they may lead to technology-induced errors). In other domains the study of error has been guided by a variety of theories, models and frameworks for understanding human error. In this paper we argue for the need to consider and extend this work to the study of technology-induced error in healthcare. Insights from the software engineering, human factors and organizational behaviour literature will be described, including a set of models and frameworks that we have been using to guide our work in detecting and preventing technology-induced error in healthcare.

Mini Stare-HI: guidelines for reporting health informatics evaluations in conference papers.

BACKGROUND: To improve the quality of reports of health informatics evaluations we recently devised and published a guideline named STARE-HI, now formally endorsed by IMIA. OBJECTIVE: To develop a prioritization framework of ranked items (a mini-STARE-HI) to assist authors when reporting health informatics evaluation studies in a restricted space conference paper. METHOD: We invited 111 editors of health informatics conference proceedings and reviewers and authors of health informatics evaluation studies to score 106 reporting items on a scale ranging from "0 - not necessary" through to "10 - essential" by a web-based survey. RESULTS: The response rate for the survey was 63% (70 out of 111). The most important items (score >9) were "Interpret the data and give an answer to the study question", "Whether it is a laboratory, simulation or field study" and "Description of the outcome measure/evaluation criteria". Four items had a mean score <6. CONCLUSION: It has been possible to produce a ranking of reporting items from STARE-HI according to their prioritized relevance for inclusion in space-limited conference papers. We believe that this prioritization framework can improve quality and utility of conference papers on health informatics evaluation studies.

Patient safety through intelligent procedures in medication: the PSIP project.

The European project Patient Safety through Intelligent Procedures in medication (PSIP) aims at preventing medical errors. The objective are: (1) to facilitate the systematic production of epidemiological knowledge on Adverse Drug Events (ADE) and (2) to improve the entire medication cycle in a hospital environment. The first sub-objective is to produce knowledge on ADE: to know, as exactly as possible, per hospital, per medical department, their number, type, consequences and causes, including human factors. Data Mining of structured hospital data bases, and semantic mining of free-texts will provide a list of observed ADE, with frequencies and probabilities, thus giving a better understanding of potential risks. The second sub-objective is to develop innovative knowledge based on the mining results and to deliver professionals and patients contextualized alerts and recommendations fitting the local risk parameters. This knowledge will be implemented in a PSIP-Platform independent of existing ICT applications.

Linking informaticians and end users - using the STARE-HI evaluation reporting framework as a unifying design approach.

There is understandable concern about low uptake and sub-optimal use of health informatics systems, which is often caused by a lack of shared objectives and values by the different stakeholders. Moreover, all parties work to different ethical codes. For future success, all need to work to the same values and objectives, measured by agreed outcomes data, creating robust evidence. The Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI), by being recently endorsed by IMIA, EFMI and the EQUATOR Network, may therefore provide a generic objectives framework to help achieve common goals.

Introducing guidelines for good evaluation practice in health informatics.

Good evaluation practice guidelines have been developed through a consensus making process by a core team and the health informatics community. A set of 60 issues has been identified that is relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover all phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. Issues of risk management and project control are also addressed in the guidelines. Through application of these guidelines the general validity and generalization of evaluation studies are likely to be increased, since these guidelines aim at avoiding a number of omissions, pitfalls and risks.

STARE-HI--Statement on reporting of evaluation studies in Health Informatics.

OBJECTIVE: Development of guidelines for publication of evaluation studies of Health Informatics applications. METHODS: An initial list of issues to be addressed in reports on evaluation studies was drafted based on experiences as editors and reviewers of journals in Health Informatics and as authors of systematic reviews of Health Informatics studies, taking into account guidelines for reporting of medical research. This list has been discussed in several rounds by an increasing number of experts in Health Informatics evaluation during conferences and by using e-mail and has been put up for comments on the web. RESULTS: A set of STARE-HI principles to be addressed in papers describing evaluations of Health Informatics interventions is presented. These principles include formulation of title and abstract, of introduction (e.g. scientific background, study objectives), study context (e.g. organizational setting, system details), methods (e.g. study design, outcome measures), results (e.g. study findings, unexpected observations) and discussion and conclusion of an IT evaluation paper. CONCLUSION: A comprehensive list of principles relevant for properly describing Health Informatics evaluations has been developed. When manuscripts submitted to Health Informatics journals and general medical journals adhere to these aspects, readers will be better positioned to place the studies in a proper context and judge their validity and generalisability. It will also be possible to judge better whether papers will fit in the scope of meta-analyses of Health Informatics interventions. STARE-HI may also be used for study planning and hence positively influence the quality of evaluation studies in Health Informatics. We believe that better publication of both quantitative and qualitative evaluation studies is an important step toward the vision of evidence-based Health Informatics. LIMITATIONS: This study is based on experiences from editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the principles has not been evaluated in real practice. Only when authors start to use these principles for reporting, shortcomings in the principles will emerge.

Evaluation of robustness of a user requirements specification approach in a purchase context, a LIS case study.

OBJECTIVES: To verify the usefulness in terms of robustness of a specific User Requirements Specification Document (URD) approach in a contractual context. METHODS: The URD was specified in the users' terminology and in a hierarchical structure (strategic, tactical and operational requirements). The organisational setting concerns the purchase of a LIS for 6 laboratories, anticipated being taken into operation May 2008. RESULTS: An immediate description of the observations is that: i) amazingly few contract amendments or other changes have been necessary, ii) the budget is kept except for the labour resources within the user organisations, caused by the vendor's delayed final delivery, and iii) the functionality at present is judged to fulfil the functional needs of the user organisations. CONCLUSION: The recommendations were helpful in the users' quality assurance of the purchase and implementation process: The contractual foundation as well as the budget were indeed very stable despite several years of delays, presumably because it is based on a purpose-oriented and hence fairly timeless URD.

Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck.

BACKGROUND: Health care is entering the Information Society. It is evident that the use of modern information and communication technology offers tremendous opportunities to improve health care. However, there are also hazards associated with information technology in health care. Evaluation is a means to assess the quality, value, effects and impacts of information technology and applications in the health care environment, to improve health information applications and to enable the emergence of an evidence-based health informatics profession and practice. OBJECTIVE: In order to identify and address the frequent problems of getting evaluation understood and recognised, to promote transdisciplinary exchange within evaluation research, and to promote European cooperation, the Exploratory Workshop on "New Approaches to the Systematic Evaluation of Health Information Systems" (HIS-EVAL) was organized by the University for Health Sciences, Medical Informatics and Technology (UMIT), Innsbruck, Austria, in April 2003 with sponsorship from the European Science Foundation (ESF). METHODS: The overall program was structured in three main parts: (a). discussion of problems and barriers to evaluation; (b). defining our visions and strategies with regard to evaluation of health information systems; and (c). organizing short-term and long-term activities to reach those visions and strategies. RESULTS: The workshop participants agreed on the Declaration of Innsbruck (see ), comprising four observations and 12 recommendations with regard to evaluation of health information systems. Future activities comprise European networking as well as the development of guidelines and standards for evaluation studies. CONCLUSION: The HIS-EVAL workshop was intended to be the starting point for setting up a network of European scientists working on evaluation of health information systems, to obtain synergy effects by combining the research traditions from different evaluation fields, leading to a new dimension and collaboration on further research on information systems' evaluation.

A case study on constructive assessment of bids to a call for tender.

A case study based on constructive assessment methodology is taking place at the purchase of a Laboratory Information System. The present report describes the strategy and approach for the selection process, together with some results and lessons learned, while a parallel paper presents the tools applied.

Tools for constructive assessment of bids to a call for tender--some experiences.

The paper describes the experiences from a full-scale case study applying a number of novel assessment techniques for selecting among income bids at a call for tender, based on a User Requirements Document that comprises non-prescriptive, goal-oriented requirements.