How the reliability and validity of the patient classification system can be ensured in daily nursing work? A follow-up study.

AIMS AND OBJECTIVES: This study aimed to determine the reliability and validity of the RAFAELA patient classification system (PCS) for qualified and efficient nurses. BACKGROUND: The number of patients per nurse or diagnosis-based determination of nursing workload are imprecise measures that do not consider the variation in patients' care needs. Ensuring the reliability and validity of the RAFAELA is important for the efficient allocation of nursing resources. METHODS: In this study, we investigated how the maintenance (parallel classification measurement and professional assessment of optimal nursing care intensity level measurement) of the RAFAELA was done with 9 years of follow-up data. The results were analysed using quantitative methods supplemented with qualitative audit descriptions. The STROBE checklist was used. RESULTS: The RAFAELA was used continuously in 44 units (40%). The length of use of the RAFAELA influenced the success of parallel classification measurements. Six per cent of units passed parallel classification measurement over 75% after 1-3 years' use, 42% after 4-6 years and 83% after 7-9 years. Among the units that used the RAFAELA PCS continuously, only four (9%) passed the professional assessment of optimal nursing care intensity level measurement. CONCLUSIONS: This study shows that ensuring the reliability and validity of the use of the RAFAELA is laborious, requires several years of use and continuous investments in nurses' skills and motivation. RELEVANCE TO CLINICAL PRACTICE: Qualified use of PCS is challenging, and organisations should invest to maintenance, training, support and user motivation. Each patient should be classified comprehensively, and nursing resources should be calculated correctly. In addition, utilisation of the nursing intensity level should be maximised. CLINICAL TRIAL REGISTRATION NUMBER: Kuopio University Hospital organisation permit number 73/2014. PATIENT OR PUBLIC CONTRIBUTION: Information regarding individual patients or nurses was not available to the researchers. All materials are in the form of summary tables.

The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

A conceptual framework and principles for trusted pervasive health.

BACKGROUND: Ubiquitous computing technology, sensor networks, wireless communication and the latest developments of the Internet have enabled the rise of a new concept-pervasive health-which takes place in an open, unsecure, and highly dynamic environment (ie, in the information space). To be successful, pervasive health requires implementable principles for privacy and trustworthiness. OBJECTIVE: This research has two interconnected objectives. The first is to define pervasive health as a system and to understand its trust and privacy challenges. The second goal is to build a conceptual model for pervasive health and use it to develop principles and policies which can make pervasive health trustworthy. METHODS: In this study, a five-step system analysis method is used. Pervasive health is defined using a metaphor of digital bubbles. A conceptual framework model focused on trustworthiness and privacy is then developed for pervasive health. On that model, principles and rules for trusted information management in pervasive health are defined. RESULTS: In the first phase of this study, a new definition of pervasive health was created. Using this model, differences between pervasive health and health care are stated. Reviewed publications demonstrate that the widely used principles of predefined and static trust cannot guarantee trustworthiness and privacy in pervasive health. Instead, such an environment requires personal dynamic and context-aware policies, awareness, and transparency. A conceptual framework model focused on information processing in pervasive health is developed. Using features of pervasive health and relations from the framework model, new principles for trusted pervasive health have been developed. The principles propose that personal health data should be under control of the data subject. The person shall have the right to verify the level of trust of any system which collects or processes his or her health information. Principles require that any stakeholder or system collecting or processing health data must support transparency and shall publish its trust and privacy attributes and even its domain specific policies. CONCLUSIONS: The developed principles enable trustworthiness and guarantee privacy in pervasive health. The implementation of principles requires new infrastructural services such as trust verification and policy conflict resolution. After implementation, the accuracy and usability of principles should be analyzed.

Applying GEP-HI for the planning of an evaluation study: a case study walkthrough (workshop).

A Good evaluation practice in Health Informatics (GEP-HI) Evaluation Practice guideline has been developed through a consensus making process. The guideline lists a set of 60 issues that are relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover the phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. In this seminar we walk through a case study to present how to plan a health information system evaluation study applying the good evaluation practice guideline.

Usability of electronic nursing record systems: definition and results from an evaluation study in Finland.

Information technologies (IT) are widely used in healthcare, however, little is known about the usability of nursing information systems. This article reports an evaluation study that aimed at researching the usability of four electronic nursing record (ENR) systems and thereby providing guidelines for further IT development. For the purposes of the study the concept of usability was defined to cover the following aspects: nurse-computer interaction in working context, information exchange, and collaboration between healthcare professionals. The study utilized two usability research methods, contextual inquiry and expert review, and was conducted with 18 nurses in Finland. Study results showed that the ENR systems share several usability problems in common, most of them relating to the efficiency of use, intuitiveness, and poor fit for multi-professional needs. Nurses had mainly negative experiences on documenting practices with ENRs: documentation requires a lot of resources, patient information is hard to find, and procedures do not meet the contextual needs. These findings suggest usability problems having significant effects on nurses' documentation practices and nursing work.

Framework model and principles for trusted information sharing in pervasive health.

Trustfulness (i.e. health and wellness information is processed ethically, and privacy is guaranteed) is one of the cornerstones for future Personal Health Systems, ubiquitous healthcare and pervasive health. Trust in today's healthcare is organizational, static and predefined. Pervasive health takes place in an open and untrusted information space where person's lifelong health and wellness information together with contextual data are dynamically collected and used by many stakeholders. This generates new threats that do not exist in today's eHealth systems. Our analysis shows that the way security and trust are implemented in today's healthcare cannot guarantee information autonomy and trustfulness in pervasive health. Based on a framework model of pervasive health and risks analysis of ubiquitous information space, we have formulated principles which enable trusted information sharing in pervasive health. Principles imply that the data subject should have the right to dynamically verify trust and to control the use of her health information, as well as the right to set situation based context-aware personal policies. Data collectors and processors have responsibilities including transparency of information processing, and openness of interests, policies and environmental features. Our principles create a base for successful management of privacy and information autonomy in pervasive health. They also imply that it is necessary to create new data models for personal health information and new architectures which support situation depending trust and privacy management.

Next steps in evaluation and evidence - from generic to context-related.

INTRODUCTION: E-health systems are increasingly important and widespread, but their selection and implementation are still frequently based on belief, rather than scientific evidence, and adverse effects are not systematically addressed. Progress is being made in promoting generic evaluation methodologies as a source of scientific evidence, but effort is now needed to consider methods for special situations. METHOD: Review of five evaluation contexts - national e-health plans, telemedicine, Health Informatics 3.0, usability and economics. CONCLUSION: Identification of requirements for approaches to be developed in these five settings.

Mini Stare-HI: guidelines for reporting health informatics evaluations in conference papers.

BACKGROUND: To improve the quality of reports of health informatics evaluations we recently devised and published a guideline named STARE-HI, now formally endorsed by IMIA. OBJECTIVE: To develop a prioritization framework of ranked items (a mini-STARE-HI) to assist authors when reporting health informatics evaluation studies in a restricted space conference paper. METHOD: We invited 111 editors of health informatics conference proceedings and reviewers and authors of health informatics evaluation studies to score 106 reporting items on a scale ranging from "0 - not necessary" through to "10 - essential" by a web-based survey. RESULTS: The response rate for the survey was 63% (70 out of 111). The most important items (score >9) were "Interpret the data and give an answer to the study question", "Whether it is a laboratory, simulation or field study" and "Description of the outcome measure/evaluation criteria". Four items had a mean score <6. CONCLUSION: It has been possible to produce a ranking of reporting items from STARE-HI according to their prioritized relevance for inclusion in space-limited conference papers. We believe that this prioritization framework can improve quality and utility of conference papers on health informatics evaluation studies.

Towards a National Health Information System evaluation.

Most EU Member States have a documented policy on eHealth. Documented follow-up and evaluation strategies for assessing whether national level systems have reached their set aims and outcomes are, however, rare. Methodologies for large scale information system assessment and evaluation are poorly established. This article describes the approach used to generate the Finnish National Health Information System (NHIS) evaluation plan. The core elements of the plan are illustrated, discussing also challenges and solutions in implementation. The article is based on NHIS evaluation planning project [15] and its presentation in the MIE workshop in Sarajevo in 2009 [16], where core issues and challenges of large-scale evaluations were discussed using the Finnish NHIS evaluation plan as a frame of reference.The Finnish plan offers other countries tools with which to assess their own plans and generate national methodologies for NHIS evaluation.

The role of architecture and ontology for interoperability.

Turning from organization-centric to process-controlled or even to personalized approaches, advanced healthcare settings have to meet special interoperability challenges. eHealth and pHealth solutions must assure interoperability between actors cooperating to achieve common business objectives. Hereby, the interoperability chain also includes individually tailored technical systems, but also sensors and actuators. For enabling corresponding pervasive computing and even autonomic computing, individualized systems have to be based on an architecture framework covering many domains, scientifically managed by specialized disciplines using their specific ontologies in a formalized way. Therefore, interoperability has to advance from a communication protocol to an architecture-centric approach mastering ontology coordination challenges.

Approaching harmonised medication information management through process and information modelling.

Medication information should be available in a correct content and format and where and when needed in medical care. In this paper process and information models are presented in order to identify the items, their relations, contents, sources and use contexts of medication information. The motivation for this study has been to develop a harmonized information model for medication information to overcome current problems and pitfalls in medication information management and information accessibility.

Linking informaticians and end users - using the STARE-HI evaluation reporting framework as a unifying design approach.

There is understandable concern about low uptake and sub-optimal use of health informatics systems, which is often caused by a lack of shared objectives and values by the different stakeholders. Moreover, all parties work to different ethical codes. For future success, all need to work to the same values and objectives, measured by agreed outcomes data, creating robust evidence. The Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI), by being recently endorsed by IMIA, EFMI and the EQUATOR Network, may therefore provide a generic objectives framework to help achieve common goals.

Introducing guidelines for good evaluation practice in health informatics.

Good evaluation practice guidelines have been developed through a consensus making process by a core team and the health informatics community. A set of 60 issues has been identified that is relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover all phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. Issues of risk management and project control are also addressed in the guidelines. Through application of these guidelines the general validity and generalization of evaluation studies are likely to be increased, since these guidelines aim at avoiding a number of omissions, pitfalls and risks.

Planning for national health information system evaluation.

Most EU member states have a documented policy on eHealth. Documented follow-up and evaluation policies to assess reaching of the set aims, as well as evaluating outcomes of implemented systems at a national level are, however, rare. Methodologies for large scale information system assessment and evaluation are poorly established. In the workshop, the Finnish evaluation plans for the National Health Information System (NHIS) are used as a case in the workshop to reflect on core issues and challenges in large-scale evaluation for supporting system development, implementation and positive impacts. The results of the discussions are documented to be used in further refinement of the Finnish evaluation methodology and for enhancing networking of respective parties in different countries. The results will also benefit participants including policy makers, developers and researchers of national eHealth systems in pursuit of national evaluation activities.

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications.

OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

Definition, structure, content, use and impacts of electronic health records: a review of the research literature.

PURPOSE: This paper reviews the research literature on electronic health record (EHR) systems. The aim is to find out (1) how electronic health records are defined, (2) how the structure of these records is described, (3) in what contexts EHRs are used, (4) who has access to EHRs, (5) which data components of the EHRs are used and studied, (6) what is the purpose of research in this field, (7) what methods of data collection have been used in the studies reviewed and (8) what are the results of these studies. METHODS: A systematic review was carried out of the research dealing with the content of EHRs. A literature search was conducted on four electronic databases: Pubmed/Medline, Cinalh, Eval and Cochrane. RESULTS: The concept of EHR comprised a wide range of information systems, from files compiled in single departments to longitudinal collections of patient data. Only very few papers offered descriptions of the structure of EHRs or the terminologies used. EHRs were used in primary, secondary and tertiary care. Data were recorded in EHRs by different groups of health care professionals. Secretarial staff also recorded data from dictation or nurses' or physicians' manual notes. Some information was also recorded by patients themselves; this information is validated by physicians. It is important that the needs and requirements of different users are taken into account in the future development of information systems. Several data components were documented in EHRs: daily charting, medication administration, physical assessment, admission nursing note, nursing care plan, referral, present complaint (e.g. symptoms), past medical history, life style, physical examination, diagnoses, tests, procedures, treatment, medication, discharge, history, diaries, problems, findings and immunization. In the future it will be necessary to incorporate different kinds of standardized instruments, electronic interviews and nursing documentation systems in EHR systems. The aspects of information quality most often explored in the studies reviewed were the completeness and accuracy of different data components. It has been shown in several studies that the use of an information system was conducive to more complete and accurate documentation by health care professionals. The quality of information is particularly important in patient care, but EHRs also provide important information for secondary purposes, such as health policy planning. CONCLUSION: Studies focusing on the content of EHRs are needed, especially studies of nursing documentation or patient self-documentation. One future research area is to compare the documentation of different health care professionals with the core information about EHRs which has been determined in national health projects. The challenge for ongoing national health record projects around the world is to take into account all the different types of EHRs and the needs and requirements of different health care professionals and consumers in the development of EHRs. A further challenge is the use of international terminologies in order to achieve semantic interoperability.

Requirements for user-friendly personal health information systems.

This paper presents background of personal health systems research and analyses requirements for personal health systems from various perspectives. The analysis shows that social and contextual aspects of personal health systems need more attention in the future and that systems development should not be technology-driven but driven by user needs and use contexts.

The Safe and Effective Use of Shared Data Underpinned by Stakeholder Engagement and Evaluation Practice.

OBJECTIVES: The paper draws attention to: i) key considerations involving the confidentiality, privacy, and security of shared data; and ii) the requirements needed to build collaborative arrangements encompassing all stakeholders with the goal of ensuring safe, secure, and quality use of shared data. METHOD: A narrative review of existing research and policy approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Care and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: The technological ability to merge, link, re-use, and exchange data has outpaced the establishment of policies, procedures, and processes to monitor the ethics and legality of shared use of data. Questions remain about how to guarantee the security of shared data, and how to establish and maintain public trust across large-scale shared data enterprises. This paper identifies the importance of data governance frameworks (incorporating engagement with all stakeholders) to underpin the management of the ethics and legality of shared data use. The paper also provides some key considerations for the establishment of national approaches and measures to monitor compliance with best practice. CONCLUSION: Data sharing endeavours can help to underpin new collaborative models of health care which provide shared information, engagement, and accountability amongst all stakeholders. We believe that commitment to rigorous evaluation and stakeholder engagement will be critical to delivering health data benefits and the establishment of collaborative models of health care into the future.

Green Turning Brown - Domain Engineering for Social and Health Services in Finland.

Being able to design information systems to an untouched domain, without the burden of existing information systems, especially legacy systems, is often seen as a dream of most information system professionals. Uncharted domains are anyway scarce, and often such greenfield projects turn into brownfield projects, also to projects where existing structures severely constrain the development of new systems. In this article we discuss the concepts of greenfield and brownfield domain engineering and software development, and reflect their possible messages to the re-engineering of the Finnish health- and social care ecosystem currently under way. In our fieldwork we could identify a lot of need and wish for greenfield domain engineering in the Finnish health and social services delivery. As well we found a lot of brownfield elements inhibiting change. Our proposal for the future is a ecosystem approach, where new and established elements could live together in a self-governed balance.

How to Teach Health IT Evaluation: Recommendations for Health IT Evaluation Courses.

Systematic health IT evaluation studies are needed to ensure system quality and safety and to provide the basis for evidence-based health informatics. Well-trained health informatics specialists are required to guarantee that health IT evaluation studies are conducted in accordance with robust standards. Also, policy makers and managers need to appreciate how good evidence is obtained by scientific process and used as an essential justification for policy decisions. In a consensus-based approach with over 80 experts in health IT evaluation, recommendations for the structure, scope and content of health IT evaluation courses on the master or postgraduate level have been developed, supported by a structured analysis of available courses and of available literature. The recommendations comprise 15 mandatory topics and 15 optional topics for a health IT evaluation course.