Innovative Power of Health Care Organisations Affects IT Adoption: A bi-National Health IT Benchmark Comparing Austria and Germany.

Multinational health IT benchmarks foster cross-country learning and have been employed at various levels, e.g. OECD and Nordic countries. A bi-national benchmark study conducted in 2007 revealed a significantly higher adoption of health IT in Austria compared to Germany, two countries with comparable healthcare systems. We now investigated whether these differences still persisted. We further studied whether these differences were associated with hospital intrinsic factors, i.e. the innovative power of the organisation and hospital demographics. We thus performed a survey to measure the "perceived IT availability" and the "innovative power of the hospital" of 464 German and 70 Austrian hospitals. The survey was based on a questionnaire with 52 items and was given to the directors of nursing in 2013/2014. Our findings confirmed a significantly greater IT availability in Austria than in Germany. This was visible in the aggregated IT adoption composite score "IT function" as well as in the IT adoption for the individual functions "nursing documentation" (OR = 5.98), "intensive care unit (ICU) documentation" (OR = 2.49), "medication administration documentation" (OR = 2.48), "electronic archive" (OR = 2.27) and "medication" (OR = 2.16). "Innovative power" was the strongest factor to explain the variance of the composite score "IT function". It was effective in hospitals of both countries but significantly more effective in Austria than in Germany. "Hospital size" and "hospital system affiliation" were also significantly associated with the composite score "IT function", but they did not differ between the countries. These findings can be partly associated with the national characteristics. Indicators point to a more favourable financial situation in Austrian hospitals; we thus argue that Austrian hospitals may possess a larger degree of financial freedom to be innovative and to act accordingly. This study is the first to empirically demonstrate the effect of "innovative power" in hospitals on health IT adoption in a bi-national health IT benchmark. We recommend directly including the financial situation into future regression models. On a political level, measures to stimulate the "innovative power" of hospitals should be considered to increase the digitalisation of healthcare.

Implementation of the Austrian Nursing Minimum Data Set (NMDS-AT): A Feasibility Study.

BACKGROUND: An Austrian Nursing Minimum Data Set (NMDS-AT) has been developed to describe the diversity of patient populations and variability of nursing care based on nursing diagnoses, nursing interventions, and nursing outcomes. The aim of this study is to test the feasibility of using this NMDS-AT by assessing the availability of data needed for the NMDS-AT in routine nursing documentation, and to assess its reliability and usefulness. METHODS: Data were collected in a general hospital from patient records of 20 patients representing 457 patient days. Availability of needed data was assessed by two raters in a chart review based on an NMDS-AT form. The interrater reliability (n=20) and intrarater reliability (n=5) was assessed using Cohen's kappa coefficient and intraclass correlation coefficient (ICC). Usefulness was assessed by verifying whether typical analysis questions can be answered by the documented NMDS-AT data. RESULTS: In the 20 patient records, thirteen nursing diagnoses, 50 nursing interventions, and five nursing outcomes occurred, representing 68 (58.6%) of the overall 116 data elements of the NMDS-AT. The data were found at different data sources (e.g., electronic nursing record or paper-based fever chart) and in various forms (e.g., standardized or free text). The interrater reliability of the thirteen nursing diagnoses showed kappa values (percentage of agreement) ranging from 0.35 (85%) to 1.00 (100%). The 50 nursing interventions showed ICCs ranging from 0.03 to 1.00. All nursing outcomes showed an ICC of 1.00. The intrarater reliability showed 100% agreement. Performing typical analysis questions showed that the extracted NMDS-AT data are able to answer questions of clinical management, of policy makers, and of nursing science. CONCLUSIONS: The NMDS-AT was found to be feasible: needed data was available in the analysed patient records, data extraction showed good reliability, and typical analysis could be performed and showed interesting results. Before the NMDS-AT can be introduced in healthcare institutions, the following challenges need to be addressed: 1. improve the quality of nursing documentation; 2. reduce fragmentation of documentation; 3. use a standardized nursing classification system; and 4. establish mappings between nursing classification systems and the NMDS-AT.

Exploring Opportunities for Clinical Data Warehouse Enhancement Through Data Catalog Integration.

Secondary use of clinical health data implies a prior integration of mostly heterogenous and multidimensional data sets. A clinical data warehouse addresses the technological and organizational framework conditions required for this, by making any data available for analysis. However, users of a data warehouse often do not have a comprehensive overview of all available data and only know about their own data in their own systems - a situation which is also referred to as 'data siloed state'. This problem can be addressed and ultimately solved by implementation of a data catalog. Its core function is a search engine, which allows for searching the metadata collected from different data sources and thereby accessing all data there is. With this in mind, we conducted an explorative online market survey followed by vendor comparison as a pre-requisite for system selection of a data catalog. Assessment of vendor performance was based on seven predetermined and weighted selection criteria. Although three vendors achieved the highest score, results were lying closely together. Detailed investigations and test installations are needed for further narrowing down the selection process.

Unveiling the Surge: A Comprehensive Analysis of the Stimulating Impact of the Covid-19 Pandemic on Telemedicine and Its Academic Landscape.

The Covid-19 pandemic spurred an unprecedented shift towards digitalization, prompting a surge in telehealth practices. This paper explores the impact of the pandemic on telemedicine through a comprehensive analysis of scientific publications. Utilizing a bibliometric approach, the study examines trends in telemedicine research before and after the onset of Covid-19. The systematic search in PubMed yielded 8,454 pre-Covid-19 publications (2016-2019) and 16,633 post-Covid-19 publications (2020-2023). A total of 21,989 distinct keywords were extracted. Co-occurrence maps reveal evolving thematic clusters, with "mhealth" and "ehealth" dominating pre-Covid-19, while "Covid-19" emerges as a top keyword post-pandemic. The Top-10 keywords shift post-Covid-19, reflecting dynamic research priorities. The bibliometric approach illuminates a heightened exploration of telehealth solutions post-pandemic, emphasizing the enduring impact of the crisis on academic discourse. Changes in key terms and shifts in key term ranking indicate dynamic research priorities and a broader consideration of multidimensional healthcare challenges. Acknowledging study limitations, the analysis offers a high-level perspective, focusing on authors' keywords. Despite challenges, the study provides a systematic overview, revealing the emergence of new telemedicine application domains and the need for further in-depth analyses. Future research directions may explore the ecological impact of telemedicine applications and other intriguing aspects, contributing to a comprehensive understanding of telemedicine's scholarly trajectory.

Understanding IT in Healthcare: Relevance and Training Needs of IT in Private Medical Practice.

BACKGROUND: The integration of Information Technology (IT) into private medical practice is crucial in modern healthcare. Physicians managing office-related IT without proper knowledge risk operational inefficiencies and security. OBJECTIVES: This study determines the relevance of specific IT topics in medical practice and identifies the training needs of physicians for enhancing IT competencies in healthcare. METHODS: In March 2023 a cross-sectional online survey was conducted with physicians comprising nine IT-related topics in Tyrol, Austria. RESULTS: The survey results highlighted a strong perceived relevance and high demand for IT education among physicians working in their medical practice, especially in areas of core medical IT and security. The majority of responses indicated high relevance (76.7%) and high demand (69.7%) for IT topics in medical practice. CONCLUSION: The findings underscore a significant need for targeted IT training and support in medical practices, particularly in areas related to the medical practice and security. Addressing these needs could lead to improved healthcare delivery and better management of technological resources in the healthcare sector.

Quality and Effectiveness of AI Tools for Students and Researchers for Scientific Literature Review and Analysis.

This study scrutinizes free AI tools tailored for supporting literature review and analysis in academic research, emphasizing their response to direct inquiries. Through a targeted keyword search, we cataloged relevant AI tools and evaluated their output variation and source validity. Our results reveal a spectrum of response qualities, with some tools integrating non-academic sources and others depending on outdated information. Notably, most tools showed a lack of transparency in source selection. Our study highlights two key limitations: the exclusion of commercial AI tools and the focus solely on tools that accept direct research queries. This raises questions about the potential capabilities of paid tools and the efficacy of combining various AI tools for enhanced research outcomes. Future research should explore the integration of diverse AI tools, assess the impact of commercial tools, and investigate the algorithms behind response variability. This study contributes to a better understanding of AI's role in academic research, emphasizing the importance of careful selection and critical evaluation of these tools in academic endeavors.

Clinical evaluation of the ADE scorecards as a decision support tool for adverse drug event analysis and medication safety management.

AIMS: The prevention of adverse drug events (ADEs) demands co-ordination of different health care professionals. ADE scorecards are a novel approach to raise the team awareness regarding ADE risks and causes. It makes information on numbers and on possible causes of possible ADE cases available to the clinical team. The aim of the study was to investigate the usage and acceptance of ADE scorecards by healthcare professionals and their impact on rates of possible ADEs. METHODS: ADE scorecards were introduced in three departments of a French hospital. A controlled time series analysis of ADE data was conducted to assess the impact of the ADE scorecards. In addition, qualitative interviews and a standardized survey with all participating staff members were performed. RESULTS: Physicians, nurses and pharmacists found ADE scorecards effective to increase medication safety and recommended future usage. The time-series analysis did not show changes in rates of possible ADEs. CONCLUSION: ADE scorecards appear to be useful to raise awareness of ADE-related issues among professionals. Although the evaluation did not show significant reductions of ADE rates, the participating physicians, nurses and pharmacists believed that the ADE scorecards could contribute to increased patient safety and to a reduction in ADE rates. Strategies need to be designed to integrate ADE scorecards better into the clinical routine and to increase the precision of ADE detection.

Parenteral diclofenac infusion significantly decreases brain-tissue oxygen tension in patients with poor-grade aneurysmal subarachnoid hemorrhage.

INTRODUCTION: Diclofenac, a nonsteroidal antiinflammatory drug, is commonly used as antipyretic therapy in intensive care. The purpose of this study was to investigate the effects of parenteral diclofenac infusion on brain homeostasis, including brain-tissue oxygen tension (PbtO2) and brain metabolism after aneurysmal subarachnoid hemorrhage (aSAH). METHODS: We conducted a prospective, observational study with retrospective analysis of 21 consecutive aSAH patients with multimodal neuromonitoring. Cerebral perfusion pressure (CPP), mean arterial pressure (MAP), intracranial pressure (ICP), body temperature, and PbtO2 were analyzed after parenteral diclofenac infusion administered over a 34-minute period (20 to 45 IQR). Data are given as mean ± standard error of mean and median with interquartile range (IQR), as appropriate. Time-series data were analyzed by using a general linear model extended by generalized estimation equations (GEEs). RESULTS: One-hundred twenty-three interventions were analyzed. Body temperature decreased from 38.3°C ± 0.05°C by 0.8°C ± 0.06°C (P < 0.001). A 10% decrease in MAP and CPP (P < 0.001) necessitated an increase of vasopressors in 26% (n = 32), colloids in 33% (n = 41), and crystalloids in 5% (n = 7) of interventions. PbtO2 decreased by 13% from a baseline value of 28.1 ± 2.2 mm Hg, resulting in brain-tissue hypoxia (PbtO2 <20 mm Hg) in 38% (n = 8) of patients and 35% (n = 43) of interventions. PbtO2 <30 mm Hg before intervention was associated with brain-tissue hypoxia after parenteral diclofenac infusion (likelihood ratio, 40; AUC, 93%; 95% confidence interval (CI), 87% to 99%; P < 0.001). Cerebral metabolism showed no significant changes after parenteral diclofenac infusion. CONCLUSIONS: Parenteral diclofenac infusion after aSAH effectively reduces body temperature, but may lead to CPP decrease and brain-tissue hypoxia, which were both associated with poor outcome after aSAH.

Telemedicine Research from Big Bang to 2022.

The Clinical Information Systems (CIS) section of the IMIA Yearbook of Medical Informatics systematically screens about 2,500 publications from more than 1,000 journals annually to find the best CIS publications. The editors of the CIS section have noticed a trend toward patient-centered care supported by AI and machine learning and increased research in cross-institutional data sharing, particularly in telemedicine. As a result, they adjusted their search query to include the MeSH term "telemedicine." As a preliminary step and to get a sense of the historical development of telemedicine research activity, they performed a bibliometric analysis of all previously published papers in PubMed indexed with the tag "Telemedicine" as MeSH Major Topic. They retrieved 29,289 publications from 1976 to 2022 and used their titles and abstracts to create a bibliometric network that visualizes the most relevant terms, their frequency and relationship to each other, and the chronological sequence of their publication. The development over time also shows a clear move toward patient-centeredness. Interestingly, the term "Covid," which has only recently come into use, takes on a central role in the network.

Transforming Clinical Information Systems: Empowering Healthcare through Telemedicine, Data Science, and Artificial Intelligence Applications.

OBJECTIVE: In this synopsis, the editors of the Clinical Information Systems (CIS) section of the IMIA Yearbook of Medical Informatics overview recent research and propose a selection of best papers published in 2022 in the CIS field. METHODS: The editors follow a systematic approach to gather relevant articles and select the best papers for the section. This year, they updated the query to incorporate the topic of telemedicine and removed search terms related to geographic information systems. The revised query resulted in a larger number of identified papers, necessitating the appointment of a third section editor to handle the increased workload. The editors narrowed the initial pool of articles to 15 candidate papers through a multi-stage selection process. At least seven independent reviews were collected for each candidate paper, and a selection meeting with the IMIA Yearbook editorial board led to the final selection of the best papers for the CIS section. RESULTS: The query was carried out in mid-January 2023 and retrieved a deduplicated result set of 5,206 articles from 1,500 journals. This year, 15 papers were nominated as candidates, and four were finally selected as the best papers in the CIS section.Including telemedicine in the query resulted in a substantial increase in the number of papers found. The analysis highlights the growing convergence between clinical information systems and telemedicine, with mobile health (mHealth) technologies and data science applications gaining prominence. The selected candidate papers emphasize the practical impact of research efforts, focusing on patient-centric outcomes and benefits, including intelligent mobile health monitoring systems and AI-assisted decision-making in healthcare. CONCLUSIONS: Looking ahead, the field of CIS is expected to continue evolving, driven by advances in telemedicine, mHealth technologies, data science, and AI integration, leading to more efficient, patient-oriented, and intelligent healthcare systems and overall improvement of global healthcare outcomes.

Making a Virtue of Necessity - A Highly Structured Clinical Data Warehouse as the Source of Assured Truth in a Hospital.

Data-driven decision-making in health care is becoming increasingly important in daily clinical use. A data warehouse, storing all the clinically relevant information in a highly structured way, is a primary basis for achieving this goal. We are developing a clinical data warehouse where more than 20 years of clinical data can be persisted, and newly generated data from different sources can be integrated. A back room was created to store all hospital information system data in a PostgreSQL database. Due to the enormous number of diverse forms in the hospital information system, a broker service was developed that integrates the individual data sources into the data warehouse as soon as they are released for storage. The front room represents the interface from the infrastructure to the targeted analysis. Database query and visualization tools or business intelligence tools can display and analyze processed and interleaved data. In all areas of business and medicine, structured and quality-adjusted data is of major importance. With the help of a clinical data warehouse system, it is possible to perform patient-centered analyses and thus realize optimal therapy. Furthermore, it is possible to provide staff and management with dashboards for control purposes.

The Surviral Epidemiological Early Warning System Tyrol Dashboard - System, Functionality and Findings.

BACKGROUND: Dashboards provide a good retrospective view of the development of the disease. Yet, current COVID-related dashboards typically lack the capability to predict future trends. However, this is important for health policy makers and health care providers in order to adopt meaningful containment strategies. OBJECTIVES: The aim of this paper is to present the Surviral dashboard, which allows the effective monitoring of infectious disease dynamics. METHODS: The presented dashboard comprises a wide range of information, including retrospective and prognostic data based on an agent-based simulation framework. It served as the basis for informed decision-making and planning of disease control strategies within the federal state of Tyrol. RESULTS: By visualizing the information in an understandable format, the dashboard provided a comprehensive overview of the COVID-19 situation in Tyrol and allowed for the identification of trends and patterns. CONCLUSION: The presented dashboard is a valuable tool for managing pandemics such as COVID-19. It provides a convenient and efficient way to monitor the spread of a disease and identify potential areas for intervention.

Physicians' perceptions on the usefulness of contextual information for prioritizing and presenting alerts in Computerized Physician Order Entry systems.

BACKGROUND: One possible approach towards avoiding alert overload and alert fatigue in Computerized Physician Order Entry (CPOE) systems is to tailor their drug safety alerts to the context of the clinical situation. Our objective was to identify the perceptions of physicians on the usefulness of clinical context information for prioritizing and presenting drug safety alerts. METHODS: We performed a questionnaire survey, inquiring CPOE-using physicians from four hospitals in four European countries to estimate the usefulness of 20 possible context factors. RESULTS: The 223 participants identified the 'severity of the effect' and the 'clinical status of the patient' as the most useful context factors. Further important factors are the 'complexity of the case' and the 'risk factors of the patient'. CONCLUSIONS: Our findings confirm the results of a prior, comparable survey inquiring CPOE researchers. Further research should focus on implementing these context factors in CPOE systems and on subsequently evaluating their impact.

"All Quiet on the Western Front" - Clinical Information Systems Research in the Year 2021.

OBJECTIVES: In this synopsis, we give an overview of recent research and propose a selection of best papers published in 2021 in the field of Clinical Information Systems (CIS). METHOD: As CIS section editors, we annually apply a systematic process to retrieve articles for the IMIA Yearbook of Medical Informatics. For eight years now, we use the same query to find relevant publications in the CIS field. Each year we retrieve more than 2,400 papers which we categorize in a multi-pass review to distill a preselection of up to 15 candidate papers. External reviewers and yearbook editors then assess the selected candidate papers. Based on the review results, the IMIA Yearbook editorial board chooses up to four best publications for the section at a selection meeting. To get a comprehensive overview of the content of the retrieved articles, we use text mining and term co-occurrence mapping techniques. RESULTS: We carried out the query in mid-January 2022 and retrieved a deduplicated result set of 2,688 articles from 1,062 different journals. This year, we nominated ten papers as candidates and finally selected two of them as the best papers in the CIS section. As in the previous years, the content analysis of the articles revealed the broad spectrum of topics covered by CIS research, but - on the other side - no real innovations or new upcoming research trends. However, the significant impact of COVID-19 on CIS research was observable also this year. CONCLUSIONS: The trends in CIS research, as seen in recent years, continue to be observable. The content analysis revealed nothing really new in the CIS domain. What was very visible was the impact of the COVID-19 pandemic, which still effects our lives and also CIS.

Medical informatics and digital health multilingual ontology (MIMO): A tool to improve international collaborations.

BACKGROUND: Even if English is the leading language for international communication, it is essential to keep in mind that research runs at the local level by local teams generally communicating in their local/national language, especially in Europe among European projects. OBJECTIVE: Therefore, the European Federation for Medical Informatics - Working Group on Health Informatics for Inter-regional Cooperation" has one objective: To develop a multilingual ontology focusing on Health Informatics and Digital Health as a collaboration tool that improves international and, in particular, European collaborations. RESULTS: We have developed the Medical Informatics and Digital Health Multilingual Ontology (MIMO). Hosted on the Health Terminology/Ontology Portal (HeTOP), MIMO contains around 1,000 concepts, 460 MeSH Descriptors, 220 MeSH Concepts, and more than 300 newly created concepts. MIMO is continuously updated to comprise as recent as possible concepts and their translations in more than 30 languages. Moreover, the MIMO's development team constantly improves MIMO content and supporting information. Thus, during workshop discussions and one-on-one exchanges, the MIMO team has collected domain experts' opinions about the community's interests and suggestions for future enhancements. Moreover, MIMO will be integrated to support the annotation and categorization of research products into the HosmartAI European project involving more than 20 countries around Europe and worldwide. CONCLUSION: MIMO is hosted by HeTOP (Health Terminology/Ontology Portal), which integrates 100 terminologies and ontologies in 55 languages. MIMO is freely available online. MIMO is portable to other knowledge platforms as part of MIMO's main aims to facilitate communication between medical librarians, translators, and researchers as well as to support students' self-learning.

The status of IT service management in health care - ITIL® in selected European countries.

BACKGROUND: Due to the strained financial situation in the healthcare sector, hospitals and other healthcare providers are facing an increasing pressure to improve their efficiency and to reduce costs. These trends challenge health care organizations to introduce innovative information technology (IT) based supportive processes. To guarantee that IT supports the clinical processes perfectly, IT must be managed proactively. However, until now, there is only very few research on IT service management especially on ITIL® implementations in the health care context. METHODS: The current study aims at exploring knowledge about and acceptance of IT service management (especially ITIL®) in hospitals in Austria and its neighboring regions Bavaria (Germany), Slovakia, South Tyrol (Italy) and Switzerland. Therefore highly standardized interviews with the respective head of information technology (CIO, IT manager) were conducted for selected hospitals from the different regions. In total 75 hospitals were interviewed. Data gathered was analyzed using descriptive statistics and where necessary methods of qualitative content analysis. RESULTS: In most regions, two-thirds or more of the participating IT managers claim to be familiar with the concepts of IT service management and of ITIL®. IT managers expect from ITIL® mostly better IT services, followed by an increased productivity and a reduction of IT cost. But only five hospitals said to have implemented at least parts of ITIL®, and eight hospitals stated to be planning to do this in the next two years. When it comes to ITIL®, Switzerland and Bavaria seem to be ahead of the other countries. There, the highest levels of knowledge, the highest number of implementations or plans of an implementation as well as the highest number of ITIL® certified staff members were observed. CONCLUSION: The results collected through this study indicate that the idea of IT services and IT service management is still not widely recognized in hospitals in the countries and regions of the study. It is also indicated that hospitals need further assistance in order to be able to successfully implement ITIL®. Overall, research on IT service management and ITIL® in health care is rare.

Development of a context model to prioritize drug safety alerts in CPOE systems.

BACKGROUND: Computerized physician order entry systems (CPOE) can reduce the number of medication errors and adverse drug events (ADEs) in healthcare institutions. Unfortunately, they tend to produce a large number of partly irrelevant alerts, in turn leading to alert overload and causing alert fatigue. The objective of this work is to identify factors that can be used to prioritize and present alerts depending on the 'context' of a clinical situation. METHODS: We used a combination of literature searches and expert interviews to identify and validate the possible context factors. The internal validation of the context factors was performed by calculating the inter-rater agreement of two researcher's classification of 33 relevant articles. RESULTS: We developed a context model containing 20 factors. We grouped these context factors into three categories: characteristics of the patient or case (e.g. clinical status of the patient); characteristics of the organizational unit or user (e.g. professional experience of the user); and alert characteristics (e.g. severity of the effect). The internal validation resulted in nearly perfect agreement (Cohen's Kappa value of 0.97). CONCLUSION: To our knowledge, this is the first structured attempt to develop a comprehensive context model for prioritizing drug safety alerts in CPOE systems. The outcome of this work can be used to develop future tailored drug safety alerting in CPOE systems.

Scorecards: a new method to prevent adverse drug events? Preliminary results from a clinical field study.

In the field of the detection and prevention of preventable ADEs, several methods have been explored to decrease the rate of ADEs due to monitoring errors. This paper describes an innovative method that aims at improving patient safety by increasing ADEs' awareness of healthcare professionals. To this end, ADE-scorecards that provide healthcare professionals with retrospective data about ADEs' causes and rates have been developed. In order to evaluate the impact of this method on the ADE rate, in-field clinical tests have been set up. Data were collected by both qualitative (semi-structured interviews) and quantitative methods (log analysis and ADE rate calculation). Preliminary results reveal that ADE-scorecards are well-accepted by most of the healthcare professionals who intend to use them as discussion supports and/or learning tools. Thus, ADE-scorecards seem to be a relevant method to improve patient safety by increasing ADE-awareness of healthcare professionals.

Validation of completeness, correctness, relevance and understandability of the PSIP CDSS for medication safety.

Medication errors and resulting Adverse Drug Events (ADEs) are an important issue of global healthcare. Within the European PSIP project that aims at developing solutions to improve medication safety, contextualized decision support modules aiming to prevent ADEs are being developed. The objective of this work was to thoroughly validate part of the CDSS (Clinical Decision Support System) and the underlying Knowledge Base, in order to detect incorrect or unclear alerts. We systematically developed a repository of test cases and used them for validation. The development of the test cases showed that there are differences among experts in interpreting the correctness of an alert, and that the clinical context is important when judging whether it is adequate. Overall, validation did not find major errors in the Knoweldge Base, but developed several recommendations for further improvement.

Contextualization of automatic alerts during electronic prescription: researchers' and users' opinions on useful context factors.

Computerized Physician Order Entry (CPOE) Systems can reduce the number of medication errors and Adverse Drug Events (ADEs). However, studies have shown that users often override alerts, as they feel these are too unspecific for the given patient context. It is unclear, however, how alerts could be contextualized, that is adapted to the clinical context. Based on a literature search, we developed a list of 20 possible context factors. We asked 69 international CPOE researchers and 120 physicians from four hospitals in two countries to judge the usefulness of each factor. Researchers judged the following factors as most important: 1.) Severity of the effect, 2.) Clinical status of the patient, 3.) Probability of occurrence, 4.) Risk factors of the patient, 5.) Strength of evidence. Physicians judged the following factors as most important: Severity of the effect, clinical status of the patients, complexity of the case, and class of drug. These top-ranked context factors could be used to re-design the way alerts are presented in CPOE systems, to increase sensitivity of alerts, to reduce overriding rates, and to improve medication safety.