Cancer as a death sentence: developing an initial program theory for an IVR intervention.

To address current trends in poor health-seeking behaviour and late cancer diagnosis in many low- and middle-income countries, like Uganda, it is important to explore innovative awareness building interventions. One possible intervention is a common digital format, an interactive voice response (IVR) system, which is suitable for individuals with low technological and reading literacy. It is increasingly acknowledged that developing digital interventions requires co-creation with relevant stakeholders and explication of program developers' assumptions, to make them effective, sustainable, and scalable. To this end, we sought to develop an initial program theory for a co-created IVR system for cancer awareness in Uganda. Utilising principles of the realist approach, a qualitative exploratory study was conducted through seven focus group discussions (FGDs) with people living with cancer (PLWC), health workers, and policy makers. Thematic analysis of the transcripts resulted in the emergence of four major themes. Through all themes the most consistent finding was that myths, misconceptions, and misinformation about cancer were related to every aspect of the cancer journey and influenced the experiences and lives of PLWC and their caregivers. Participants were positive about the potential of an IVR system but also had reservations about the design and reach of the system. The resulting initial program theory proposes that a context-specific IVR system has the potential to improve awareness on cancer, provided attention is given to aspects such as language, message framing, and accuracy.

The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals.

OBJECTIVES: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them. METHODS: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings. RESULTS: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time. CONCLUSIONS: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Managing Pandemic Responses with Health Informatics - Challenges for Assessing Digital Health Technologies.

OBJECTIVES: To highlight the role of technology assessment in the management of the COVID-19 pandemic. METHOD: An overview of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: Evaluation of digital health technologies for COVID-19 should be based on their technical maturity as well as the scale of implementation. For mature technologies like telehealth whose efficacy has been previously demonstrated, pragmatic, rapid evaluation using the complex systems paradigm which accounts for multiple sociotechnical factors, might be more suitable to examine their effectiveness and emerging safety concerns in new settings. New technologies, particularly those intended for use on a large scale such as digital contract tracing, will require assessment of their usability as well as performance prior to deployment, after which evaluation should shift to using a complex systems paradigm to examine the value of information provided. The success of a digital health technology is dependent on the value of information it provides relative to the sociotechnical context of the setting where it is implemented. CONCLUSION: Commitment to evaluation using the evidence-based medicine and complex systems paradigms will be critical to ensuring safe and effective use of digital health technologies for COVID-19 and future pandemics. There is an inherent tension between evaluation and the imperative to urgently deploy solutions that needs to be negotiated.

Assessment of organizational readiness to implement an electronic health record system in a low-resource settings cancer hospital: A cross-sectional survey.

BACKGROUND: Organizational readiness for change is a key factor in success or failure of electronic health record (EHR) system implementations. Readiness is a multifaceted and multilevel abstract construct encompassing individual and organizational aspects, which makes it difficult to assess. Available tools for assessing readiness need to be tested in different contexts. OBJECTIVE: To identify and assess relevant variables that determine readiness to implement an EHR in oncology in a low-and-middle income setting. METHODS: At the Uganda Cancer Institute (UCI), a 100-bed tertiary oncology center in Uganda,we conducted a cross-sectional survey using the Paré model. This model has 39 indicator variables (Likert-scale items) for measuring 9 latent variables that contribute to readiness. We analyzed data using partial least squares structural equation modeling (PLS-SEM). In addition, we collected comments that we analyzed by qualitative content analysis and sentiment analysis as a way of triangulating the Likert-scale survey responses. RESULTS: One hundred and forty-six clinical and non-clinical staff completed the survey, and 116 responses were included in the model. The measurement model showed good indicator reliability, discriminant validity, and internal consistency. Path coefficients for 6 of the 9 latent variables (i.e. vision clarity, change appropriateness, change efficacy, presence of an effective champion, organizational flexibility, and collective self-efficacy) were statistically significant at p < 0.05. The R2 for the outcome variable (organizational readiness) was 0.67. The sentiments were generally positive and correlated well with the survey scores (Pearson's r = 0.73). Perceived benefits of an EHR included improved quality, security and accessibility of clinical data, improved care coordination, reduction of errors, and time and cost saving. Recommended considerations for successful implementation include sensitization, training, resolution of organizational conflicts and computer infrastructure. CONCLUSION: Change management during EHR implementation in oncology in low- and middle- income setting should focus on attributes of the change and the change targets, including vision clarity, change appropriateness, change efficacy, presence of an effective champion, organizational flexibility, and collective self-efficacy. Particularly, issues of training, computer skills of staff, computer infrastructure, sensitization and strategic implementation need consideration.

The performance of acute versus antecedent patient characteristics for 1-year mortality prediction during intensive care unit admission: a national cohort study.

BACKGROUND: Multiple factors contribute to mortality after ICU, but it is unclear how the predictive value of these factors changes during ICU admission. We aimed to compare the changing performance over time of the acute illness component, antecedent patient characteristics, and ICU length of stay (LOS) in predicting 1-year mortality. METHODS: In this retrospective observational cohort study, the discriminative value of four generalized mixed-effects models was compared for 1-year and hospital mortality. Among patients with increasing ICU LOS, the models included (a) acute illness factors and antecedent patient characteristics combined, (b) acute component only, (c) antecedent patient characteristics only, and (d) ICU LOS. For each analysis, discrimination was measured by area under the receiver operating characteristics curve (AUC), calculated using the bootstrap method. Statistical significance between the models was assessed using the DeLong method (p value < 0.05). RESULTS: In 400,248 ICU patients observed, hospital mortality was 11.8% and 1-year mortality 21.8%. At ICU admission, the combined model predicted 1-year mortality with an AUC of 0.84 (95% CI 0.84-0.84). When analyzed separately, the acute component progressively lost predictive power. From an ICU admission of at least 3 days, antecedent characteristics significantly exceeded the predictive value of the acute component for 1-year mortality, AUC 0.68 (95% CI 0.68-0.69) versus 0.67 (95% CI 0.67-0.68) (p value < 0.001). For hospital mortality, antecedent characteristics outperformed the acute component from a LOS of at least 7 days, comprising 7.8% of patients and accounting for 52.4% of all bed days. ICU LOS predicted 1-year mortality with an AUC of 0.52 (95% CI 0.51-0.53) and hospital mortality with an AUC of 0.54 (95% CI 0.53-0.55) for patients with a LOS of at least 7 days. CONCLUSIONS: Comparing the predictive value of factors influencing 1-year mortality for patients with increasing ICU LOS, antecedent patient characteristics are more predictive than the acute component for patients with an ICU LOS of at least 3 days. For hospital mortality, antecedent patient characteristics outperform the acute component for patients with an ICU LOS of at least 7 days. After the first week of ICU admission, LOS itself is not predictive of hospital nor 1-year mortality.

Elicitation and prioritization of requirements for electronic health records for oncology in low resource settings: A concept mapping study.

BACKGROUND: Understanding functional and non-functional requirements is essential to successfully implement electronic medical record (EMR) systems. Actual requirements will be different for different contexts. OBJECTIVE: To elicit and prioritize requirements for implementing EMRs in oncology in low and middle income countries (LMICs), and to relate these to requirements from high-income countries. PARTICIPANTS AND SETTING: Cancer care stakeholders including oncologists, general doctors, nurses, biostatisticians, information technologists, from different LMICs, were involved. METHODS: Concept mapping was used. Statements of requirements were obtained during focus group discussions (FGDs) and interviews. Using surveys, the requirements were clustered and ranked on importance and feasibility. Data were analyzed in SPSS using agglomerative hierarchical clustering and multidimensional scaling, to create cluster maps and go-zone maps reflecting the relationships between the requirements and their prioritization. RESULTS: Four FGD sessions, with twenty participants, were conducted. In addition, six participants were interviewed. Twenty-two participants clustered the requirements and sixty-three participants ranked them on importance and feasibility. One hundred and sixty requirement statements were generated which were reduced to sixty-four after de-duplication and merging. Nine clusters were obtained encompassing the following domains, in order of importance: Security, Conducive organization, Management/Governance, General EMR functionalities, Computer infrastructure, Data management, Usability, Oncology decision support, and Ancillary requirements. On ranking, the requirements scored between 3.74 and 4.80 on importance, and between 3.55 and 4.46 on feasibility, on a 5-point Likert scale. We generated concept maps for use when communicating with stakeholders. CONCLUSION: For oncology EMRs in LMICs, requirements overlap those from high-income countries, but generic EMR functionalities, Infrastructural and organizational requirements are still considered priority in LMICs compared to oncology-specific requirements or advanced EMR features e.g. computerized decision support or interoperability. Concept mapping is a fast and cost-effective method for eliciting and prioritizing EMR requirements in a user-centered manner.

An Ontology for Describing Health IT Interventions: Methodological Considerations.

Health informatics as a young, interdisciplinary discipline lacks a unified terminology in some areas. This is especially true when trying to properly describe health informatics interventions developed and deployed to improve quality and efficiency of patient care. We aim at developing a health IT ontology which allows systematically describing health IT interventions. To achieve this, we combine a deductive and an inductive approach. First results are promising and may later be extended by a folksonomy.

Adjusting for Disease Severity Across ICUs in Multicenter Studies.

OBJECTIVES: To compare methods to adjust for confounding by disease severity during multicenter intervention studies in ICU, when different disease severity measures are collected across centers. DESIGN: In silico simulation study using national registry data. SETTING: Twenty mixed ICUs in The Netherlands. SUBJECTS: Fifty-five-thousand six-hundred fifty-five ICU admissions between January 1, 2011, and January 1, 2016. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: To mimic an intervention study with confounding, a fictitious treatment variable was simulated whose effect on the outcome was confounded by Acute Physiology and Chronic Health Evaluation IV predicted mortality (a common measure for disease severity). Diverse, realistic scenarios were investigated where the availability of disease severity measures (i.e., Acute Physiology and Chronic Health Evaluation IV, Acute Physiology and Chronic Health Evaluation II, and Simplified Acute Physiology Score II scores) varied across centers. For each scenario, eight different methods to adjust for confounding were used to obtain an estimate of the (fictitious) treatment effect. These were compared in terms of relative (%) and absolute (odds ratio) bias to a reference scenario where the treatment effect was estimated following correction for the Acute Physiology and Chronic Health Evaluation IV scores from all centers. Complete neglect of differences in disease severity measures across centers resulted in bias ranging from 10.2% to 173.6% across scenarios, and no commonly used methodology-such as two-stage modeling or score standardization-was able to effectively eliminate bias. In scenarios where some of the included centers had (only) Acute Physiology and Chronic Health Evaluation II or Simplified Acute Physiology Score II available (and not Acute Physiology and Chronic Health Evaluation IV), either restriction of the analysis to Acute Physiology and Chronic Health Evaluation IV centers alone or multiple imputation of Acute Physiology and Chronic Health Evaluation IV scores resulted in the least amount of relative bias (0.0% and 5.1% for Acute Physiology and Chronic Health Evaluation II, respectively, and 0.0% and 4.6% for Simplified Acute Physiology Score II, respectively). In scenarios where some centers used Acute Physiology and Chronic Health Evaluation II, regression calibration yielded low relative bias too (relative bias, 12.4%); this was not true if these same centers only had Simplified Acute Physiology Score II available (relative bias, 54.8%). CONCLUSIONS: When different disease severity measures are available across centers, the performance of various methods to control for confounding by disease severity may show important differences. When planning multicenter studies, researchers should make contingency plans to limit the use of or properly incorporate different disease measures across centers in the statistical analysis.

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications.

OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

How to Teach Health IT Evaluation: Recommendations for Health IT Evaluation Courses.

Systematic health IT evaluation studies are needed to ensure system quality and safety and to provide the basis for evidence-based health informatics. Well-trained health informatics specialists are required to guarantee that health IT evaluation studies are conducted in accordance with robust standards. Also, policy makers and managers need to appreciate how good evidence is obtained by scientific process and used as an essential justification for policy decisions. In a consensus-based approach with over 80 experts in health IT evaluation, recommendations for the structure, scope and content of health IT evaluation courses on the master or postgraduate level have been developed, supported by a structured analysis of available courses and of available literature. The recommendations comprise 15 mandatory topics and 15 optional topics for a health IT evaluation course.

Publishing Health IT Evaluation Studies.

Progress in science is based on evidence from well-designed studies. However, publication quality of health IT evaluation studies is often low, making exploitation of published evidence within systematic reviews and meta-analysis a challenging task. Consequently, reporting guidelines have been published and recommended to be used. After a short overview of publication guidelines relevant for health IT evaluation studies (such as CONSORT and PRISMA), the STARE-HI guidelines for publishing health IT evaluation studies are presented. Health IT evaluation publications should take into account published guidelines, to improve the quality of published evidence. Publication guidelines, in line with addressing publication bias and low study quality, help strengthening the evidence available in the public domain to enable effective evidence-based health informatics.

Evaluation of Health IT in Low-Income Countries.

Low and middle income countries (LMICs) bear a disproportionate burden of major global health challenges. Health IT could be a promising solution in these settings but LMICs have the weakest evidence of application of health IT to enhance quality of care. Various systematic reviews show significant challenges in the implementation and evaluation of health IT. Key barriers to implementation include lack of adequate infrastructure, inadequate and poorly trained health workers, lack of appropriate legislation and policies and inadequate financial 333indicating the early state of generation of evidence to demonstrate the effectiveness of health IT in improving health outcomes and processes. The implementation challenges need to be addressed. The introduction of new guidelines such as GEP-HI and STARE-HI, as well as models for evaluation such as SEIPS, and the prioritization of evaluations in eHealth strategies of LMICs provide an opportunity to focus on strategic concepts that transform the demands of a modern integrated health care system into solutions that are secure, efficient and sustainable.

Influences of definition ambiguity on hospital performance indicator scores: examples from The Netherlands.

RESEARCH OBJECTIVE: Reliable and unambiguously defined performance indicators are fundamental to objective and comparable measurements of hospitals' quality of care. In two separate case studies (intensive care and breast cancer care), we investigated if differences in definition interpretation of performance indicators affected the indicator scores. DESIGN: Information about possible definition interpretations was obtained by a short telephone survey and a Web survey. We quantified the interpretation differences using a patient-level dataset from a national clinical registry (Case I) and a hospital's local database (Case II). In Case II, there was additional textual information available about the patients' status, which was reviewed to get more insight into the origin of the differences. PARTICIPANTS: For Case I, we investigated 15 596 admissions of 33 intensive care units in 2009. Case II consisted of 144 admitted patients with a breast tumour surgically treated in one hospital in 2009. RESULTS: In both cases, hospitals reported different interpretations of the indicators, which lead to significant differences in the indicator values. Case II revealed that these differences could be explained by patient-related factors such as severe comorbidity and patients' individual preference in surgery date. CONCLUSIONS: With this article, we hope to increase the awareness on pitfalls regarding the indicator definitions and the quality of the underlying data. To enable objective and comparable measurements of hospitals' quality of care, organizations that request performance information should formalize the indicators they use, including standardization of all data elements of which the indicator is composed (procedures, diagnoses).

Exploring a methodology for eHealth indicator development.

Indicators provide a practical method to monitor and benchmark eHealth progress towards objectives set in local, national and international policies, and to offer evidence for eHealth management. There is no agreed methodology to develop and define these indicators. The purpose of this paper is to present a proposal for an indicator development methodology and indicator classification. This proposal combines expert-led top-down and community-based bottom-up approaches. It offers a holistic approach for developing indicators for measuring progress and impacts of eHealth development consisting of four phases: (1) defining the context for measurement, (2) defining the goal of measurement, (3) defining the methods for indicator selection and indicator categorization and (4) defining the data to be collected and analyzed to calculate the indicator. Our preliminary results will be used as a starting point for developing a more detailed description of methods for indicator development and for identifying and classifying eHealth indicators and on testing them in practice.

A pilot study on tertiary teledermatology: feasibility and acceptance of telecommunication among dermatologists.

Tertiary teledermatology (TTD), where a general dermatologist consults a specialized dermatologist on difficult cases, is a relatively new telemedicine service. We evaluated TTD in a Dutch university hospital, where 13 general dermatologists used TTD to consult 11 specialized dermatologists and two residents at the university medical centre. We measured the avoided referrals to the university centre, the usability of the system and the user acceptance of it. During a three-month study, general dermatologists consulted via TTD 28 times. In 17 of the consultations (61%), the general dermatologists would have referred their patients to the university centre if teledermatology had not been available. Referral was not necessary after teledermatology for 12 of these 17 consultations (71%). The mean usability score (0-100) of all the users was 80. All dermatologists were satisfied with TTD (mean satisfaction of 7.6 on a 10-point scale) and acceptance was high. The baseline measurements showed that half of tertiary referrals were suitable for TTD. These results suggest that TTD reduces unnecessary physical referrals and that users are satisfied with it. A large-scale evaluation is now required.

Tertiary teledermatology: a systematic review.

Telemedicine is becoming widely used in healthcare. Dermatology, because of its visual character, is especially suitable for telemedicine applications. Most common is teledermatology between general practitioners and dermatologists (secondary teledermatology). Another form of the teledermatology process is communication among dermatologists (tertiary teledermatology). The objective of this systematic review is to give an overview of studies on tertiary teledermatology with emphasis on the categories of use. A systematic literature search on tertiary teledermatology studies used all databases of the Cochrane Library, MEDLINE (1966-November 2007) and EMBASE (1980-November 2007). Categories of use were identified for all included articles and the modalities of tertiary teledermatology were extracted, together with technology, the setting the outcome measures, and their results. The search resulted in 1,377 publications, of which 11 were included. Four categories of use were found: getting an expert opinion from a specialized, often academic dermatologist (6/11); resident training (2/11); continuing medical education (4/11); and second opinion from a nonspecialized dermatologist (2/11). Three modalities were found: a teledermatology consultation application (7/11), a Web site (2/11), and an e-mail list (1/11). The majority (7/11) used store-and-forward, and 3/11 used store-and-forward and real-time. Outcome measures mentioned were learning effect (6), costs (5), diagnostic accuracy (1), validity (2) and reliability (2), patient and physician satisfaction (1), and efficiency improvement (3). Tertiary teledermatology's main category of use is getting an expert opinion from a specialized, often academic dermatologist. Tertiary teledermatology research is still in early development. Future research should focus on identifying the scale of tertiary teledermatology and on what modality of teledermatology is most suited for what purpose in communication among dermatologists.