Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study.

BACKGROUND: Electronic health records (EHRs) play an increasingly important role in delivering HIV care in low- and middle-income countries. The data collected are used for direct clinical care, quality improvement, program monitoring, public health interventions, and research. Despite widespread EHR use for HIV care in African countries, challenges remain, especially in collecting high-quality data. OBJECTIVE: We aimed to assess data completeness, accuracy, and timeliness compared to paper-based records, and factors influencing data quality in a large-scale EHR deployment in Rwanda. METHODS: We randomly selected 50 health facilities (HFs) using OpenMRS, an EHR system that supports HIV care in Rwanda, and performed a data quality evaluation. All HFs were part of a larger randomized controlled trial, with 25 HFs receiving an enhanced EHR with clinical decision support systems. Trained data collectors visited the 50 HFs to collect 28 variables from the paper charts and the EHR system using the Open Data Kit app. We measured data completeness, timeliness, and the degree of matching of the data in paper and EHR records, and calculated concordance scores. Factors potentially affecting data quality were drawn from a previous survey of users in the 50 HFs. RESULTS: We randomly selected 3467 patient records, reviewing both paper and EHR copies (194,152 total data items). Data completeness was >85% threshold for all data elements except viral load (VL) results, second-line, and third-line drug regimens. Matching scores for data values were close to or >85% threshold, except for dates, particularly for drug pickups and VL. The mean data concordance was 10.2 (SD 1.28) for 15 (68%) variables. HF and user factors (eg, years of EHR use, technology experience, EHR availability and uptime, and intervention status) were tested for correlation with data quality measures. EHR system availability and uptime was positively correlated with concordance, whereas users' experience with technology was negatively correlated with concordance. The alerts for missing VL results implemented at 11 intervention HFs showed clear evidence of improving timeliness and completeness of initially low matching of VL results in the EHRs and paper records (11.9%-26.7%; P<.001). Similar effects were seen on the completeness of the recording of medication pickups (18.7%-32.6%; P<.001). CONCLUSIONS: The EHR records in the 50 HFs generally had high levels of completeness except for VL results. Matching results were close to or >85% threshold for nondate variables. Higher EHR stability and uptime, and alerts for entering VL both strongly improved data quality. Most data were considered fit for purpose, but more regular data quality assessments, training, and technical improvements in EHR forms, data reports, and alerts are recommended. The application of quality improvement techniques described in this study should benefit a wide range of HFs and data uses for clinical care, public health, and disease surveillance.

The Global Health Informatics landscape and JAMIA.

Global Health Informatics (GHI) as a branch of health informatics has been established for 2 decades now. During that time, great strides have been made in the creation and implementation of informatics tools to improve healthcare delivery and outcomes in the most vulnerable and remote communities worldwide. In many of the most successful projects, innovation has been shared between teams in high- and low- or middle-income countries (LMICs). In this perspective, we review the state of the academic field of GHI and the work published in JAMIA in the last 6 1/2 years. We apply criteria for articles about LMICs, those on international health, and on indigenous and refugee population, and subtypes of research. For comparison, we apply those criteria to JAMIA Open and 3 other health informatics journals which publish articles on GHI. We make recommendations for future directions and the role that journals like JAMIA can play in strengthening this work worldwide.

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study.

BACKGROUND: Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions. OBJECTIVE: The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED). METHODS: We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker's diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model. RESULTS: A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although "satisfaction" and "enjoyment" were rated low. CONCLUSIONS: This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

User Perceptions and Use of an Enhanced Electronic Health Record in Rwanda With and Without Clinical Alerts: Cross-sectional Survey.

BACKGROUND: Electronic health records (EHRs) have been implemented in many low-resource settings but lack strong evidence for usability, use, user confidence, scalability, and sustainability. OBJECTIVE: This study aimed to evaluate staff use and perceptions of an EHR widely used for HIV care in >300 health facilities in Rwanda, providing evidence on factors influencing current performance, scalability, and sustainability. METHODS: A randomized, cross-sectional, structured interview survey of health center staff was designed to assess functionality, use, and attitudes toward the EHR and clinical alerts. This study used the associated randomized clinical trial study sample (56/112, 50% sites received an enhanced EHR), pulling 27 (50%) sites from each group. Free-text comments were analyzed thematically using inductive coding. RESULTS: Of the 100 participants, 90 (90% response rate) were interviewed at 54 health centers: 44 (49%) participants were clinical and 46 (51%) were technical. The EHR top uses were to access client data easily or quickly (62/90, 69%), update patient records (56/89, 63%), create new patient records (49/88, 56%), generate various reports (38/85, 45%), and review previous records (43/89, 48%). In addition, >90% (81/90) of respondents agreed that the EHR made it easier to make informed decisions, was worth using, and has improved patient information quality. Regarding availability, (66/88) 75% said they could always or almost always count on the EHR being available, whereas (6/88) 7% said never/almost never. In intervention sites, staff were significantly more likely to update existing records (P=.04), generate summaries before (P<.001) or during visits (P=.01), and agree that "the EHR provides useful alerts, and reminders" (P<.01). CONCLUSIONS: Most users perceived the EHR as well accepted, appropriate, and effective for use in low-resource settings despite infrastructure limitation in 25% (22/88) of the sites. The implementation of EHR enhancements can improve the perceived usefulness and use of key functions. Successful scale-up and use of EHRs in small health facilities could improve clinical documentation, care, reporting, and disease surveillance in low- and middle-income countries.

A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care.

BACKGROUND: Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining. OBJECTIVES: This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users. METHODS: Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies. RESULTS: Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of "good." We reviewed persona-based case studies that illustrate functionalities of this novel prototype. CONCLUSION: The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.

Clinicians perceptions of a telemedicine system: a mixed method study of Makassar City, Indonesia.

BACKGROUND: This case study in Makassar City, Indonesia aims to investigate the clinicians' perceptions, including both satisfaction and barriers in using telemedicine in a large, established program which supported 3974 consultations in 2017. METHODS: A mixed methodology was used in this research utilizing a questionnaire with 12 questions, and semi-structured interviews. A purposeful sample of clinicians using the telemedicine system at the 39 primary care clinics in Makassar City were surveyed. A total of 100 clinicians participated in this study. All of them completed the questionnaires (76.9% response rate) and 15 of them were interviewed. RESULTS: The result showed that 78% of the clinicians were satisfied with the telemedicine system. In free text responses 69% said that telemedicine allowed quicker diagnosis and treatment, 47% said poor internet connectivity was a significant obstacle in using the system, and 40% suggested improvement to the infrastructure including internet connection and electricity. CONCLUSION: Overall, the clinicians were satisfied with the system, with the main benefit of rendering the diagnosis faster and easier for patients. However, poor internet connectivity was indicated as the main barrier. Most of the clinicians suggested improving the infrastructure especially the internet network.

Barriers to the success of an electronic pharmacovigilance reporting system in Kenya: an evaluation three years post implementation.

Objective: Electronic pharmacovigilance reporting systems are being implemented in many developing countries in an effort to improve reporting rates. This study sought to establish the factors that acted as barriers to the success of an electronic pharmacovigilance reporting system in Kenya 3 years after its implementation. Materials and Methods: Factors that could act as barriers to using electronic reporting systems were identified in a review of literature and then used to develop a survey questionnaire that was administered to pharmacists working in government hospitals in 6 counties in Kenya. Results: The survey was completed by 103 out of the 115 targeted pharmacists (89.5%) and included free-text comments. The key factors identified as barriers were: unavailable, unreliable, or expensive Internet access; challenges associated with a hybrid system of paper and electronic reporting tools; and system usability issues. Coordination challenges at the national pharmacovigilance center and changes in the structure of health management in the country also had an impact on the success of the electronic reporting system. Discussion: Different personal, organizational, infrastructural, and reporting system factors affect the success of electronic reporting systems in different ways, depending on the context. Context-specific formative evaluations are useful in establishing the performance of electronic reporting systems to identify problems and ensure that they achieve the desired objectives. Conclusion: While several factors hindered the optimal use of the electronic pharmacovigilance reporting system in Kenya, all were considered modifiable. Effort should be directed toward tackling the identified issues in order to facilitate use and improve pharmacovigilance reporting rates.

Reducing communication delays and improving quality of care with a tuberculosis laboratory information system in resource poor environments: a cluster randomized controlled trial.

BACKGROUND: Lost, delayed or incorrect laboratory results are associated with delays in initiating treatment. Delays in treatment for Multi-Drug Resistant Tuberculosis (MDR-TB) can worsen patient outcomes and increase transmission. The objective of this study was to evaluate the impact of a laboratory information system in reducing delays and the time for MDR-TB patients to culture convert (stop transmitting). SETTING: 78 primary Health Centers (HCs) in Lima, Peru. Participants lived within the catchment area of participating HCs and had at least one MDR-TB risk factor. The study design was a cluster randomized controlled trial with baseline data. The intervention was the e-Chasqui web-based laboratory information system. Main outcome measures were: times to communicate a result; to start or change a patient's treatment; and for that patient to culture convert. RESULTS: 1671 patients were enrolled. Intervention HCs took significantly less time to receive drug susceptibility test (DST) (median 11 vs. 17 days, Hazard Ratio 0.67 [0.62-0.72]) and culture (5 vs. 8 days, 0.68 [0.65-0.72]) results. The time to treatment was not significantly different, but patients in intervention HCs took 16 days (20%) less time to culture convert (p = 0.047). CONCLUSIONS: The eChasqui system reduced the time to communicate results between laboratories and HCs and time to culture conversion. It is now used in over 259 HCs covering 4.1 million people. This is the first randomized controlled trial of a laboratory information system in a developing country for any disease and the only study worldwide to show clinical impact of such a system. TRIAL REGISTRATION: ClinicalTrials.gov NCT01201941.

E-health systems for management of MDR-TB in resource-poor environments: a decade of experience and recommendations for future work.

INTRODUCTION: Multi-drug resistant TB (MDR-TB) is a complex infectious disease that is a growing threat to global health. It requires lengthy treatment with multiple drugs and specialized laboratory testing. To effectively scale up treatment to thousands of patients requires good information systems to support clinical care, reporting, drug forecasting, supply chain management and monitoring. METHODS: Over the last decade we have developed the PIH-EMR electronic medical record system, and subsequently OpenMRS-TB, to support the treatment of MDR-TB in Peru, Haiti, Pakistan, and other resource-poor environments. RESULTS: We describe here the experience with implementing these systems and evaluating many aspects of their performance, and review other systems for MDR-TB management. CONCLUSIONS: We recommend a new approach to information systems to address the barriers to scale up MDR-TB treatment, particularly access to the appropriate drugs and lab data. We propose moving away from fragmented, vertical systems to focus on common platforms, addressing all stages of TB care, support for open data standards and interoperability, care for a wide range of diseases including HIV, integration with mHealth applications, and ability to function in resource-poor environments.

Aggressive regimens for multidrug-resistant tuberculosis decrease all-cause mortality.

RATIONALE: A better understanding of the composition of optimal treatment regimens for multidrug-resistant tuberculosis (MDR-TB) is essential for expanding universal access to effective treatment and for developing new therapies for MDR-TB. Analysis of observational data may inform the definition of an optimized regimen. OBJECTIVES: This study assessed the impact of an aggressive regimen-one containing at least five likely effective drugs, including a fluoroquinolone and injectable-on treatment outcomes in a large MDR-TB patient cohort. METHODS: This was a retrospective cohort study of patients treated in a national outpatient program in Peru between 1999 and 2002. We examined the association between receiving an aggressive regimen and the rate of death. MEASUREMENTS AND MAIN RESULTS: In total, 669 patients were treated with individualized regimens for laboratory-confirmed MDR-TB. Isolates were resistant to a mean of 5.4 (SD 1.7) drugs. Cure or completion was achieved in 66.1% (442) of patients; death occurred in 20.8% (139). Patients who received an aggressive regimen were less likely to die (crude hazard ratio [HR]: 0.62; 95% CI: 0.44,0.89), compared to those who did not receive such a regimen. This association held in analyses adjusted for comorbidities and indicators of severity (adjusted HR: 0.63; 95% CI: 0.43,0.93). CONCLUSIONS: The aggressive regimen is a robust predictor of MDR-TB treatment outcome. TB policy makers and program directors should consider this standard as they design and implement regimens for patients with drug-resistant disease. Furthermore, the aggressive regimen should be considered the standard background regimen when designing randomized trials of treatment for drug-resistant TB.

Adaptation of a web-based, open source electronic medical record system platform to support a large study of tuberculosis epidemiology.

BACKGROUND: In 2006, we were funded by the US National Institutes of Health to implement a study of tuberculosis epidemiology in Peru. The study required a secure information system to manage data from a target goal of 16,000 subjects who needed to be followed for at least one year. With previous experience in the development and deployment of web-based medical record systems for TB treatment in Peru, we chose to use the OpenMRS open source electronic medical record system platform to develop the study information system. Supported by a core technical and management team and a large and growing worldwide community, OpenMRS is now being used in more than 40 developing countries. We adapted the OpenMRS platform to better support foreign languages. We added a new module to support double data entry, linkage to an existing laboratory information system, automatic upload of GPS data from handheld devices, and better security and auditing of data changes. We added new reports for study managers, and developed data extraction tools for research staff and statisticians. Further adaptation to handle direct entry of laboratory data occurred after the study was launched. RESULTS: Data collection in the OpenMRS system began in September 2009. By August 2011 a total of 9,256 participants had been enrolled, 102,274 forms and 13,829 laboratory results had been entered, and there were 208 users. The system is now entirely supported by the Peruvian study staff and programmers. CONCLUSIONS: The information system served the study objectives well despite requiring some significant adaptations mid-stream. OpenMRS has more tools and capabilities than it did in 2008, and requires less adaptations for future projects. OpenMRS can be an effective research data system in resource poor environments, especially for organizations using or considering it for clinical care as well as research.

Impacts of e-health on the outcomes of care in low- and middle-income countries: where do we go from here?

E-health encompasses a diverse set of informatics tools that have been designed to improve public health and health care. Little information is available on the impacts of e-health programmes, particularly in low- and middle-income countries. We therefore conducted a scoping review of the published and non-published literature to identify data on the effects of e-health on health outcomes and costs. The emphasis was on the identification of unanswered questions for future research, particularly on topics relevant to low- and middle-income countries. Although e-health tools supporting clinical practice have growing penetration globally, there is more evidence of benefits for tools that support clinical decisions and laboratory information systems than for those that support picture archiving and communication systems. Community information systems for disease surveillance have been implemented successfully in several low- and middle-income countries. Although information on outcomes is generally lacking, a large project in Brazil has documented notable impacts on health-system efficiency. Meta-analyses and rigorous trials have documented the benefits of text messaging for improving outcomes such as patients' self-care. Automated telephone monitoring and self-care support calls have been shown to improve some outcomes of chronic disease management, such as glycaemia and blood pressure control, in low- and middle-income countries. Although large programmes for e-health implementation and research are being conducted in many low- and middle-income countries, more information on the impacts of e-health on outcomes and costs in these settings is still needed.

Full impact of laboratory information system requires direct use by clinical staff: cluster randomized controlled trial.

OBJECTIVE: To evaluate the time to communicate laboratory results to health centers (HCs) between the e-Chasqui web-based information system and the pre-existing paper-based system. METHODS: Cluster randomized controlled trial in 78 HCs in Peru. In the intervention group, 12 HCs had web access to results via e-Chasqui (point-of-care HCs) and forwarded results to 17 peripheral HCs. In the control group, 22 point-of-care HCs received paper results directly and forwarded them to 27 peripheral HCs. Baseline data were collected for 15 months. Post-randomization data were collected for at least 2 years. Comparisons were made between intervention and control groups, stratified by point-of-care versus peripheral HCs. RESULTS: For point-of-care HCs, the intervention group took less time to receive drug susceptibility tests (DSTs) (median 9 vs 16 days, p<0.001) and culture results (4 vs 8 days, p<0.001) and had a lower proportion of 'late' DSTs taking >60 days to arrive (p<0.001) than the control. For peripheral HCs, the intervention group had similar communication times for DST (median 22 vs 19 days, p=0.30) and culture (10 vs 9 days, p=0.10) results, as well as proportion of 'late' DSTs (p=0.57) compared with the control. CONCLUSIONS: Only point-of-care HCs with direct access to the e-Chasqui information system had reduced communication times and fewer results with delays of >2 months. Peripheral HCs had no benefits from the system. This suggests that health establishments should have point-of-care access to reap the benefits of electronic laboratory reporting.

Experience implementing a point-of-care electronic medical record system for primary care in Malawi.

Due to the fact that health care professionals in Malawi are often overstretched, the use and quality of health data can be compromised. The Malawi Health Management Information System (HMIS) has streamlined data collection and reporting and increased the use of data to improve care. Obstacles remain, including incomplete reporting and low staff morale. With the Baobab Health Trust and the Malawi Ministry of Health, Partners In Health piloted an innovative point-of-care data system for primary care that functions alongside OpenMRS, an open source medical record platform. The system has given access to a patient-level primary care dataset in real time. Initial results highlight some of the benefits of a point-of-care system such as improved data quality, emphasize the importance of sharing data with clinical practitioners, and shed light on how this approach could strengthen HMIS.

Using electronic medical records for HIV care in rural Rwanda.

Partners In Health (PIH) implemented an electronic medical record (EMR) system in Rwanda in 2005 to support and improve HIV and TB patient care. The system holds detailed patient records, accessible to clinicians through printed reports or directly via a computer in the consultation rooms. Ongoing assessment of data quality and clinical data use has led multiple interventions to be put in place. One such evaluation cycle led to the implementation of a system which identified 15 previously undiagnosed pediatric patients with HIV. Another cycle led to an EMR intervention which helped to decrease the proportion of completed critical CD4 lab results that did not reach clinicians by 34.2% (p=.002). Additionally an automated data quality improvement system reduced known errors by 92% by providing local data officers a tool and training to allow them to easily access and correct data errors. Electronic systems can be used to support care in rural resource-poor settings, and frequent assessment of data quality and clinical use of data can be used to support that goal.

Training software developers for electronic medical records in Rwanda.

In many developing countries, electronic medical record (EMR) systems are being implemented in resource-poor settings. Essential to such implementations are software developers with a high technical capacity, a good understanding of medical informatics and an awareness of local clinical needs. This paper describes a training program which has been run in Rwanda to enable local computer science graduates to play a significant role in that country's forthcoming implementation of a national EMR system. Such a training program is unique in that region of Africa and we discuss the challenges inherent in such an undertaking. We describe the development of the curriculum and the evolution of the teaching methodologies over the course of the year and discuss its potential integration with academic institutions in Rwanda. Finally we propose that training programs of this nature which produce local software developers who are familiar with medical informatics are a requirement for successful and sustainable eHealth implementations in the developing world.

E-health technologies show promise in developing countries.

Is there any evidence that e-health-using information technology to manage patient care-can have a positive impact in developing countries? Our systematic review of evaluations of e-health implementations in developing countries found that systems that improve communication between institutions, assist in ordering and managing medications, and help monitor and detect patients who might abandon care show promise. Evaluations of personal digital assistants and mobile devices convincingly demonstrate that such devices can be very effective in improving data collection time and quality. Donors and funders should require and sponsor outside evaluations to ensure that future e-health investments are well-targeted.

A toolkit for e-health partnerships in low-income nations.

Collecting, managing, and communicating information is a critical part of delivering high-quality, efficient health care. Low-income countries often lack the information technology that is taking root in developed countries to manage health data and work toward evidence-based practice and culture. Partnerships between academic and government institutions in high- and low-income countries can help establish health informatics programs. These programs, in turn, can capture and manage data that are useful to all parties. Several partnerships among academic institutions and public and private organizations, in areas such as sub-Saharan Africa, Haiti, and Peru, are leading the way.