Open Source Software in Healthcare: International Case Series from the IMIA Open Source Working Group.

In this case series, we demonstrate how open-source software has been widely adopted as the primary health information system in many low- and middle-income countries, and for government-developed applications in high-income settings. We discuss the concept of Digital Global Goods and how the general approach of releasing software developed through public funding under open-source licences could improve the delivery of healthcare in all settings through increased transparency and collaboration as well as financial efficiency.

Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study.

BACKGROUND: Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients. OBJECTIVE: The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews. METHODS: We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated "unsafe" or "too cautious." RESULTS: Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%). CONCLUSIONS: ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation.

Underlying factors determining grain morphologies in high-strength titanium alloys processed by additive manufacturing.

In recent research, additions of solute to Ti and some Ti-based alloys have been employed to produce equiaxed microstructures when processing these materials using additive manufacturing. The present study develops a computational scheme for guiding the selection of such alloying additions, and the minimum amounts required, to effect the columnar to equiaxed microstructural transition. We put forward two physical mechanisms that may produce this transition; the first and more commonly discussed is based on growth restriction factors, and the second on the increased freezing range effected by the alloying addition coupled with the imposed rapid cooling rates associated with AM techniques. We show in the research described here, involving a number of model binary as well as complex multi-component Ti alloys, and the use of two different AM approaches, that the latter mechanism is more reliable regarding prediction of the grain morphology resulting from given solute additions.

Barriers, Priorities and Lessons Learned in Achieving Electronic Health Records Interoperability in Low- and Middle-Income Countries: Workshop Findings.

Electronic health records (EHR) interoperability is a complex topic that continues to gain traction in the digital health landscape. We facilitated a qualitative workshop consisting of domain experts in EHR implementation and health IT managers. The workshop aimed to identify critical barriers to achieving interoperability, priorities for new EHR implementations and lessons learned from managing existing implementations. The workshop highlighted that data modelling and interoperability standards are vital priorities for maternal and child health data services in low- and middle-income countries (LMICs).

The Global Health Informatics landscape and JAMIA.

Global Health Informatics (GHI) as a branch of health informatics has been established for 2 decades now. During that time, great strides have been made in the creation and implementation of informatics tools to improve healthcare delivery and outcomes in the most vulnerable and remote communities worldwide. In many of the most successful projects, innovation has been shared between teams in high- and low- or middle-income countries (LMICs). In this perspective, we review the state of the academic field of GHI and the work published in JAMIA in the last 6 1/2 years. We apply criteria for articles about LMICs, those on international health, and on indigenous and refugee population, and subtypes of research. For comparison, we apply those criteria to JAMIA Open and 3 other health informatics journals which publish articles on GHI. We make recommendations for future directions and the role that journals like JAMIA can play in strengthening this work worldwide.

Ultrastrong nanotwinned titanium alloys through additive manufacturing.

Titanium alloys, widely used in the aerospace, automotive and energy sectors, require complex casting and thermomechanical processing to achieve the high strengths required for load-bearing applications. Here we reveal that additive manufacturing can exploit thermal cycling and rapid solidification to create ultrastrong and thermally stable titanium alloys, which may be directly implemented in service. As demonstrated in a commercial titanium alloy, after simple post-heat treatment, adequate elongation and tensile strengths over 1,600 MPa are achieved. The excellent properties are attributed to the unusual formation of dense, stable and internally twinned nanoprecipitates, which are rarely observed in traditionally processed titanium alloys. These nanotwinned precipitates are shown to originate from a high density of dislocations with a dominant screw character and formed from the additive manufacturing process. The work here paves the way to fabricate structural materials with unique microstructures and excellent properties for broad applications.

A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials.

In computed TEM tomography, image segmentation represents one of the most basic tasks with implications not only for 3D volume visualization, but more importantly for quantitative 3D analysis. In case of large and complex 3D data sets, segmentation can be an extremely difficult and laborious task, and thus has been one of the biggest hurdles for comprehensive 3D analysis. Heterogeneous catalysts have complex surface and bulk structures, and often sparse distribution of catalytic particles with relatively poor intrinsic contrast, which possess a unique challenge for image segmentation, including the current state-of-the-art deep learning methods. To tackle this problem, we apply a deep learning-based approach for the multi-class semantic segmentation of a γ-Alumina/Pt catalytic material in a class imbalance situation. Specifically, we used the weighted focal loss as a loss function and attached it to the U-Net's fully convolutional network architecture. We assessed the accuracy of our results using Dice similarity coefficient (DSC), recall, precision, and Hausdorff distance (HD) metrics on the overlap between the ground-truth and predicted segmentations. Our adopted U-Net model with the weighted focal loss function achieved an average DSC score of 0.96 ± 0.003 in the γ-Alumina support material and 0.84 ± 0.03 in the Pt NPs segmentation tasks. We report an average boundary-overlap error of less than 2 nm at the 90th percentile of HD for γ-Alumina and Pt NPs segmentations. The complex surface morphology of γ-Alumina and its relation to the Pt NPs were visualized in 3D by the deep learning-assisted automatic segmentation of a large data set of high-angle annular dark-field (HAADF) scanning transmission electron microscopy (STEM) tomography reconstructions.

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study.

BACKGROUND: Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions. OBJECTIVE: The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED). METHODS: We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker's diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model. RESULTS: A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although "satisfaction" and "enjoyment" were rated low. CONCLUSIONS: This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

Implementing Electronic Health Records in Primary Care Using the Theory of Change: Nigerian Case Study.

BACKGROUND: Digital health has been a tool of transformation for the delivery of health care services globally. An electronic health record (EHR) system can solve the bottleneck of paper documentation in health service delivery if it is successfully implemented, but poor implementation can lead to a waste of resources. The study of EHR system implementation in low- and middle-income countries (LMICs) is of particular interest to health stakeholders such as policy makers, funders, and care providers because of the efficiencies and evidence base that could result from the appropriate evaluation of such systems. OBJECTIVE: We aimed to develop a theory of change (ToC) for the implementation of EHRs for maternal and child health care delivery in LMICs. The ToC is an outcomes-based approach that starts with the long-term goals and works backward to the inputs and mediating components required to achieve these goals for complex programs. METHODS: We used the ToC approach for the whole implementation's life cycle to guide the pilot study and identify the preconditions needed to realize the study's long-term goal at Festac Primary Health Centre in Lagos, Nigeria. To evaluate the maturity of the implementation, we adapted previously defined success factors to supplement the ToC approach. RESULTS: The initial ToC map showed that the long-term goal was an improved service delivery in primary care with the introduction of EHRs. The revised ToC revealed that the long-term change was the improved maternal and child health care delivery at Festac Primary Health Center using EHRs. We proposed a generic ToC map that implementers in LMICs can use to introduce an optimized EHR system, with assumptions about sustainability and other relevant factors. The outcomes from the critical success factors were sustainability: the sustained improvements included trained health care professionals, a change in mindset from using paper systems toward digital health transformation, and using the project's laptops to collect aggregate data for the District Health Information System 2-based national health information management system; financial: we secured funding to procure IT equipment, including servers, laptops, and networking, but the initial cost of implementation was high, and funds mainly came from the funding partner; and organizational: the health professionals, especially the head of nursing and health information officers, showed significant commitment to adopting the EHR system, but certain physicians and midwives were unwilling to use the EHR system initially until they were persuaded or incentivized by the management. CONCLUSIONS: This study shows that the ToC is a rewarding approach to framing dialogue with stakeholders and serves as a framework for planning, evaluation, learning, and reflection. We hypothesized that any future health IT implementation in primary care could adapt our ToC approach to their contexts with necessary modifications based on inherent characteristics.

Open Source Digital Health Software for Resilient, Accessible and Equitable Healthcare Systems.

OBJECTIVE: To assess the impact of open-source projects on making healthcare systems more resilient, accessible and equitable. METHODS: In response to the International Medical Informatics Association (IMIA) call for working group contributions for the IMIA Yearbook, the Open Source Working Group (OSWG) conducted a rapid review of current open source digital health projects to illustrate how they can contribute to making healthcare systems more resilient, accessible and equitable. We sought case studies from the OSWG membership to illustrate these three concepts and how open source software (OSS) addresses these concepts in the real world. These case studies are discussed against the background of literature identified through the rapid review. RESULTS: To illustrate the concept of resilience, we present case studies from the adoption of District Health Information Software version 2 (DHIS2) for managing the Covid pandemic in Rwanda, and the adoption of the OpenEHR open Health IT standard. To illustrate accessibility, we show how open source design systems for user interface design have been used by governments to ensure accessibility of digital health services for patients and healthy individuals, and by the OpenMRS community to standardise their user interface design. Finally, to illustrate the concept of equity, we describe the OpenWHO framework and two open source digital health projects, GNU Health and openIMIS, that both aim to reduce health inequities through the use of open source digital health software. CONCLUSION: This review has demonstrated that open source software addresses many of the challenges involved in making healthcare more accessible, equitable and resilient in high and low income settings.

Crystallographic and Compositional Evolution of Ordered B2 and Disordered BCC Phases During Isothermal Annealing of Refractory High-Entropy Alloys.

Refractory high-entropy alloys (RHEAs) are promising candidates for next-generation high-temperature materials. RHEAs containing Al, often exhibit a checkered pattern microstructure comprising a combination of disordered BCC and ordered B2 phases. Since the ordered B2 phase is based on the BCC parent matrix, distinguishing these two phases can be rather challenging. Advanced characterization techniques are necessary for a reliable qualitative and quantitative analysis of BCC and B2 phases in RHEAs. Additionally, there is a tendency for transformation of the ordered B2 phase into more complex ordered-omega type phases that are usually deleterious to mechanical properties. The current study focuses on the phase stability of a candidate RHEA, Al0.5Mo0.5NbTa0.5TiZr. Correlative transmission electron microscopy (TEM) and atom probe tomography (APT) have been employed to investigate the phase stability and transformation pathway of this RHEA when isothermally annealed at 800°C. The results show that a metastable two-phase BCC + B2 microstructure formed at the early stages of decomposition, eventually transforming into a three-phase BCC + B2 + hP18 microstructure. The hP18 phase is an ordered omega derivative of the ordered B2 phase. The correlative microscopy techniques (TEM and APT) reveal a very interesting interplay of compositional partitioning between the different phases and their respective stability.

User Perceptions and Use of an Enhanced Electronic Health Record in Rwanda With and Without Clinical Alerts: Cross-sectional Survey.

BACKGROUND: Electronic health records (EHRs) have been implemented in many low-resource settings but lack strong evidence for usability, use, user confidence, scalability, and sustainability. OBJECTIVE: This study aimed to evaluate staff use and perceptions of an EHR widely used for HIV care in >300 health facilities in Rwanda, providing evidence on factors influencing current performance, scalability, and sustainability. METHODS: A randomized, cross-sectional, structured interview survey of health center staff was designed to assess functionality, use, and attitudes toward the EHR and clinical alerts. This study used the associated randomized clinical trial study sample (56/112, 50% sites received an enhanced EHR), pulling 27 (50%) sites from each group. Free-text comments were analyzed thematically using inductive coding. RESULTS: Of the 100 participants, 90 (90% response rate) were interviewed at 54 health centers: 44 (49%) participants were clinical and 46 (51%) were technical. The EHR top uses were to access client data easily or quickly (62/90, 69%), update patient records (56/89, 63%), create new patient records (49/88, 56%), generate various reports (38/85, 45%), and review previous records (43/89, 48%). In addition, >90% (81/90) of respondents agreed that the EHR made it easier to make informed decisions, was worth using, and has improved patient information quality. Regarding availability, (66/88) 75% said they could always or almost always count on the EHR being available, whereas (6/88) 7% said never/almost never. In intervention sites, staff were significantly more likely to update existing records (P=.04), generate summaries before (P<.001) or during visits (P=.01), and agree that "the EHR provides useful alerts, and reminders" (P<.01). CONCLUSIONS: Most users perceived the EHR as well accepted, appropriate, and effective for use in low-resource settings despite infrastructure limitation in 25% (22/88) of the sites. The implementation of EHR enhancements can improve the perceived usefulness and use of key functions. Successful scale-up and use of EHRs in small health facilities could improve clinical documentation, care, reporting, and disease surveillance in low- and middle-income countries.

A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care.

BACKGROUND: Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining. OBJECTIVES: This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users. METHODS: Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies. RESULTS: Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of "good." We reviewed persona-based case studies that illustrate functionalities of this novel prototype. CONCLUSION: The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.

Modelling Prenatal Care Pathways at a Central Hospital in Zimbabwe.

BACKGROUND: Maternal mortality remains a problem in low-income countries (LICs). In Zimbabwe, there has been an unprecedented increase in maternal mortality in the last 2.5 decades. Effective prenatal care delivery, particularly early visits, appropriate number of visits, and receiving recommended care is viewed as key to reducing fatal care outcomes. AIMS: This study sought to model and identify gaps requiring service and care delivery improvement in prenatal care pathways for pregnant women visiting Mpilo Central Hospital in Bulawayo, Zimbabwe. METHODS: This was a case study of the services offered by an antenatal care department at Mpilo Central Hospital in Bulawayo, Zimbabwe. Evidence from literature in low-income countries was used to develop prenatal care pathway guidelines as a tool to guide care delivery and identify gaps in care and service delivery. One hundred cases of prenatal care records were reviewed to determine the prenatal care pathway and care delivered to pregnant women. This data was complemented by interviews with 20 maternity care clinicians. RESULTS: In 100 maternity case records studied, 53% booked for prenatal care. Of the 53% (n = 53) pregnant women who booked, their first visit on their pregnancy was late at an average gestational age of 27.1 weeks with extremes of 30 to 40 weeks in 38% (n = 20) cases. Missing scheduled prenatal care appointments was prevalent, with only 11% (n = 6) having attended all the expected 5 visits, whilst 60% (n = 32) missed 3 or more. There were inadequacies in the care delivered to women in each visit compared to that expected in such areas as obstetrics, physical examinations and haematological tests. Maternity care clinicians attributed the cost of prenatal booking fees in the background of poverty and poor family support systems as key factors hindering women's access to prenatal services. CONCLUSIONS: The current prenatal care pathway at MCH requires improvement in the areas of referral, adherence to appointment by pregnant women and visiting prenatal care early. Clinicians also need to adhere to standard clinical tests recommended for each specific pregnant woman's visit. In the Zimbabwean setting with limited resources, where the number of visits is already low, pathways with reduced visits may not be appropriate. An investment into prenatal care by the government is recommended to enable the utilisation of interventions such as e-health technologies that may improve care delivery as well as adherence to best practices. E-health and mobile health technologies involving e-referrals, e-booking, decision support, and reminder systems are recommended for clinicians to manage and deliver appropriate care to patients as well as pregnant women to adhere to scheduled visits.

Optimizing image contrast of second phases in metal alloys.

Novel imaging strategies in the scanning electron microscope aimed at significantly improved image contrast of second phases in metal alloys are described. These include the use of low accelerating voltages, small working distances, and a novel detection system. Contrast is assessed as a function of voltage and optimized imaging conditions which result in much improved image quality are presented. These strategies are applied to two precipitation hardened Ni-base alloys, a cast single crystal and a hot isostatically pressed sample.

Using Open Source, Open Data, and Civic Technology to Address the COVID-19 Pandemic and Infodemic.

OBJECTIVES: The emerging COVID-19 pandemic has caused one of the world's worst health disasters compounded by social confusion with misinformation, the so-called "Infodemic". In this paper, we discuss how open technology approaches - including data sharing, visualization, and tooling - can address the COVID-19 pandemic and infodemic. METHODS: In response to the call for participation in the 2020 International Medical Informatics Association (IMIA) Yearbook theme issue on Medical Informatics and the Pandemic, the IMIA Open Source Working Group surveyed recent works related to the use of Free/Libre/Open Source Software (FLOSS) for this pandemic. RESULTS: FLOSS health care projects including GNU Health, OpenMRS, DHIS2, and others, have responded from the early phase of this pandemic. Data related to COVID-19 have been published from health organizations all over the world. Civic Technology, and the collaborative work of FLOSS and open data groups were considered to support collective intelligence on approaches to managing the pandemic. CONCLUSION: FLOSS and open data have been effectively used to contribute to managing the COVID-19 pandemic, and open approaches to collaboration can improve trust in data.

How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs.

OBJECTIVES: To compare breadth of condition coverage, accuracy of suggested conditions and appropriateness of urgency advice of eight popular symptom assessment apps. DESIGN: Vignettes study. SETTING: 200 primary care vignettes. INTERVENTION/COMPARATOR: For eight apps and seven general practitioners (GPs): breadth of coverage and condition-suggestion and urgency advice accuracy measured against the vignettes' gold-standard. PRIMARY OUTCOME MEASURES: (1) Proportion of conditions 'covered' by an app, that is, not excluded because the user was too young/old or pregnant, or not modelled; (2) proportion of vignettes with the correct primary diagnosis among the top 3 conditions suggested; (3) proportion of 'safe' urgency advice (ie, at gold standard level, more conservative, or no more than one level less conservative). RESULTS: Condition-suggestion coverage was highly variable, with some apps not offering a suggestion for many users: in alphabetical order, Ada: 99.0%; Babylon: 51.5%; Buoy: 88.5%; K Health: 74.5%; Mediktor: 80.5%; Symptomate: 61.5%; Your.MD: 64.5%; WebMD: 93.0%. Top-3 suggestion accuracy was GPs (average): 82.1%±5.2%; Ada: 70.5%; Babylon: 32.0%; Buoy: 43.0%; K Health: 36.0%; Mediktor: 36.0%; Symptomate: 27.5%; WebMD: 35.5%; Your.MD: 23.5%. Some apps excluded certain user demographics or conditions and their performance was generally greater with the exclusion of corresponding vignettes. For safe urgency advice, tested GPs had an average of 97.0%±2.5%. For the vignettes with advice provided, only three apps had safety performance within 1 SD of the GPs-Ada: 97.0%; Babylon: 95.1%; Symptomate: 97.8%. One app had a safety performance within 2 SDs of GPs-Your.MD: 92.6%. Three apps had a safety performance outside 2 SDs of GPs-Buoy: 80.0% (p<0.001); K Health: 81.3% (p<0.001); Mediktor: 87.3% (p=1.3×10-3). CONCLUSIONS: The utility of digital symptom assessment apps relies on coverage, accuracy and safety. While no digital tool outperformed GPs, some came close, and the nature of iterative improvements to software offers scalable improvements to care.

Clinicians perceptions of a telemedicine system: a mixed method study of Makassar City, Indonesia.

BACKGROUND: This case study in Makassar City, Indonesia aims to investigate the clinicians' perceptions, including both satisfaction and barriers in using telemedicine in a large, established program which supported 3974 consultations in 2017. METHODS: A mixed methodology was used in this research utilizing a questionnaire with 12 questions, and semi-structured interviews. A purposeful sample of clinicians using the telemedicine system at the 39 primary care clinics in Makassar City were surveyed. A total of 100 clinicians participated in this study. All of them completed the questionnaires (76.9% response rate) and 15 of them were interviewed. RESULTS: The result showed that 78% of the clinicians were satisfied with the telemedicine system. In free text responses 69% said that telemedicine allowed quicker diagnosis and treatment, 47% said poor internet connectivity was a significant obstacle in using the system, and 40% suggested improvement to the infrastructure including internet connection and electricity. CONCLUSION: Overall, the clinicians were satisfied with the system, with the main benefit of rendering the diagnosis faster and easier for patients. However, poor internet connectivity was indicated as the main barrier. Most of the clinicians suggested improving the infrastructure especially the internet network.