Microsoft Bing outperforms five other generative artificial intelligence chatbots in the Antwerp University multiple choice medical license exam.

Recently developed chatbots based on large language models (further called bots) have promising features which could facilitate medical education. Several bots are freely available, but their proficiency has been insufficiently evaluated. In this study the authors have tested the current performance on the multiple-choice medical licensing exam of University of Antwerp (Belgium) of six widely used bots: ChatGPT (OpenAI), Bard (Google), New Bing (Microsoft), Claude instant (Anthropic), Claude+ (Anthropic) and GPT-4 (OpenAI). The primary outcome was the performance on the exam expressed as a proportion of correct answers. Secondary analyses were done for a variety of features in the exam questions: easy versus difficult questions, grammatically positive versus negative questions, and clinical vignettes versus theoretical questions. Reasoning errors and untruthful statements (hallucinations) in the bots' answers were examined. All bots passed the exam; Bing and GPT-4 (both 76% correct answers) outperformed the other bots (62-67%, p = 0.03) and students (61%). Bots performed worse on difficult questions (62%, p = 0.06), but outperformed students (32%) on those questions even more (p<0.01). Hallucinations were found in 7% of Bing's and GPT4's answers, significantly lower than Bard (22%, p<0.01) and Claude Instant (19%, p = 0.02). Although the creators of all bots try to some extent to avoid their bots being used as a medical doctor, none of the tested bots succeeded as none refused to answer all clinical case questions.Bing was able to detect weak or ambiguous exam questions. Bots could be used as a time efficient tool to improve the quality of a multiple-choice exam.

Local initiative supports case isolation and contact tracing during a SARS-CoV-2 surge in summer 2020: a community case study in Antwerp, Belgium.

In Antwerp, Belgium's second largest city, a COVID-19 surge in July 2020 predominantly affected neighborhoods with high ethnic diversity. Local volunteers reacted and set up an initiative to support contact tracing and self-isolation. We describe the origin, implementation, and transfer of this local initiative, based on semi-structured interviews of five key informants and document review. The initiative started in July 2020, when family physicians signaled a surge of SARS-CoV-2 infections among people of Moroccan descent. Family physicians feared that the mainstream contact tracing organized by the Flemish government through centralized call centers would not be efficient in halting this outbreak. They anticipated language barriers, mistrust, inability to investigate case clusters, and practical problems with self-isolation. It took 11 days to start up the initiative, with logistical support from the province and city of Antwerp. Family physicians referred SARS-CoV-2-infected index cases with complex needs (including language and social situation) to the initiative. Volunteer COVID coaches contacted cases, got a contextualized understanding of their living conditions, assisted with backward and forward contact tracing, offered support during self-isolation, and checked if infected contacts also needed support. Interviewed coaches were positive about the quality of the interaction: they described extensive open conversations with cases. The coaches reported back to referring family physicians and coordinators of the local initiative, who took additional action if necessary. Although interactions with affected communities were perceived as good, respondents considered that the number of referrals by family physicians was too low to have a meaningful impact on the outbreak. In September 2020, the Flemish government assigned the tasks of local contact tracing and case support to the local health system level (primary care zones). While doing so, they adopted elements of this local initiative, such as COVID coaches, tracing system, and extended questionnaires to talk with cases and contacts. This community case study illustrates how urgency can motivate people to action yet support from people with access to resources and coordination capacity is vital for effective organization and transition to long-term sustainability. From their conception, health policies should consider adaptability of new interventions to local contexts.

Cost effects of nurse led triage at an emergency department with the advice to consult the adjacent general practice cooperative for low-risk patients, a cluster randomised trial.

BACKGROUND: During the TRIAGE trial, emergency nurses diverted 13.3% of patients with low-risk complaints from a Belgian emergency department (ED) to the adjacent general practitioner cooperative (GPC). We examined the effects of this diversion on the total cost, insurance costs and patient costs, as charged on the invoice. Changes in the cost composition and the direct impact on revenues of both locations were examined as a secondary objective. METHODS: The differences in costs between intervention and control weekends were tested with two-sample t-tests and Kolmogorov-Smirnov (KS) tests. For the main outcomes an additional generalised linear model was created. Proportions of patients charged with certain costs were examined using Pearson's chi-square tests. Average revenues per weekend were compared using pooled t-tests. RESULTS: During intervention weekends, total costs increased by 3% (€3.3). The costs decreased by 8% (€2.2) for patients and increased by 6% (€5.5) for insurance, mainly driven by differences in physician fees. More patients were charged a consultation fee only (25% vs. 19%, p-value<0.01). The GPC's revenues increased by 13% (p-value=0.06); no change was found for the ED's revenues. CONCLUSION: The intervention reduced costs slightly for patients, while total costs and insurance costs slightly increased. When implementing triage systems with primary care involvement, the effects on the costs and revenues of the stakeholders should be monitored.

Differences in emergency nurse triage between a simulated setting and the real world, post hoc analysis of a cluster randomised trial.

OBJECTIVES: In the TRIAGE trial, a cluster randomised trial about diverting emergency department (ED) patients to a general practice cooperative (GPC) using a new extension to the Manchester Triage System, the difference in the proportion of patients assigned to the GPC was striking: 13.3% in the intervention group (patients were encouraged to comply to an ED or GPC assignment, real-world setting) and 24.7% in the control group (the assignment was not communicated, all remained at the ED, simulated setting). In this secondary analysis, we assess the differences in the use of the triage tool between intervention and control group and differences in costs and hospitalisations for patients assigned to the GPC. SETTING: ED of a general hospital and the adjacent GPC. PARTICIPANTS: 8038 patients (6294 intervention and 1744 control).Primary and secondary outcome measures proportion of patients with triage parameters (reason for encounter, discriminator and urgency category) leading to an assignment to the ED, proportion of patients for which the computer-generated GPC assignment was overruled, motivations for choosing certain parameters, costs (invoices) and hospitalisations. RESULTS: An additional 3.1% (p<0.01) of the patients in the intervention group were classified as urgent. Discriminators leading to the ED were registered for an additional 16.2% (p<0.01), mainly because of a perceived need for imaging. Nurses equally chose flow charts leading to the ED (p=0.41) and equally overruled the protocol (p=0.91). In the intervention group, the mean cost for patients assigned to the GPC was €23 (p<0.01) lower and less patients with an assignment to the GPC were hospitalised (1.0% vs 1.6%, p<0.01). CONCLUSION: Nurses used a triage tool more risk averse when it was used to divert patients to primary care as compared with a theoretical assignment to primary care. Outcomes from a simulated setting should not be extrapolated to real patients. TRIAL REGISTRATION NUMBER: NCT03793972.

Triaging and referring in adjacent general and emergency departments (the TRIAGE-trial): A process evaluation of medical staff experiences in a nurse-led triage system.

AIMS: This process evaluation aims at identifying the facilitators and inhibitors that influenced the successful uptake of a nurse-led triage system streaming low-risk patients from an emergency department (ED) to the general practitioner (GP). DESIGN & METHODS: Semi-structured interviews with ED nurses (n = 12), ED doctors (n = 6) from the ED of a Belgian general hospital and GPs (n = 5) affiliated with the adjacent GP cooperative (GPC). The process evaluation ran in parallel with the TRIAGE trial that started in March 2019 and ended 31st of December 2019. The first set of interviews was conducted in June 2019 and the second set in January 2020. Data were analysed based on grounded theory. RESULTS: Through a deductive framework, facilitators and inhibitors could be identified on three levels: the organisational, group and individual level. Main inhibitors are the degree of risk aversion of individual nurses, possible language barriers during delivery of the triage advice and the non-adapted ED infrastructure. Training on both the use of the triage protocol and effective delivery of the triage advice, in combination with periodical feedback from the GPC were the most important facilitators. CONCLUSION: Based on the process evaluation we can conclude that a consensus exists among stakeholders that the ED Nurses are considered ideally positioned to perform the triage of walk-in patients, although a certain degree of experience is necessary. Although the extended triage protocol and GPC referral increases the complexity and duration of triage and entails a higher workload for the triage nurses, ED nurses found it did lead to a lower (perceived) workload for the ED in general.

Non-compliance with a nurse's advice to visit the primary care provider: an exploratory secondary analysis of the TRIAGE-trial.

BACKGROUND: During the cluster randomised TRIAGE-trial, a nurse advised 13% of low-risk patients presenting at an emergency department in Belgium to visit the adjacent general practitioner cooperative. Patients had the right to refuse this advice. This exploratory study examines the characteristics of refusers by uncovering the determinants of non-compliance and its impact on costs, as charged on the patient's invoice. METHODS: Bivariate analyses with logistic regressions and T-tests were used to test the differences in patient characteristics, patient status, timing characteristics, and costs between refusers and non-refusers. A chi-square automatic interaction detection analysis was used to find the predictors of non-compliance. RESULTS: 23.50% of the patients refused the advice to visit the general practitioner cooperative. This proportion was mainly influenced by the nurse on duty (non-compliance rates per nurse ranging from 2.9% to 52.8%) and the patients' socio-economic status (receiving increased reimbursement versus not OR 1.37, 95%CI: 0.96 to 1.95). Additionally, non-compliance was associated (at the 0.10 significance level) with being male, not living nearby and certain reasons for encounter. Fewer patients refused when the nurse perceived crowding level as quiet relative to normal, and more patients refused during the evening. The mean cost was significantly higher for patients who refused, which was a result of more extensive examination and higher out-of-pocket expenses at the ED. CONCLUSIONS: The nurse providing the advice to visit the general practitioner cooperative has a central role in the likelihood of patients' refusal. Interventions to reduce non-compliance should aim at improving nurse-patient communication. Special attention may be required when managing patients with a lower socio-economic status. The overall mean cost was higher for refusers, illustrating the importance of compliance. TRIAL REGISTRATION: The trial was registered on registration number NCT03793972 on 04/01/2019.

How accurate is telephone triage in out-of-hours care? An observational trial in real patients.

OBJECTIVES: Patients in Belgium needing out-of-hours medical care have two options: the emergency department (ED) or a general practitioner (GP) on call. Currently, there is no triage system in Belgium, so patients do not know where they should go. However, patients who could be managed by a GP frequently present themselves at an ED without referral. GPs often organise themselves in a General Practitioners Cooperative (GPC). This study assesses the accuracy of a newly developed telephone triage guideline. METHODS: Observational real-time simulation: all walk-in patients at two GPCs and three EDs were asked to call a triage telephone number with their current medical problem. The operator handling this call registered an urgency level and a resource (ED, GP or ambulance) to deploy. The treating physician's opinion was used a the gold standard for correct triage. Patients were not informed about the outcome of the triage and continued the standard care path they had chosen. RESULTS: The overall sensitivity of the telephone triage for detecting patients who could be managed by a GP was 82% with a specificity of 53%. The correctness of the advice given by the operator according to the physicians was 71%, with 12% underestimation of urgency and 17% overestimation. At the GPC, the sensitivity for detecting patients requiring GP management/care was 91% with a specificity of 36%. At the ED, the sensitivity for detecting GP patients was 67% with a specificity of 48%. CONCLUSION: This study evaluates a new guideline for telephone triage, showing potential overtriage for patients wanting to attend the GPC, with possible inefficiency, and potential undertriage for patients wanting to attend the ED, with possible safety issues.

Antibiotic Prescribing Trends in Belgian Out-of-Hours Primary Care during the COVID-19 Pandemic: Observational Study Using Routinely Collected Health Data.

Antibiotic overprescribing is one of the main drivers of the global and growing problem of antibiotic resistance, especially in primary care and for respiratory tract infections (RTIs). RTIs are the most common reason for patients to consult out-of-hours (OOH) primary care. The COVID-19 pandemic has changed the way general practitioners (GPs) work, both during office hours and OOH. In Belgian OOH primary care, remote consultations with the possibility of issuing prescriptions and telephone triage were implemented. We aimed to describe the impact of COVID-19 on GPs' antibiotic prescribing during OOH primary care. In an observational study, using routinely collected health data from GP cooperatives (GPCs) in Flanders, we analyzed GPs' antibiotic prescriptions in 2019 (10 GPCs) and 2020 (20 GPCs) during OOH consultations (telephone and face-to-face). We used autoregressive integrated moving average (ARIMA) modeling to identify any changes after lockdowns were implemented. In total, 388,293 contacts and 268,430 prescriptions were analyzed in detail. The number of antibiotic prescriptions per weekend, per 100,000 population was 11.47 (95% CI: 9.08-13.87) or 42.9% lower after compared to before the implementation of lockdown among all contacts. For antibiotic prescribing per contact, we found a decrease of 12.2 percentage points (95% CI: 10.6-13.7) or 56.5% among all contacts and of 5.3 percentage points (95% CI: 3.7-6.9) or 23.2% for face-to-face contacts only. The decrease in the number of prescriptions was more pronounced for cases with respiratory symptoms that corresponded with symptoms of COVID-19 and for antibiotics that are frequently prescribed for RTIs, such as amoxicillin (a decrease of 64.9%) and amoxicillin/clavulanate (a decrease of 38.1%) but did not appear for others such as nitrofurantoin. The implementation of COVID-19 lockdown measures coincided with an unprecedented drop in the number of antibiotic prescriptions, which can be explained by a decrease in face-to-face patient contacts, as well as a lower number of antibiotics prescriptions per face-to-face patient contact. The decrease was seen for antibiotics used for RTIs but not for nitrofurantoin, the first-choice antibiotic for urinary tract infections.

Triaging and referring in adjacent general and emergency departments (the TRIAGE trial): A cluster randomised controlled trial.

OBJECTIVES: To determine whether a new triage system safely diverts a proportion of emergency department (ED) patients to a general practitioner cooperative (GPC). METHODS: Unblinded randomised controlled trial with weekends serving as clusters (three intervention clusters for each control). The intervention was triage by a nurse using a new extension to the Manchester Triage System assigning low-risk patients to the GPC. During intervention weekends, patients were encouraged to follow this assignment; it was not communicated during control weekends (all patients remained at the ED). The primary outcome was the proportion of patients assigned to and handled by the GPC during intervention weekends. The trial was randomised for the secondary outcome: the proportion of patients assigned to the GPC. Additional outcomes were association of these outcomes with possible confounders (study tool parameters, nurse, and patient characteristics), proportion of patients referred back to the ED by the GPC, hospitalisations, and performance of the study tool to detect primary care patients (the opinion of the treating physician was the gold standard). RESULTS: In the intervention group, 838/6294 patients (13.3%, 95% CI 12.5 to 14.2) were assigned to the GPC, in the control group this was 431/1744 (24.7%, 95% CI 22.7 to 26.8). In total, 599/6294 patients (9.5%, 95% CI 8.8 to 10.3) experienced the primary outcome which was influenced by the reason for encounter, age, and the nurse. 24/599 patients (4.0%, 95% CI 2.7 to 5.9) were referred back to the ED, three were hospitalised. Positive and negative predictive values of the studied tool during intervention weekends were 0.96 (95%CI 0.94 to 0.97) and 0.60 (95% CI 0.58 to 0.62). Out of the patients assigned to the GPC, 2.4% (95% CI 1.7 to 3.4) were hospitalised. CONCLUSIONS: ED nurses using a new tool safely diverted 9.5% of the included patients to primary care. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT03793972.

Increase in Anxiety-Related Out-of-Hours Primary Care Consultations Since COVID-19: An Observational Study Using Routine Data in Flanders.

Background: Survey studies suggest that COVID-19 has had a negative impact on the population's mental well-being. Routine registration data allow a more objective way for investigating such associations, complementary to self-report measures. This study investigates the level of out of hours (OOH) consultations for psychological problems since the start of the COVID-19 pandemic in Flanders, Belgium. Methods: The iCAREdata database is a clinical research database with routine data of OOH care, covering a large area in Flanders, Belgium. After defining the first wave and the second wave of COVID-19 in Flanders in time, we compared the number of consultations regarding psychological problems (in general, anxiety-related, depression-related, and sleep-related) between those periods, the period in between these waves, and the period before the start of COVID-19. Results: A significant rise in OOH consultations due to psychological-and more specifically, anxiety-related-problems is observed since the start of the COVID-19 pandemic in Flanders. Elevated levels are observed until the second wave. This finding is in sharp contrast with the general pattern of lower demand for primary healthcare during this period. The majority of these supplementary consultations happened by phone. Consultations regarding depression-related problems did not change over time. Sleep disturbances in the OOH setting were more common after the first wave. Conclusion: Despite some limitations, a large Flemish database with routine data on OOH care shows an increase in the number of consultations regarding psychological problems in general and anxiety-related problems since the start of the COVID-19-pandemic until the second wave.

Organisation and characteristics of out-of-hours primary care during a COVID-19 outbreak: A real-time observational study.

BACKGROUND: During the COVID-19 pandemic, general practitioners worldwide re-organise care in very different ways because of the lack of evidence-based protocols. OBJECTIVE: This paper describes the organisation and the characteristics of consultations in Belgian out-of-hours primary care during five weekends at the peak of a COVID-19 outbreak and compares it to a similar period in 2019. METHODS: Real-time observational study using pseudonymised routine clinical data extracted out of reports from home visits, telephone- and physical consultations (iCAREdata). Nine general practice cooperatives (GPCs) participated covering a population of 1 513 523. RESULTS: All GPCs rapidly re-organised care in order to handle the outbreak and provide a safe working environment. The average consultation rate was 222 per 100 000 citizens per weekend. These consultations were handled by telephone alone in 40% (N = 6293). A diagnosis at risk of COVID-19 was registered in 6692 (43%) consultations,. Out of 5311 physical consultations, 1460 were at risk of COVID-19 of which 443 (30%) did not receive prior telephone consultation to estimate this risk. Compared to 2019, the workload initially increased due to telephone consultations but afterwards declined drastically. The physical consultation rate declined by 45% with a marked decline in diagnoses unrelated to COVID-19. CONCLUSIONS: General practitioners can rapidly re-organise out-of-hours care to handle patient flows during a COVID-19 outbreak. Forty percent of the out-of-hours primary care contacts are handled by telephone consultations alone. We recommend to give a telephone consultation to all patients and not to rely on call takers to differentiate between infectious and regular care. The demand for physical consultations declined drastically provoking questions about patient's safety for care unrelated to COVID-19.

Performance of a new guideline for telephone triage in out-of-hours services in Belgium: A pilot study using simulated patients.

BACKGROUND: Patients in Belgium needing out-of-hours care have two options: the emergency department or the general practitioner on call often organised in a general practitioner cooperative. Currently, there is no triage system in Belgium so patients do not know where to go. METHODS: Our primary objective was to examine the ability of a newly developed telephone guideline, called 1733, to adequately estimate the urgency of health problems presented by simulated patients. Ten clinical vignettes were presented to 12 operators in a simulated phone call. The operators had to assign a protocol, urgency level and resource to dispatch (ambulance, general practitioner house visit, etc.) to each case. RESULTS: A total of 120 phone calls were analysed. The operators chose the right protocol in 69% and the correct urgency level in 35% of the cases. The proportion of under- and over-triage was 26% and 39%, respectively. There was important variation in between the operators. The sensitivity for detecting highly urgent cases was 0.42, the specificity 0.92. CONCLUSION: Using the new Belgian 1733 guideline for telephone triage, operators mostly chose the appropriate protocol but only chose the correct urgency in one out of three cases. In this phase of development, the studied telephone guideline is not ready for implementation.

Self-triage at an urgent care collaboration with and without information campaign.

BACKGROUND: Patients in Belgium needing out-of-hours care have two options: the emergency department (ED) or the general practitioner on call. The latter is often organized in a General Practice Cooperative (GPC). At the ED, there is an overload of patients who could be helped more efficiently by the GPC. RESEARCH QUESTION: What is the proportion of patients switching from the ED to the GPC (called voluntary switchers) with and without an information campaign? What are the characteristics of these patients? METHODS: Single-center prospective intervention trial during the opening hours of the GPC (only weekends: Friday 7.00 pm to Monday 7.00 am). The first 10 weekends there was no intervention. The next 24 weekends, patients in the ED were informed about the out-of-hours care in Belgium. The information contained several topics: characteristics of both services, where to go using examples, practicalities, and costs. This information was distributed through leaflets and broadcasted on a screen in five languages. RESULTS: During the study period, 7,453 patients entered the ED of which 330 are voluntary switchers. The proportion of voluntary switchers was 1.7 percent before and 5.4 percent after the intervention (p < 0.01). This effect remained stable for 10 more months after the study. The average number of patients presenting at the ED per hour was 3.1, whereas on hours with voluntary switchers was 5.1 (p < 0.01). The age distribution and epidemiological profile of the voluntary switchers resembles the one of primary care patients. The general practicioners (GPs) referred 6 percent of the voluntary switchers back to the ED. CONCLUSION: Co-location of the GPC and the ED and informing patients is a meaningful step toward a more profound collaboration.