Evaluating the Reliability of a Remote Acuity Prediction Tool in a Canadian Academic Emergency Department.

STUDY OBJECTIVE: There is increasing interest in harnessing artificial intelligence to virtually triage patients seeking care. The objective was to examine the reliability of a virtual machine learning algorithm to remotely predict acuity scores for patients seeking emergency department (ED) care by applying the algorithm to retrospective ED data. METHODS: This was a retrospective review of adult patients conducted at an academic tertiary care ED (annual census 65,000) from January 2021 to August 2022. Data including ED visit date and time, patient age, sex, reason for visit, presenting complaint and patient-reported pain score were used by the machine learning algorithm to predict acuity scores. The algorithm was designed to up-triage high-risk complaints to promote safety for remote use. The predicted scores were then compared to nurse-led triage scores previously derived in real time using the electronic Canadian Triage and Acuity Scale (eCTAS), an electronic triage decision-support tool used in the ED. Interrater reliability was estimated using kappa statistics with 95% confidence intervals (CIs). RESULTS: In total, 21,469 unique ED patient encounters were included. Exact modal agreement was achieved for 10,396 (48.4%) patient encounters. Interrater reliability ranged from poor to fair, as estimated using unweighted kappa (0.18, 95% CI 0.17 to 0.19), linear-weighted kappa (0.25, 95% CI 0.24 to 0.26), and quadratic-weighted kappa (0.36, 95% CI 0.35 to 0.37) statistics. Using the nurse-led eCTAS score as the reference, the machine learning algorithm overtriaged 9,897 (46.1%) and undertriaged 1,176 (5.5%) cases. Some of the presenting complaints under-triaged were conditions generally requiring further probing to delineate their nature, including abnormal lab/imaging results, visual disturbance, and fever. CONCLUSION: This machine learning algorithm needs further refinement before being safely implemented for patient use.

The Implementation of a Virtual Emergency Department: Multimethods Study Guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) Framework.

BACKGROUND: While the COVID-19 pandemic dramatically increased virtual care uptake across many health settings, it remains significantly underused in urgent care. OBJECTIVE: This study evaluated the implementation of a pilot virtual emergency department (VED) at an Ontario hospital that connected patients to emergency physicians through a web-based portal. We sought to (1) assess the acceptability of the VED model, (2) evaluate whether the VED was implemented as intended, and (3) explore the impact on quality of care, access to care, and continuity of care. METHODS: This evaluation used a multimethods approach informed by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. Data included semistructured interviews with patients and physicians as well as postvisit surveys from patients. Interviews were transcribed and analyzed using thematic analysis. Data from the surveys were described using summary statistics. RESULTS: From December 2020 to December 2021, the VED had a mean of 153 (SD 25) visits per month. Among them, 67% (n=677) were female, and 75% (n=758) had a family physician. Patients reported that the VED provided high-quality, timely access to care and praised the convenience, shorter appointments, and benefit of the calm, safe space afforded through virtual appointments. In instances where patients were directed to come into the emergency department (ED), physicians were able to provide a "warm handoff" to improve efficiency. This helped manage patient expectations, and the direct advice of the ED physician reassured them that the visit was warranted. There was broad initial uptake of VED shifts among ED physicians with 60% (n=22) completing shifts in the first 2 months and 42% (n=15) completing 1 or more shifts per month over the course of the pilot. There were no difficulties finding sufficient ED physicians for shifts. Most physicians enjoyed working in the VED, saw value for patients, and were motivated by patient satisfaction. However, some physicians were hesitant as they felt their expertise and skills as ED physicians were underused. The VED was implemented using an iterative staged approach with increased service capabilities over time, including access to ultrasounds, virtual follow-ups after a recent ED visit, and access to blood work, urine tests, and x-rays (at the hospital or a local community laboratory). Physicians recognized the value in supporting patients by advising on the need for an in-person visit, booking a diagnostic test, or referring them to a specialist. CONCLUSIONS: The VED had the support of physicians and facilitated care for low-acuity presentations with immediate benefits for patients. It has the potential to benefit the health care system by seeing patients through the web and guiding patients to in-person care only when necessary. Long-term sustainability requires a focus on understanding digital equity and enhanced access to rapid testing or investigations.

Health care utilization and outcomes of patients seen by virtual urgent care versus in-person emergency department care.

BACKGROUND: Virtual urgent care (VUC) is intended to support diversion of patients with low-acuity complaints and reduce the need for in-person emergency department visits. We aimed to describe subsequent health care utilization and outcomes of patients who used VUC compared with similar patients who had an in-person emergency department visit. METHODS: We used patient-level encounter data that were prospectively collected for patients using VUC services provided by 14 pilot programs in Ontario, Canada. We linked the data to provincial administrative databases to identify subsequent 30-day health care utilization and outcomes. We defined 2 subgroups of VUC users; those with a documented prompt referral to an emergency department by a VUC provider, and those without. We matched patients in each cohort to an equal number of patients presenting to an emergency department in person, based on encounter date, medical concern and the logit of a propensity score. For the subgroup of patients not promptly referred to an emergency department, we matched patients to those who were seen in an emergency department and then discharged home. RESULTS: Of the 19 595 patient VUC visits linked to administrative data, we matched 2129 patients promptly referred to the emergency department by a VUC provider to patients presenting to the emergency department in person. Index visit hospital admissions (9.4% v. 8.7%), 30-day emergency department visits (17.0% v. 17.5%), and hospital admissions (12.9% v. 11.0%) were similar between the groups. We matched 14 179 patients who were seen by a VUC provider with no documented referral to the emergency department. Patients seen by VUC were more likely to have a subsequent in-person emergency department visit within 72 hours (13.7% v. 7.0%), 7 days (16.5% v. 10.3%) and 30 days (21.9% v. 17.9%), but hospital admissions were similar within 72 hours (1.1% v. 1.3%), and higher within 30 days for patients who were discharged home from the emergency department (2.6% v. 3.4%). INTERPRETATION: The impact of the provincial VUC pilot program on subsequent health care utilization was limited. There is a need to better understand the inherent limitations of virtual care and ensure future virtual providers have timely access to in-person outpatient resources, to prevent subsequent emergency department visits.

Development of a machine learning-based acuity score prediction model for virtual care settings.

OBJECTIVE: Healthcare is increasingly digitized, yet remote and automated machine learning (ML) triage prediction systems for virtual urgent care use remain limited. The Canadian Triage and Acuity Scale (CTAS) is the gold standard triage tool for in-person care in Canada. The current work describes the development of a ML-based acuity score modelled after the CTAS system. METHODS: The ML-based acuity score model was developed using 2,460,109 de-identified patient-level encounter records from three large healthcare organizations (Ontario, Canada). Data included presenting complaint, clinical modifiers, age, sex, and self-reported pain. 2,041,987 records were high acuity (CTAS 1-3) and 416,870 records were low acuity (CTAS 4-5). Five models were trained: decision tree, k-nearest neighbors, random forest, gradient boosting regressor, and neural net. The outcome variable of interest was the acuity score predicted by the ML system compared to the CTAS score assigned by the triage nurse. RESULTS: Gradient boosting regressor demonstrated the greatest prediction accuracy. This final model was tuned toward up triaging to minimize patient risk if adopted into the clinical context. The algorithm predicted the same score in 47.4% of cases, and the same or more acute score in 95.0% of cases. CONCLUSIONS: The ML algorithm shows reasonable predictive accuracy and high predictive safety and was developed using the largest dataset of its kind to date. Future work will involve conducting a pilot study to validate and prospectively assess reliability of the ML algorithm to assign acuity scores remotely.

"It saved me from the emergency department": A qualitative study of patient experience of virtual urgent care in Ontario.

INTRODUCTION: In response to the COVID-19 pandemic, the Ontario Ministry of Health introduced a pilot program of 14 virtual urgent care (VUC) initiatives across the province to encourage physical distancing and provision of care by telephone and video-enabled visits. The implementation of the VUC pilot is currently being evaluated by an external academic team. The objective of this study was to understand patient experiences with VUC to determine barriers and facilitators to optimal virtual care as it rapidly expands during the current pandemic and beyond. METHOD: The qualitative component of the evaluation used one-on-one telephone interviews with patients, families, providers, and program administrators as the main method of data collection. Patient and family participants were invited to participate by the triage nurse after their VUC visit. Data analysis, using thematic analysis, occurred in conjunction with data collection to monitor emerging themes and areas for further exploration. RESULTS: Between April and October 2021, we completed 14 patient and/or family interviews from a representative cross-section of 6 pilot sites. Participants had a range of presenting complaints including infection, injury, medication side effects, and abdominal pain. The vast majority of participants were female (90%), and 70% were VUC patients themselves. Our analysis identified three key themes in the data which characterise patient and family member experience with VUC: a) emphasis on access to the ED; b) efficiency and quality of care; c) obtaining reassurance and next steps. CONCLUSION: Virtual care options are valued by patients and families; however, the nature of care needed by those accessing VUC and who can best provide that care needs to be evaluated to position it for sustainability. Understanding how virtual care performs from both a provider and patient perspective during the current crisis has implications for designing alternative care options beyond the COVID-19 pandemic.

Training Residents for the Future: A Virtual Care Rotation for Emergency Medicine.

OBJECTIVES: Virtual care (VC) is increasingly becoming a part of emergency medicine (EM) physician workflows, yet no formal digital health curricula exist within Canadian EM training programs. The objective was to design and pilot a VC elective rotation for EM residents to help address this gap and better prepare them for future VC practice. METHODS: The current work describes the design and implementation of a 4-week VC elective rotation for EM residents. The rotation consisted of VC shifts, medical transport shifts, one-on-one discussions with various stakeholders, weekly thematic articles, and a final project deliverable. RESULTS: The rotation was well received by all stakeholders, and the quality of feedback and one-on-one teaching were highlighted as strengths. Future work will consider the optimal delivery timing of this type of curricula, whether all EM residents should receive basic training in VC, and how our current findings may be generalizable to other VC sites. CONCLUSION: A formal digital health curriculum for EM residents supports competency development for delivering VC as part of future EM practice.

Learning From a Regional Approach: Integration to Scale, Spread, and Sustain Virtual Urgent Care.

While new offerings of virtual urgent care services from peer hospitals faltered after initial provincial pilot funding lapsed, our 3 regional academic health sciences centers decided to partner to enhance patient access, achieve efficiencies, and support long-term sustainability. Utilizing the Development Model for Integrated Care framework, we progressed through the 4 phases to ensure joint success and high-quality care: (1) initiative and design phase-individual parallel projects but with strong collaborations and broad stakeholder engagement; (2) experimental and execution phase-continuous quality improvement approach for governance, policies, and processes; (3) expansion and monitoring phase-weekly leadership touchpoints on key performance indicators; and (4) consolidation and transformation phase-sustainability through ongoing funding.

Demographic characteristics, outcomes and experience of patients using virtual urgent care services from 14 emergency department led sites in Ontario.

INTRODUCTION: As part of the COVID-19 pandemic response, the Ontario Ministry of Health funded a virtual care pilot program intended to support emergency department (ED) diversion of patients with low acuity complaints and reduce the need for face-to-face contact. The objective was to describe the demographic characteristics, outcomes and experience of patients using the provincial pilot program. METHODS: This was a prospective cohort study of patients using virtual care services provided by 14 ED-led pilot sites from December 2020 to September 2021. Patients who completed a virtual visit were invited by email to complete a standardized, 25-item online survey, which included questions related to satisfaction and patient-reported outcome measures. RESULTS: There were 22,278 virtual visits. When patients were asked why they contacted virtual urgent care, of the 82.7% patients who had a primary care provider, 31.0% said they could not make a timely appointment with their family physician. Rash, fever, abdominal pain, and COVID-19 vaccine queries represented 30% of the presenting complaints. Of 19,613 patients with a known disposition, 12,910 (65.8%) were discharged home and 3,179 (16.2%) were referred to the ED. Of the 2,177 survey responses, 94% rated their overall experience as 8/10 or greater. More than 80% said they had answers to all the questions they had related to their health concern, believed they were able to manage the issue, had a plan they could follow, and knew what to do if the issue got worse or came back. CONCLUSIONS: Many presenting complaints were low acuity, and most patients had a primary care provider, but timely access was not available. Future work should focus on health equity to ensure virtual care is accessible to underserved populations. We question if virtual urgent care can be safely and more economically provided by non-emergency physicians.

RéSUMé: INTRODUCTION: Dans le cadre de la réponse à la pandémie de COVID-19, le ministère de la Santé de l'Ontario a financé un programme pilote de soins virtuels visant à soutenir la réorientation vers les services d'urgence des patients présentant des problèmes de faible acuité et à réduire le besoin de contact en personne. L'objectif était de décrire les caractéristiques démographiques, les résultats et l'expérience des patients utilisant le programme pilote provincial. MéTHODES: Il s'agissait d'une étude de cohorte prospective de patients utilisant des services de soins virtuels fournis par 14 sites pilotes dirigés par des services d'urgence, de décembre 2020 à septembre 2021. Les patients qui ont effectué une visite virtuelle ont été invités par courriel à répondre à une enquête en ligne standardisée de 25 questions, qui comprenait des questions relatives à la satisfaction et aux résultats rapportés par les patients. RéSULTATS: Il y a eu 22 278 visites virtuelles. Lorsqu'on a demandé aux patients pourquoi ils avaient contacté les soins urgents virtuels, sur les 82,7 % de patients qui avaient un prestataire de soins primaires, 31,0 % ont répondu qu'ils n'avaient pas pu obtenir un rendez-vous en temps voulu avec leur médecin de famille. Les éruptions cutanées, la fièvre, les douleurs abdominales et les interrogations sur le vaccin COVID-19 représentaient 30 % des plaintes présentées. Sur les 19 613 patients dont la disposition était connue, 12 910 (65,8 %) ont été renvoyés chez eux et 3 179 (16,2 %) ont été orientés vers les urgences. Sur les 2 177 réponses à l'enquête, 94 % ont attribué une note de 8/10 ou plus à leur expérience globale. Plus de 80 % d'entre eux ont déclaré avoir obtenu des réponses à toutes les questions qu'ils se posaient sur leur problème de santé, se croire capables de le gérer, avoir un plan qu'ils pouvaient suivre et savoir quoi faire si le problème s'aggravait ou revenait. CONCLUSIONS: De nombreuses plaintes présentées étaient de faible acuité, et la plupart des patients avaient un fournisseur de soins primaires, mais l'accès en temps opportun n'était pas disponible. Les travaux futurs devraient se concentrer sur l'équité en matière de santé pour s'assurer que les soins virtuels sont accessibles aux populations mal desservies et nous nous demandons si ces services peuvent être fournis en toute sécurité et de manière plus économique par des médecins non urgentistes.

Implementing a Virtual Emergency Department: Qualitative Study Using the Normalization Process Theory.

BACKGROUND: COVID-19 necessitated the rapid implementation and uptake of virtual health care; however, virtual care's potential role remains unclear in the urgent care setting. In December 2020, the first virtual emergency department (ED) in the Greater Toronto Area was piloted at Sunnybrook Health Sciences Centre by connecting patients to emergency physicians through an online portal. OBJECTIVE: This study aims to understand whether and how ED physicians were able to integrate a virtual ED alongside in-person operations. METHODS: We conducted semistructured interviews with ED physicians guided by the Normalization Process Theory (NPT). The NPT provides a framework to understand how individuals and teams navigate the process of embedding new models of care as part of normal practice. All physicians who had worked within the virtual ED model were invited to participate. Data were analyzed using a combination of inductive and deductive techniques informed by the NPT. RESULTS: A total of 14 physicians were interviewed. Participant experiences were categorized into 1 of 2 groups: 1 group moved to normalize the virtual ED in practice, while the other described barriers to routine adoption. These groups differed in their perception of the patient benefits as well as the perceived role in the virtual ED. The group that normalized the virtual ED model saw value for patients (coherence) and was motivated by patient satisfaction witnessed (reflexive monitoring) at the end of the virtual appointment. By contrast, the other group did not find virtual ED work reflective of the perceived role of urgent care (cognitive participation) and felt their skills as ED physicians were underutilized. The limited ability to examine patients and a sense that patient issues were not fully resolved at the end of the virtual appointment caused frustration among the second group. CONCLUSIONS: As further digital integration within the health care system occurs, it will be essential to support the evolution of staff skill sets to ensure physicians are satisfied with the care they are providing to their patients, while also ensuring the technology and process are efficient.

Designs, facilitators, barriers, and lessons learned during the implementation of emergency department led virtual urgent care programs in Ontario, Canada.

Introduction: Virtual patient care has seen incredible growth since the beginning of the COVID-19 pandemic. To provide greater access to safe and timely urgent care, in the fall of 2020, the Ministry of Health introduced a pilot program of 14 virtual urgent care (VUC) initiatives across the province of Ontario. The objective of this paper was to describe the overall design, facilitators, barriers, and lessons learned during the implementation of seven emergency department (ED) led VUC pilot programs in Ontario, Canada. Methods: We assembled an expert panel of 13 emergency medicine physicians and researchers with experience leading and implementing local VUC programs. Each VUC program lead was asked to describe their local pilot program, share common facilitators and barriers to adoption of VUC services, and summarize lessons learned for future VUC design and development. Results: Models of care interventions varied across VUC pilot programs related to triage, staffing, technology, and physician remuneration. Common facilitators included local champions to guide program delivery, provincial funding support, and multi-modal marketing and promotions. Common barriers included behaviour change strategies to support adoption of a new service, access to high-quality information technology to support new workflow models that consider privacy, risk, and legal perspectives, and standardized data collection which underpin overall objective impact assessments. Conclusions: These pilot programs were rapidly implemented to support safe access to care and ED diversion of patients with low acuity issues during the COVID-19 pandemic. The heterogeneity of program implementation respects local autonomy yet may present challenges for sustainability efforts and future funding considerations.

Residents' perspective of quality improvement and patient safety education in Canadian emergency medicine residency programs.

OBJECTIVES: Quality improvement and patient safety (QIPS) competencies are increasingly important in emergency medicine (EM) and are now included in the CanMEDS framework. We conducted a survey aimed at determining the Canadian EM residents' perspectives on the level of QIPS education and support available to them. METHODS: An electronic survey was distributed to all Canadian EM residents from the Royal College and Family Medicine training streams. The survey consisted of multiple-choice, Likert, and free-text entry questions aimed at understanding familiarity with QIPS, local opportunities for QIPS projects and mentorship, and the desire for further QIPS education and involvement. RESULTS: Of 535 EM residents, 189 (35.3%) completed the survey, representing all 17 medical schools; 77.2% of respondents were from the Royal College stream; 17.5% of respondents reported that QIPS methodologies were formally taught in their residency program; 54.7% of respondents reported being "somewhat" or "very" familiar with QIPS; 47.2% and 51.5% of respondents reported either "not knowing" or "not having readily available" opportunities for QIPS projects and QIPS mentorship, respectively; 66.9% of respondents indicated a desire for increased QIPS teaching; and 70.4% were interested in becoming involved with QIPS training and initiatives. CONCLUSIONS: Many Canadian EM residents perceive a lack of QIPS educational opportunities and support in their local setting. They are interested in receiving more QIPS education, as well as project and mentorship opportunities. Supporting residents with a robust QIPS educational and mentorship framework may build a cohort of providers who can enhance the local delivery of care.

Faculty development in the age of competency-based medical education: A needs assessment of Canadian emergency medicine faculty and senior trainees.

OBJECTIVES: The Royal College of Physicians and Surgeons of Canada (RCPSC) emergency medicine (EM) programs transitioned to the Competence by Design training framework in July 2018. Prior to this transition, a nation-wide survey was conducted to gain a better understanding of EM faculty and senior resident attitudes towards the implementation of this new program of assessment. METHODS: A multi-site, cross-sectional needs assessment survey was conducted. We aimed to document perceptions about competency-based medical education, attitudes towards implementation, perceived/prompted/unperceived faculty development needs. EM faculty and senior residents were nominated by program directors across RCPSC EM programs. Simple descriptive statistics were used to analyse the data. RESULTS: Between February and April 2018, 47 participants completed the survey (58.8% response rate). Most respondents (89.4%) thought learners should receive feedback during every shift; 55.3% felt that they provided adequate feedback. Many respondents (78.7%) felt that the ED would allow for direct observation, and most (91.5%) participants were confident that they could incorporate workplace-based assessments (WBAs). Although a fair number of respondents (44.7%) felt that Competence by Design would not impact patient care, some (17.0%) were worried that it may negatively impact it. Perceived faculty development priorities included feedback delivery, completing WBAs, and resident promotion decisions. CONCLUSIONS: RCPSC EM faculty have positive attitudes towards competency-based medical education-relevant concepts such as feedback and opportunities for direct observation via WBAs. Perceived threats to Competence by Design implementation included concerns that patient care and trainee education might be negatively impacted. Faculty development should concentrate on further developing supervisors' teaching skills, focusing on feedback using WBAs.

INTRODUCTION: Les programmes de médecine d'urgence (MU) du Collège royal des médecins et chirurgiens du Canada sont passés, en juillet 2018, à un nouveau modèle de formation appelé Compétence par conception. Avant ce passage, une enquête avait été menée à l'échelle nationale pour dégager une meilleure compréhension des attitudes du personnel enseignant et des résidents séniors en MU à l'égard de la mise en œuvre du nouveau programme d'évaluation. MÉTHODE: Il s'agit d'une enquête transversale et multicentrique sur l'évaluation des besoins. Elle visait à recueillir des renseignements sur les perceptions des personnes concernées sur la formation médicale axée sur les compétences, sur leurs attitudes à l'égard de la mise en application du programme ainsi que sur les besoins perçus, suscités ou inaperçus du personnel en matière de perfectionnement. Les membres du personnel enseignant en MU et les résidents séniors ont été désignés par les directeurs de tous les programmes de MU du Collège royal. Les données ont été analysées à l'aide de simples statistiques descriptives. RÉSULTATS: Au total, 47 participants ont rempli le questionnaire d'enquête (taux de réponse : 58,8%) entre février et avril 2018. La plupart des répondants (89,4%) ont indiqué que les apprenants devraient recevoir de la rétroaction à tous les postes de travail, et 55,3% avaient l'impression de donner une rétroaction adéquate. Bon nombre de répondants (78,7%) étaient également d'avis que les services des urgences se prêtaient bien à l'observation directe, et la plupart des participants (91,5%) avaient bon espoir d'intégrer les évaluations en milieu de travail (EMT). Par ailleurs, si un assez bon nombre de répondants (44,7%) croyaient que la formation axée sur la Compétence par conception n'aurait aucune incidence sur les soins aux patients, d'autres (17,0%) s'en inquiétaient. Enfin, les priorités perçues en matière de perfectionnement du personnel comprenaient la communication des rétroactions, la réalisation des EMT et les décisions relatives à la promotion des résidents. CONCLUSION: Le personnel enseignant en MU du Collège royal a des attitudes favorables à l'égard de certains concepts liés à la formation médicale axée sur les compétences, tels que les rétroactions et les possibilités d'observation directe au moyen des EMT. Par contre, la mise en œuvre du programme de Compétence par conception suscite également des craintes, comme une incidence défavorable sur les soins aux patients et la formation des stagiaires. Le perfectionnement du personnel enseignant devrait donc porter davantage sur les aptitudes à enseigner des superviseurs, notamment sur la communication des rétroactions à l'aide des EMT.