An Institutional Curriculum for Opioid Prescribing Education: Outcomes From 2017 to 2022.

INTRODUCTION: Prescription opioids, including those prescribed after surgery, have greatly contributed to the US opioid epidemic. Educating opioid prescribers is a crucial component of ensuring the safe use of opioids among surgical patients. METHODS: An annual opioid prescribing education curriculum was implemented among new surgical prescribers at our institution between 2017 and 2022. The curriculum includes a single 75-min session which is comprised of several components: pain medications (dosing, indications, and contraindications); patients at high risk for uncontrolled pain and/or opioid misuse or abuse; patient monitoring and care plans; and state and federal regulations. Participants were asked to complete an opioid knowledge assessment before and after the didactic session. RESULTS: Presession and postsession assessments were completed by 197 (89.6%) prescribers. Across the five studied years, the median presession score was 54.5%. This increased to 63.6% after completion of the curriculum, representing a median relative knowledge increase of 18.2%. The median relative improvement was greatest for preinterns and interns (18.2% for both groups); smaller improvements were observed for postgraduate year 2-5 residents (9.1%) and advanced practice providers (9.1%). On a scale of 1 to 10 (with 5 being comfortable), median (interquartile range) self-reported comfort in prescribing opioids increased from 3 (2-5) before education to 5 (4-6) after education (P < 0.001). CONCLUSIONS: Each year, the curriculum substantially improved provider knowledge of and comfort in opioid prescribing. Despite increased national awareness of the opioid epidemic and increasing institutional initiatives to improve opioid prescribing practices, there was a sustained knowledge and comfort gap among new surgical prescribers. The observed effects of our opioid education curriculum highlight the value of a simple and efficient educational initiative.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions.

OBJECTIVE: To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting. BACKGROUND: To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown. METHODS: Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. RESULTS: Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy. CONCLUSIONS: Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

The Impact of Concurrent Multi-Service Coverage on Quality and Safety in Trauma Care.

BACKGROUND: At many trauma centers in the United States, one acute care surgeon is responsible for overnight coverage of both the emergency general surgery (EGS) and trauma services. The impact of this scheduling phenomenon on the quality and safety of trauma care has not been studied. METHODS: Overnight (12:00 AM to 7:00 AM) trauma admissions to an academic Level 1 trauma center from 2013-2015 were studied after the institution adopted this scheduling phenomenon. Admissions were divided into two groups based on whether the admitting surgeon covered only the trauma service, or both the trauma and EGS services ("multi-service coverage"). Four major outcomes (e.g., mortality and complications), six quality metrics (e.g., time to first OR visit and unplanned transfers to the ICU), and procedural utilization patterns were compared. RESULTS: A total of 1046 admissions were included. There were no differences in any major outcomes between the two exposure groups, including any National Trauma Data Bank-defined complication (OR 1.1, 95% CI 0.8-1.5, P= 0.5). Quality metrics dependent on the admitting surgeon remained unchanged, including attending presence at the highest-level trauma activations within 15 min of arrival (93% versus 86%, P= 0.07) and time to urgent operative intervention (68 min versus 82 min, P= 0.9). There were no differences in the number of laboratory and imaging studies (4.1 versus 4.1, P= 0.9) or bedside interventions (1.8 versus 2.1, P= 0.4) performed per patient by the admitting surgeon. Multivariate logistic regression did not identify multi-service coverage as an independent risk factor for adverse patient outcomes or quality metrics. CONCLUSIONS: Trauma admissions under a surgeon covering multiple services simultaneously had similar outcomes, quality metrics, and procedural utilization patterns compared to trauma admissions under surgeons covering only the trauma service. Despite concerns that multiple-service coverage may overburden one acute care surgeon, time-dependent quality metrics and studies done during the initial workup of trauma patients remained unchanged. These findings suggest that simultaneous trauma and EGS service coverage by one acute care surgeon does not adversely impact trauma patient care.

Distinguishing Admissions Specifically for COVID-19 From Incidental SARS-CoV-2 Admissions: National Retrospective Electronic Health Record Study.

BACKGROUND: Admissions are generally classified as COVID-19 hospitalizations if the patient has a positive SARS-CoV-2 polymerase chain reaction (PCR) test. However, because 35% of SARS-CoV-2 infections are asymptomatic, patients admitted for unrelated indications with an incidentally positive test could be misclassified as a COVID-19 hospitalization. Electronic health record (EHR)-based studies have been unable to distinguish between a hospitalization specifically for COVID-19 versus an incidental SARS-CoV-2 hospitalization. Although the need to improve classification of COVID-19 versus incidental SARS-CoV-2 is well understood, the magnitude of the problems has only been characterized in small, single-center studies. Furthermore, there have been no peer-reviewed studies evaluating methods for improving classification. OBJECTIVE: The aims of this study are to, first, quantify the frequency of incidental hospitalizations over the first 15 months of the pandemic in multiple hospital systems in the United States and, second, to apply electronic phenotyping techniques to automatically improve COVID-19 hospitalization classification. METHODS: From a retrospective EHR-based cohort in 4 US health care systems in Massachusetts, Pennsylvania, and Illinois, a random sample of 1123 SARS-CoV-2 PCR-positive patients hospitalized from March 2020 to August 2021 was manually chart-reviewed and classified as "admitted with COVID-19" (incidental) versus specifically admitted for COVID-19 ("for COVID-19"). EHR-based phenotyping was used to find feature sets to filter out incidental admissions. RESULTS: EHR-based phenotyped feature sets filtered out incidental admissions, which occurred in an average of 26% of hospitalizations (although this varied widely over time, from 0% to 75%). The top site-specific feature sets had 79%-99% specificity with 62%-75% sensitivity, while the best-performing across-site feature sets had 71%-94% specificity with 69%-81% sensitivity. CONCLUSIONS: A large proportion of SARS-CoV-2 PCR-positive admissions were incidental. Straightforward EHR-based phenotypes differentiated admissions, which is important to assure accurate public health reporting and research.

Evaluation of U.S. state opioid prescribing restrictions using patient opioid consumption patterns from a single, urban, academic institution.

Background: Since 2017, states, insurers, and pharmacies have placed blanket limits on the duration and quantity of opioid prescriptions. In many states, overlapping duration and daily dose limits yield maximum prescription limits of 150-350 morphine milligram equivalents (MMEs). There is limited knowledge of how these restrictions compare with actual patient opioid consumption; while changes in prescription patterns and opioid misuse rates have been studied, these are, at best, weak proxies for actual pain control consumption. We sought to determine how patients undergoing surgery would be affected by opioid prescribing restrictions using actual patient opioid consumption data. Methods: We constructed a prospective database of post-discharge opioid consumption: patients undergoing surgery at one institution were called after discharge to collect opioid consumption data. Patients whose opioid consumption exceeded 150 and 350 MME were identified. Results: Two thousand nine hundred and seventy-one patients undergoing 54 common surgical procedures were included in our study. Twenty-one percent of patients consumed more than the 150 MME limit. Only 7% of patients consumed above the 350 MME limit. Typical (non-outlier) opioid consumption, defined as less than the 75th percentile of consumption for any given procedure, exceeded the 150 MME and 350 MME limits for 41 and 7% of procedures, respectively. Orthopedic, spinal/neurosurgical, and complex abdominal procedures most commonly exceeded these limits. Conclusions: While most patients undergoing surgery are unaffected by recent blanket prescribing limits, those undergoing a specific subset of procedures are likely to require more opioids than the restrictions permit; providers should be aware that these patients may require a refill to adequately control post-surgical pain. Real consumption data should be used to guide these restrictions and inform future interventions so the risk of worsened pain control (and its troublesome effects on opioid misuse) is minimized. Procedure-specific prescribing limits may be one approach to prevent misuse, while also optimizing post-operative pain control.

Electronic Communication of Protected Health Information: Privacy, Security, and HIPAA Compliance.

PURPOSE: Technology has enhanced modern health care delivery, particularly through accessibility to health information and ease of communication with tools like mobile device messaging (texting). However, text messaging has created new risks for breach of protected health information (PHI). In the current study, we sought to evaluate hand surgeons' knowledge and compliance with privacy and security standards for electronic communication by text message. METHODS: A cross-sectional survey of the American Society for Surgery of the Hand membership was conducted in March and April 2016. Descriptive and inferential statistical analyses were performed of composite results as well as relevant subgroup analyses. RESULTS: A total of 409 responses were obtained (11% response rate). Although 63% of surgeons reported that they believe that text messaging does not meet Health Insurance Portability and Accountability Act of 1996 security standards, only 37% reported they do not use text messages to communicate PHI. Younger surgeons and respondents who believed that their texting was compliant were statistically significantly more like to report messaging of PHI (odds ratio, 1.59 and 1.22, respectively). DISCUSSION: A majority of hand surgeons in this study reported the use of text messaging to communicate PHI. Of note, neither the Health Insurance Portability and Accountability Act of 1996 statute nor US Department of Health and Human Services specifically prohibits this form of electronic communication. To be compliant, surgeons, practices, and institutions need to take reasonable security precautions to prevent breach of privacy with electronic communication. CLINICAL RELEVANCE: Communication of clinical information by text message is not prohibited under Health Insurance Portability and Accountability Act of 1996, but surgeons should use appropriate safeguards to prevent breach when using this form of communication.

An operational guide to translational clinical machine learning in academic medical centers.

Few published data science tools are ever translated from academia to real-world clinical settings for which they were intended. One dimension of this problem is the software engineering task of turning published academic projects into tools that are usable at the bedside. Given the complexity of the data ecosystem in large health systems, this task often represents a significant barrier to the real-world deployment of data science tools for prospective piloting and evaluation. Many information technology companies have created Machine Learning Operations (MLOps) teams to help with such tasks at scale, but the low penetration of home-grown data science tools in regular clinical practice precludes the formation of such teams in healthcare organizations. Based on experiences deploying data science tools at two large academic medical centers (Beth Israel Deaconess Medical Center, Boston, MA; Mayo Clinic, Rochester, MN), we propose a strategy to facilitate this transition from academic product to operational tool, defining the responsibilities of the principal investigator, data scientist, machine learning engineer, health system IT administrator, and clinician end-user throughout the process. We first enumerate the technical resources and stakeholders needed to prepare for model deployment. We then propose an approach to planning how the final product will work from data extraction and analysis to visualization of model outputs. Finally, we describe how the team should execute on this plan. We hope to guide health systems aiming to deploy minimum viable data science tools and realize their value in clinical practice.

Mobilizing data during a crisis: Building rapid evidence pipelines using multi-institutional real world data.

The COVID-19 pandemic generated tremendous interest in using real world data (RWD). Many consortia across the public and private sectors formed in 2020 with the goal of rapidly producing high-quality evidence from RWD to guide medical decision-making, public health priorities, and more. Experiences were gathered from five large consortia on rapid multi-institutional evidence generation during the COVID-19 pandemic. Insights have been compiled across five dimensions: consortium composition, governance structure and alignment of priorities, data sharing, data analysis, and evidence dissemination. The purpose of this piece is to offer guidance on building large-scale multi-institutional RWD analysis pipelines for future public health issues. The composition of each consortium was largely influenced by existing collaborations. A central set of priorities for evidence generation guided each consortium, however different approaches to governance emerged. Challenges surrounding limited access to clinical data due to various contributors were overcome in unique ways. While all consortia used different methods to construct and analyze patient cohorts ranging from centralized to federated approaches, all proved effective for generating meaningful real-world evidence. Actionable recommendations for clinical practice and public health agencies were made from translating insights from consortium analyses. Each consortium was successful in rapidly answering questions about COVID-19 diagnosis and treatment despite all taking slightly different approaches to data sharing and analysis. Leveraging RWD, leveraged in a manner that applies scientific rigor and transparency, can complement higher-level evidence and serve as an important adjunct to clinical trials to quickly guide policy and critical care, especially for a pandemic response.

Postoperative Opioid Prescribing via Rule-Based Guidelines Derived from In-Hospital Consumption: An Assessment of Efficacy Based on Postdischarge Opioid Use.

BACKGROUND: Many institutions have developed operation-specific guidelines for opioid prescribing. These guidelines rarely incorporate in-hospital opioid consumption, which is highly correlated with consumption. We compare outcomes of several patient-centered approaches to prescribing that are derived from in-hospital consumption, including several experimental, rule-based prescribing guidelines and our current institutional guideline. STUDY DESIGN: We performed a retrospective, cohort study of all adults undergoing surgery at a single-academic medical center. Several rule-based guidelines, derived from in-hospital consumption (quantity of opioids consumed within 24 hours of discharge), were used to specify the theoretical quantity of opioid prescribed on discharge. The efficacy of the experimental guidelines was compared with 3 references: an approximation of our institution's tailored prescribing guideline; prescribing all patients the typical quantity of opioids consumed for patients undergoing the same operation; and a representative rule-based, tiered framework. For each scenario, we calculated the penalized residual sum of squares (reflecting the composite deviation from actual patient consumption, with 15% penalty for overprescribing) and the proportion of opioids consumed relative to prescribed. RESULTS: A total of 1,048 patients met inclusion criteria. Mean (SD) and median (interquartile range [IQR]) quantity of opioids consumed within 24 hours of discharge were 11.2 (26.9) morphine milligram equivalents and 0 (0 to 15) morphine milligram equivalents. Median (IQR) postdischarge consumption was 16 (0 to 150) morphine milligram equivalents. Our institutional guideline and the previously validated rule-based guideline outperform alternate approaches, with median (IQR) differences in prescribed vs consumed opioids of 0 (-60 to 27.25) and 37.5 (-37.5 to 37.5), respectively, corresponding to penalized residual sum of squares of 39,817,602 and 38,336,895, respectively. CONCLUSIONS: Rather than relying on fixed quantities for defined operations, rule-based guidelines offer a simple yet effective method for tailoring opioid prescribing to in-hospital consumption.

Evaluating capabilities of large language models: Performance of GPT-4 on surgical knowledge assessments.

BACKGROUND: Artificial intelligence has the potential to dramatically alter health care by enhancing how we diagnose and treat disease. One promising artificial intelligence model is ChatGPT, a general-purpose large language model trained by OpenAI. ChatGPT has shown human-level performance on several professional and academic benchmarks. We sought to evaluate its performance on surgical knowledge questions and assess the stability of this performance on repeat queries. METHODS: We evaluated the performance of ChatGPT-4 on questions from the Surgical Council on Resident Education question bank and a second commonly used surgical knowledge assessment, referred to as Data-B. Questions were entered in 2 formats: open-ended and multiple-choice. ChatGPT outputs were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat queries. RESULTS: A total of 167 Surgical Council on Resident Education and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71.3% and 67.9% of multiple choice and 47.9% and 66.1% of open-ended questions for Surgical Council on Resident Education and Data-B, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained nonobvious insights. Common reasons for incorrect responses included inaccurate information in a complex question (n = 16, 36.4%), inaccurate information in a fact-based question (n = 11, 25.0%), and accurate information with circumstantial discrepancy (n = 6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of questions answered incorrectly on the first query; the response accuracy changed for 6/16 (37.5%) questions. CONCLUSION: Consistent with findings in other academic and professional domains, we demonstrate near or above human-level performance of ChatGPT on surgical knowledge questions from 2 widely used question banks. ChatGPT performed better on multiple-choice than open-ended questions, prompting questions regarding its potential for clinical application. Unique to this study, we demonstrate inconsistency in ChatGPT responses on repeat queries. This finding warrants future consideration including efforts at training large language models to provide the safe and consistent responses required for clinical application. Despite near or above human-level performance on question banks and given these observations, it is unclear whether large language models such as ChatGPT are able to safely assist clinicians in providing care.