An Institutional Curriculum for Opioid Prescribing Education: Outcomes From 2017 to 2022.

INTRODUCTION: Prescription opioids, including those prescribed after surgery, have greatly contributed to the US opioid epidemic. Educating opioid prescribers is a crucial component of ensuring the safe use of opioids among surgical patients. METHODS: An annual opioid prescribing education curriculum was implemented among new surgical prescribers at our institution between 2017 and 2022. The curriculum includes a single 75-min session which is comprised of several components: pain medications (dosing, indications, and contraindications); patients at high risk for uncontrolled pain and/or opioid misuse or abuse; patient monitoring and care plans; and state and federal regulations. Participants were asked to complete an opioid knowledge assessment before and after the didactic session. RESULTS: Presession and postsession assessments were completed by 197 (89.6%) prescribers. Across the five studied years, the median presession score was 54.5%. This increased to 63.6% after completion of the curriculum, representing a median relative knowledge increase of 18.2%. The median relative improvement was greatest for preinterns and interns (18.2% for both groups); smaller improvements were observed for postgraduate year 2-5 residents (9.1%) and advanced practice providers (9.1%). On a scale of 1 to 10 (with 5 being comfortable), median (interquartile range) self-reported comfort in prescribing opioids increased from 3 (2-5) before education to 5 (4-6) after education (P < 0.001). CONCLUSIONS: Each year, the curriculum substantially improved provider knowledge of and comfort in opioid prescribing. Despite increased national awareness of the opioid epidemic and increasing institutional initiatives to improve opioid prescribing practices, there was a sustained knowledge and comfort gap among new surgical prescribers. The observed effects of our opioid education curriculum highlight the value of a simple and efficient educational initiative.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions.

OBJECTIVE: To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting. BACKGROUND: To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown. METHODS: Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. RESULTS: Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy. CONCLUSIONS: Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

The Impact of Concurrent Multi-Service Coverage on Quality and Safety in Trauma Care.

BACKGROUND: At many trauma centers in the United States, one acute care surgeon is responsible for overnight coverage of both the emergency general surgery (EGS) and trauma services. The impact of this scheduling phenomenon on the quality and safety of trauma care has not been studied. METHODS: Overnight (12:00 AM to 7:00 AM) trauma admissions to an academic Level 1 trauma center from 2013-2015 were studied after the institution adopted this scheduling phenomenon. Admissions were divided into two groups based on whether the admitting surgeon covered only the trauma service, or both the trauma and EGS services ("multi-service coverage"). Four major outcomes (e.g., mortality and complications), six quality metrics (e.g., time to first OR visit and unplanned transfers to the ICU), and procedural utilization patterns were compared. RESULTS: A total of 1046 admissions were included. There were no differences in any major outcomes between the two exposure groups, including any National Trauma Data Bank-defined complication (OR 1.1, 95% CI 0.8-1.5, P= 0.5). Quality metrics dependent on the admitting surgeon remained unchanged, including attending presence at the highest-level trauma activations within 15 min of arrival (93% versus 86%, P= 0.07) and time to urgent operative intervention (68 min versus 82 min, P= 0.9). There were no differences in the number of laboratory and imaging studies (4.1 versus 4.1, P= 0.9) or bedside interventions (1.8 versus 2.1, P= 0.4) performed per patient by the admitting surgeon. Multivariate logistic regression did not identify multi-service coverage as an independent risk factor for adverse patient outcomes or quality metrics. CONCLUSIONS: Trauma admissions under a surgeon covering multiple services simultaneously had similar outcomes, quality metrics, and procedural utilization patterns compared to trauma admissions under surgeons covering only the trauma service. Despite concerns that multiple-service coverage may overburden one acute care surgeon, time-dependent quality metrics and studies done during the initial workup of trauma patients remained unchanged. These findings suggest that simultaneous trauma and EGS service coverage by one acute care surgeon does not adversely impact trauma patient care.

Distinguishing Admissions Specifically for COVID-19 From Incidental SARS-CoV-2 Admissions: National Retrospective Electronic Health Record Study.

BACKGROUND: Admissions are generally classified as COVID-19 hospitalizations if the patient has a positive SARS-CoV-2 polymerase chain reaction (PCR) test. However, because 35% of SARS-CoV-2 infections are asymptomatic, patients admitted for unrelated indications with an incidentally positive test could be misclassified as a COVID-19 hospitalization. Electronic health record (EHR)-based studies have been unable to distinguish between a hospitalization specifically for COVID-19 versus an incidental SARS-CoV-2 hospitalization. Although the need to improve classification of COVID-19 versus incidental SARS-CoV-2 is well understood, the magnitude of the problems has only been characterized in small, single-center studies. Furthermore, there have been no peer-reviewed studies evaluating methods for improving classification. OBJECTIVE: The aims of this study are to, first, quantify the frequency of incidental hospitalizations over the first 15 months of the pandemic in multiple hospital systems in the United States and, second, to apply electronic phenotyping techniques to automatically improve COVID-19 hospitalization classification. METHODS: From a retrospective EHR-based cohort in 4 US health care systems in Massachusetts, Pennsylvania, and Illinois, a random sample of 1123 SARS-CoV-2 PCR-positive patients hospitalized from March 2020 to August 2021 was manually chart-reviewed and classified as "admitted with COVID-19" (incidental) versus specifically admitted for COVID-19 ("for COVID-19"). EHR-based phenotyping was used to find feature sets to filter out incidental admissions. RESULTS: EHR-based phenotyped feature sets filtered out incidental admissions, which occurred in an average of 26% of hospitalizations (although this varied widely over time, from 0% to 75%). The top site-specific feature sets had 79%-99% specificity with 62%-75% sensitivity, while the best-performing across-site feature sets had 71%-94% specificity with 69%-81% sensitivity. CONCLUSIONS: A large proportion of SARS-CoV-2 PCR-positive admissions were incidental. Straightforward EHR-based phenotypes differentiated admissions, which is important to assure accurate public health reporting and research.

Evaluation of U.S. state opioid prescribing restrictions using patient opioid consumption patterns from a single, urban, academic institution.

Background: Since 2017, states, insurers, and pharmacies have placed blanket limits on the duration and quantity of opioid prescriptions. In many states, overlapping duration and daily dose limits yield maximum prescription limits of 150-350 morphine milligram equivalents (MMEs). There is limited knowledge of how these restrictions compare with actual patient opioid consumption; while changes in prescription patterns and opioid misuse rates have been studied, these are, at best, weak proxies for actual pain control consumption. We sought to determine how patients undergoing surgery would be affected by opioid prescribing restrictions using actual patient opioid consumption data. Methods: We constructed a prospective database of post-discharge opioid consumption: patients undergoing surgery at one institution were called after discharge to collect opioid consumption data. Patients whose opioid consumption exceeded 150 and 350 MME were identified. Results: Two thousand nine hundred and seventy-one patients undergoing 54 common surgical procedures were included in our study. Twenty-one percent of patients consumed more than the 150 MME limit. Only 7% of patients consumed above the 350 MME limit. Typical (non-outlier) opioid consumption, defined as less than the 75th percentile of consumption for any given procedure, exceeded the 150 MME and 350 MME limits for 41 and 7% of procedures, respectively. Orthopedic, spinal/neurosurgical, and complex abdominal procedures most commonly exceeded these limits. Conclusions: While most patients undergoing surgery are unaffected by recent blanket prescribing limits, those undergoing a specific subset of procedures are likely to require more opioids than the restrictions permit; providers should be aware that these patients may require a refill to adequately control post-surgical pain. Real consumption data should be used to guide these restrictions and inform future interventions so the risk of worsened pain control (and its troublesome effects on opioid misuse) is minimized. Procedure-specific prescribing limits may be one approach to prevent misuse, while also optimizing post-operative pain control.

Electronic Communication of Protected Health Information: Privacy, Security, and HIPAA Compliance.

PURPOSE: Technology has enhanced modern health care delivery, particularly through accessibility to health information and ease of communication with tools like mobile device messaging (texting). However, text messaging has created new risks for breach of protected health information (PHI). In the current study, we sought to evaluate hand surgeons' knowledge and compliance with privacy and security standards for electronic communication by text message. METHODS: A cross-sectional survey of the American Society for Surgery of the Hand membership was conducted in March and April 2016. Descriptive and inferential statistical analyses were performed of composite results as well as relevant subgroup analyses. RESULTS: A total of 409 responses were obtained (11% response rate). Although 63% of surgeons reported that they believe that text messaging does not meet Health Insurance Portability and Accountability Act of 1996 security standards, only 37% reported they do not use text messages to communicate PHI. Younger surgeons and respondents who believed that their texting was compliant were statistically significantly more like to report messaging of PHI (odds ratio, 1.59 and 1.22, respectively). DISCUSSION: A majority of hand surgeons in this study reported the use of text messaging to communicate PHI. Of note, neither the Health Insurance Portability and Accountability Act of 1996 statute nor US Department of Health and Human Services specifically prohibits this form of electronic communication. To be compliant, surgeons, practices, and institutions need to take reasonable security precautions to prevent breach of privacy with electronic communication. CLINICAL RELEVANCE: Communication of clinical information by text message is not prohibited under Health Insurance Portability and Accountability Act of 1996, but surgeons should use appropriate safeguards to prevent breach when using this form of communication.

An operational guide to translational clinical machine learning in academic medical centers.

Few published data science tools are ever translated from academia to real-world clinical settings for which they were intended. One dimension of this problem is the software engineering task of turning published academic projects into tools that are usable at the bedside. Given the complexity of the data ecosystem in large health systems, this task often represents a significant barrier to the real-world deployment of data science tools for prospective piloting and evaluation. Many information technology companies have created Machine Learning Operations (MLOps) teams to help with such tasks at scale, but the low penetration of home-grown data science tools in regular clinical practice precludes the formation of such teams in healthcare organizations. Based on experiences deploying data science tools at two large academic medical centers (Beth Israel Deaconess Medical Center, Boston, MA; Mayo Clinic, Rochester, MN), we propose a strategy to facilitate this transition from academic product to operational tool, defining the responsibilities of the principal investigator, data scientist, machine learning engineer, health system IT administrator, and clinician end-user throughout the process. We first enumerate the technical resources and stakeholders needed to prepare for model deployment. We then propose an approach to planning how the final product will work from data extraction and analysis to visualization of model outputs. Finally, we describe how the team should execute on this plan. We hope to guide health systems aiming to deploy minimum viable data science tools and realize their value in clinical practice.

Mobilizing data during a crisis: Building rapid evidence pipelines using multi-institutional real world data.

The COVID-19 pandemic generated tremendous interest in using real world data (RWD). Many consortia across the public and private sectors formed in 2020 with the goal of rapidly producing high-quality evidence from RWD to guide medical decision-making, public health priorities, and more. Experiences were gathered from five large consortia on rapid multi-institutional evidence generation during the COVID-19 pandemic. Insights have been compiled across five dimensions: consortium composition, governance structure and alignment of priorities, data sharing, data analysis, and evidence dissemination. The purpose of this piece is to offer guidance on building large-scale multi-institutional RWD analysis pipelines for future public health issues. The composition of each consortium was largely influenced by existing collaborations. A central set of priorities for evidence generation guided each consortium, however different approaches to governance emerged. Challenges surrounding limited access to clinical data due to various contributors were overcome in unique ways. While all consortia used different methods to construct and analyze patient cohorts ranging from centralized to federated approaches, all proved effective for generating meaningful real-world evidence. Actionable recommendations for clinical practice and public health agencies were made from translating insights from consortium analyses. Each consortium was successful in rapidly answering questions about COVID-19 diagnosis and treatment despite all taking slightly different approaches to data sharing and analysis. Leveraging RWD, leveraged in a manner that applies scientific rigor and transparency, can complement higher-level evidence and serve as an important adjunct to clinical trials to quickly guide policy and critical care, especially for a pandemic response.

Evaluating capabilities of large language models: Performance of GPT-4 on surgical knowledge assessments.

BACKGROUND: Artificial intelligence has the potential to dramatically alter health care by enhancing how we diagnose and treat disease. One promising artificial intelligence model is ChatGPT, a general-purpose large language model trained by OpenAI. ChatGPT has shown human-level performance on several professional and academic benchmarks. We sought to evaluate its performance on surgical knowledge questions and assess the stability of this performance on repeat queries. METHODS: We evaluated the performance of ChatGPT-4 on questions from the Surgical Council on Resident Education question bank and a second commonly used surgical knowledge assessment, referred to as Data-B. Questions were entered in 2 formats: open-ended and multiple-choice. ChatGPT outputs were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat queries. RESULTS: A total of 167 Surgical Council on Resident Education and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71.3% and 67.9% of multiple choice and 47.9% and 66.1% of open-ended questions for Surgical Council on Resident Education and Data-B, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained nonobvious insights. Common reasons for incorrect responses included inaccurate information in a complex question (n = 16, 36.4%), inaccurate information in a fact-based question (n = 11, 25.0%), and accurate information with circumstantial discrepancy (n = 6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of questions answered incorrectly on the first query; the response accuracy changed for 6/16 (37.5%) questions. CONCLUSION: Consistent with findings in other academic and professional domains, we demonstrate near or above human-level performance of ChatGPT on surgical knowledge questions from 2 widely used question banks. ChatGPT performed better on multiple-choice than open-ended questions, prompting questions regarding its potential for clinical application. Unique to this study, we demonstrate inconsistency in ChatGPT responses on repeat queries. This finding warrants future consideration including efforts at training large language models to provide the safe and consistent responses required for clinical application. Despite near or above human-level performance on question banks and given these observations, it is unclear whether large language models such as ChatGPT are able to safely assist clinicians in providing care.

Postoperative Opioid Prescribing via Rule-Based Guidelines Derived from In-Hospital Consumption: An Assessment of Efficacy Based on Postdischarge Opioid Use.

BACKGROUND: Many institutions have developed operation-specific guidelines for opioid prescribing. These guidelines rarely incorporate in-hospital opioid consumption, which is highly correlated with consumption. We compare outcomes of several patient-centered approaches to prescribing that are derived from in-hospital consumption, including several experimental, rule-based prescribing guidelines and our current institutional guideline. STUDY DESIGN: We performed a retrospective, cohort study of all adults undergoing surgery at a single-academic medical center. Several rule-based guidelines, derived from in-hospital consumption (quantity of opioids consumed within 24 hours of discharge), were used to specify the theoretical quantity of opioid prescribed on discharge. The efficacy of the experimental guidelines was compared with 3 references: an approximation of our institution's tailored prescribing guideline; prescribing all patients the typical quantity of opioids consumed for patients undergoing the same operation; and a representative rule-based, tiered framework. For each scenario, we calculated the penalized residual sum of squares (reflecting the composite deviation from actual patient consumption, with 15% penalty for overprescribing) and the proportion of opioids consumed relative to prescribed. RESULTS: A total of 1,048 patients met inclusion criteria. Mean (SD) and median (interquartile range [IQR]) quantity of opioids consumed within 24 hours of discharge were 11.2 (26.9) morphine milligram equivalents and 0 (0 to 15) morphine milligram equivalents. Median (IQR) postdischarge consumption was 16 (0 to 150) morphine milligram equivalents. Our institutional guideline and the previously validated rule-based guideline outperform alternate approaches, with median (IQR) differences in prescribed vs consumed opioids of 0 (-60 to 27.25) and 37.5 (-37.5 to 37.5), respectively, corresponding to penalized residual sum of squares of 39,817,602 and 38,336,895, respectively. CONCLUSIONS: Rather than relying on fixed quantities for defined operations, rule-based guidelines offer a simple yet effective method for tailoring opioid prescribing to in-hospital consumption.

Clinical prediction tool pitfalls and considerations: Data and algorithms.

In recent years, many surgical prediction models have been developed and published to augment surgeon decision-making, predict postoperative patient trajectories, and more. Collectively underlying all of these models is a wide variety of data sources and algorithms. Each data set and algorithm has its unique strengths, weaknesses, and type of prediction task for which it is best suited. The purpose of this piece is to highlight important characteristics of common data sources and algorithms used in surgical prediction model development so that future researchers interested in developing models of their own may be able to critically evaluate them and select the optimal ones for their study.

The role of digital technology in surgical home hospital programs.

Home hospital (HH), a care delivery model of providing hospital-grade care to patients in their homes, has become increasingly common in medical settings, though surgical uptake has been limited. HH programs have been shown to be safe and effective in a variety of medical contexts, with increased usage of this care pathway during the COVID-19 pandemic. Though surgical patients have unique clinical considerations, surgical Home Hospital (SHH) programs may have important benefits for this population. Various technologies exist for the delivery of hospital care in the home, such as clinical risk prediction models and remote patient monitoring platforms. Here, we use institutional experiences at Brigham and Women's Hospital (BWH) to discuss the utility of technology in enabling SHH programs and highlight current limitations. Additionally, we comment on the importance of data interoperability, access for all patients, and clinical workflow design in successfully implementing SHH programs.

The digital transformation of surgery.

Rapid advances in digital technology and artificial intelligence in recent years have already begun to transform many industries, and are beginning to make headway into healthcare. There is tremendous potential for new digital technologies to improve the care of surgical patients. In this piece, we highlight work being done to advance surgical care using machine learning, computer vision, wearable devices, remote patient monitoring, and virtual and augmented reality. We describe ways these technologies can be used to improve the practice of surgery, and discuss opportunities and challenges to their widespread adoption and use in operating rooms and at the bedside.

Open science practices in research published in surgical journals: A cross-sectional study.

Open science practices are research tools used to improve research quality and transparency. These practices have been used by researchers in various medical fields, though the usage of these practices in the surgical research ecosystem has not been quantified. In this work, we studied the use of open science practices in general surgery journals. Eight of the highest-ranked general surgery journals by SJR2 were chosen and their author guidelines were reviewed. From each journal, 30 articles published between January 1, 2019 and August 11, 2021 were randomly chosen and analyzed. Five open science practices were measured (preprint publication prior to peer-reviewed publication, use of Equator guidelines, study protocol preregistration prior to peer-reviewed publication, published peer review, and public accessibility of data, methods, and/or code). Across all 240 articles, 82 (34%) used one or more open science practices. Articles in the International Journal of Surgery showed greatest use of open science practices, with a mean of 1.6 open science practices compared to 0.36 across the other journals (p<.001). Adoption of open science practices in surgical research remains low, and further work is needed to increase utilization of these tools.

Trends in Medical Management of Moderately to Severely Active Ulcerative Colitis: A Nationwide Retrospective Analysis.

BACKGROUND: With an increasing number of therapeutic options available for the management of ulcerative colitis (UC), the variability in treatment and prescribing patterns is not well known. While recent guidelines have provided updates on how these therapeutic options should be used, patterns of long-term use of these drugs over the past 2 decades remain unclear. METHODS: We analyzed a retrospective, nationwide cohort of more than 1.7 million prescriptions for trends in prescribing behaviors and to evaluate practices suggested in guidelines relating to ordering biologics, step-up therapy, and combination therapy. The primary outcome was 30-day steroid-free remission and secondary outcomes included hospitalization, cost, and additional steroid usage. A pipeline was created to identify cohorts of patients under active UC medical management grouped by prescribing strategies to evaluate comparative outcomes between strategies. Cox proportional hazards and multivariate regression models were utilized to assess postexposure outcomes and adjust for confounders. RESULTS: Among 6 major drug categories, we noted major baseline differences in patient characteristics at first exposure corresponding to disease activity. We noted earlier use of biologics in patient trajectories (762 days earlier relative to UC diagnosis, 2018 vs 2008; P < .001) and greater overall use of biologics over time (2.53× more in 2018 vs 2008; P < .00001) . Among biologic-naive patients, adalimumab was associated with slightly lower rates of remission compared with infliximab or vedolizumab (odds ratio, 0.92; P < .005). Comparisons of patients with early biologic initiation to patients who transitioned to biologics from 5-aminosalicylic acid suggest lower steroid consumption for early biologic initiation (-761 mg prednisone; P < .001). Combination thiopurine-biologic therapy was associated with higher odds of remission compared with biologic monotherapy (odds ratio, 1.36; P = .01). CONCLUSIONS: As biologic drugs have become increasingly available for UC management, they have increasingly been used at earlier stages of disease management. Large-scale analyses of prescribing behaviors provide evidence supporting early use of biologics compared with step-up therapy and use of thiopurine and biologic combination therapy.

Population-scale analysis reveals patterns in prescribing trends for ulcerative colitis management. Findings include (1) earlier use of biologics in patient trajectories, (2) associations of step-up therapy with higher corticosteroid exposure, and (3) association of combination therapy with positive patient outcomes.

How surgeons use risk calculators and non-clinical factors for informed consent and shared decision making: A qualitative study.

BACKGROUND: The discussion of risks, benefits, and alternatives to surgery with patients is a defining component of informed consent. As shared-decision making has become central to surgeon-patient communication, risk calculators have emerged as a tool to aid communication and decision-making. To optimize informed consent, it is necessary to understand how surgeons assess and communicate risk, and the role of risk calculators in this process. METHODS: We conducted interviews with 13 surgeons from two institutions to understand how surgeons assess risk, the role of risk calculators in decision-making, and how surgeons approach risk communication during informed consent. We performed a qualitative analysis of interviews based on SRQR guidelines. RESULTS: Our analysis yielded insights regarding (a) the landscape and approach to obtaining surgical consent; (b) detailed perceptions regarding the value and design of assessing and communicating risk; and (c) practical considerations regarding the future of personalized risk communication in decision-making. Above all, we found that non-clinical factors such as health and risk literacy are changing how surgeons assess and communicate risk, which diverges from traditional risk calculators. CONCLUSION: Principally, we found that surgeons incorporate a range of clinical and non-clinical factors to risk stratify patients and determine how to optimally frame and discuss risk with individual patients. We observed that surgeons' perception of risk communication, and the importance of eliciting patient preferences to direct shared-decision making, did not consistently align with patient priorities. This study underscored criticisms of risk calculators and novel decision-aids - which must be addressed prior to greater adoption.

Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge Assessments.

Background: Artificial intelligence (AI) has the potential to dramatically alter healthcare by enhancing how we diagnosis and treat disease. One promising AI model is ChatGPT, a large general-purpose language model trained by OpenAI. The chat interface has shown robust, human-level performance on several professional and academic benchmarks. We sought to probe its performance and stability over time on surgical case questions. Methods: We evaluated the performance of ChatGPT-4 on two surgical knowledge assessments: the Surgical Council on Resident Education (SCORE) and a second commonly used knowledge assessment, referred to as Data-B. Questions were entered in two formats: open-ended and multiple choice. ChatGPT output were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat encounters. Results: A total of 167 SCORE and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71% and 68% of multiple-choice SCORE and Data-B questions, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained non-obvious insights. Common reasons for inaccurate responses included: inaccurate information in a complex question (n=16, 36.4%); inaccurate information in fact-based question (n=11, 25.0%); and accurate information with circumstantial discrepancy (n=6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of inaccurate questions; the response accuracy changed for 6/16 questions. Conclusion: Consistent with prior findings, we demonstrate robust near or above human-level performance of ChatGPT within the surgical domain. Unique to this study, we demonstrate a substantial inconsistency in ChatGPT responses with repeat query. This finding warrants future consideration and presents an opportunity to further train these models to provide safe and consistent responses. Without mental and/or conceptual models, it is unclear whether language models such as ChatGPT would be able to safely assist clinicians in providing care.

Comparing Rationale for Opioid Prescribing Decisions after Surgery with Subsequent Patient Consumption: A Survey of the Highest Quartile of Prescribers.

BACKGROUND: Opioid prescribing patterns, including those after surgery, have been implicated as a significant contributor to the US opioid crisis. A plethora of interventions-from nudges to reminders-have been deployed to improve prescribing behavior, but reasons for persistent outlier behavior are often unknown. STUDY DESIGN: Our institution employs multiple prescribing resources and a near real-time, feedback-based intervention to promote appropriate opioid prescribing. Since 2019, an automated system has emailed providers when a prescription exceeds the 75th percentile of typical opioid consumption for a given procedure-as defined by institutional data collection. Emails include population consumption metrics and an optional survey on rationale for prescribing. Responses were analyzed to understand why providers choose to prescribe atypically large discharge opioid prescriptions. We then compared provider prescriptions against patient consumption. RESULTS: During the study period, 10,672 eligible postsurgical patients were discharged; 2,013 prescriptions (29.4% of opioid prescriptions) exceeded our institutional guideline. Surveys were completed by outlier prescribers for 414 (20.6%) encounters. Among patients where both consumption data and prescribing rationale surveys were available, 35.2% did not consume any opioids after discharge and 21.5% consumed <50% of their prescription. Only 93 (39.9%) patients receiving outlier prescriptions were outlier consumers. Most common reasons for prescribing outlier amounts were attending preference (34%) and prescriber analysis of patient characteristics (34%). CONCLUSIONS: The top quartile of opioid prescriptions did not align with, and often far exceeded, patient postdischarge opioid consumption. Providers cite assessment of patient characteristics as a common driver of decision-making, but this did not align with patient usage for approximately 50% of patients.

Quantifying the Prognostic Value of Preoperative Surgeon Intuition: Comparing Surgeon Intuition and Clinical Risk Prediction as Derived from the American College of Surgeons NSQIP Risk Calculator.

BACKGROUND: Surgical risk prediction models traditionally use patient attributes and measures of physiology to generate predictions about postoperative outcomes. However, the surgeon's assessment of the patient may be a valuable predictor, given the surgeon's ability to detect and incorporate factors that existing models cannot capture. We compare the predictive utility of surgeon intuition and a risk calculator derived from the American College of Surgeons (ACS) NSQIP. STUDY DESIGN: From January 10, 2021 to January 9, 2022, surgeons were surveyed immediately before performing surgery to assess their perception of a patient's risk of developing any postoperative complication. Clinical data were abstracted from ACS NSQIP. Both sources of data were independently used to build models to predict the likelihood of a patient experiencing any 30-day postoperative complication as defined by ACS NSQIP. RESULTS: Preoperative surgeon assessment was obtained for 216 patients. NSQIP data were available for 9,182 patients who underwent general surgery (January 1, 2017 to January 9, 2022). A binomial regression model trained on clinical data alone had an area under the receiver operating characteristic curve (AUC) of 0.83 (95% CI 0.80 to 0.85) in predicting any complication. A model trained on only preoperative surgeon intuition had an AUC of 0.70 (95% CI 0.63 to 0.78). A model trained on surgeon intuition and a subset of clinical predictors had an AUC of 0.83 (95% CI 0.77 to 0.89). CONCLUSIONS: Preoperative surgeon intuition alone is an independent predictor of patient outcomes; however, a risk calculator derived from ACS NSQIP is a more robust predictor of postoperative complication. Combining intuition and clinical data did not strengthen prediction.

Deploying digital health tools within large, complex health systems: key considerations for adoption and implementation.

In recent years, the number of digital health tools with the potential to significantly improve delivery of healthcare services has grown tremendously. However, the use of these tools in large, complex health systems remains comparatively limited. The adoption and implementation of digital health tools at an enterprise level is a challenge; few strategies exist to help tools cross the chasm from clinical validation to integration within the workflows of a large health system. Many previously proposed frameworks for digital health implementation are difficult to operationalize in these dynamic organizations. In this piece, we put forth nine dimensions along which clinically validated digital health tools should be examined by health systems prior to adoption, and propose strategies for selecting digital health tools and planning for implementation in this setting. By evaluating prospective tools along these dimensions, health systems can evaluate which existing digital health solutions are worthy of adoption, ensure they have sufficient resources for deployment and long-term use, and devise a strategic plan for implementation.