Developing and Validating a Prediction Model For Death or Critical Illness in Hospitalized Adults, an Opportunity for Human-Computer Collaboration.

Hospital early warning systems that use machine learning (ML) to predict clinical deterioration are increasingly being used to aid clinical decision-making. However, it is not known how ML predictions complement physician and nurse judgment. Our objective was to train and validate a ML model to predict patient deterioration and compare model predictions with real-world physician and nurse predictions. DESIGN: Retrospective and prospective cohort study. SETTING: Academic tertiary care hospital. PATIENTS: Adult general internal medicine hospitalizations. MEASUREMENTS AND MAIN RESULTS: We developed and validated a neural network model to predict in-hospital death and ICU admission in 23,528 hospitalizations between April 2011 and April 2019. We then compared model predictions with 3,374 prospectively collected predictions from nurses, residents, and attending physicians about their own patients in 960 hospitalizations between April 30, and August 28, 2019. ML model predictions achieved clinician-level accuracy for predicting ICU admission or death (ML median F1 score 0.32 [interquartile range (IQR) 0.30-0.34], AUC 0.77 [IQ 0.76-0.78]; clinicians median F1-score 0.33 [IQR 0.30-0.35], AUC 0.64 [IQR 0.63-0.66]). ML predictions were more accurate than clinicians for ICU admission. Of all ICU admissions and deaths, 36% occurred in hospitalizations where the model and clinicians disagreed. Combining human and model predictions detected 49% of clinical deterioration events, improving sensitivity by 16% compared with clinicians alone and 24% compared with the model alone while maintaining a positive predictive value of 33%, thus keeping false alarms at a clinically acceptable level. CONCLUSIONS: ML models can complement clinician judgment to predict clinical deterioration in hospital. These findings demonstrate important opportunities for human-computer collaboration to improve prognostication and personalized medicine in hospital.

Analyzing supply and demand on a general internal medicine ward: a cross-sectional study.

BACKGROUND: The capacity of general internal medicine (GIM) clinical teaching units has been strained by decreasing resident supply and increasing patient demand. The objective of our study was to compare the number of residents (supply) with the volume and duration of patient care activities (demand) to identify inefficiency. METHODS: Using the most recently available data from an academic teaching hospital in Toronto, Ontario, we identified each occurrence of a set of patient care activities that took place on the clinical teaching unit from 2015 to 2019. We completed a descriptive analysis of the frequencies of these activities and how the frequencies varied by hour, day, week, month and year. Patient care activities included admissions, rounds, responding to pages, meeting with patients and their families, patient transfers, discharges and responding to cardiac arrests. The estimated time to complete each task was based on the available data in our electronic medical record system and interviews with GIM physicians and trainees. To calculate resident utilization, the person-hours of patient care tasks was divided by the person-hours of resident supply. Resident utilization was computed for 3 scenarios corresponding to varying levels of resident absenteeism. RESULTS: During the study period, there were 14 581 consultations to GIM from the emergency department. Patient volumes tended to be highest during January and lowest during May and June, and highest on Monday morning and lowest on Friday night. Daily admissions to hospital from the emergency department were higher on weekdays than on weekends, and hourly admissions peaked at 8 am and between 3 pm and 1 am. Weekday resident utilization was generally highest between 8 am and 2 pm, and lowest between 1 am and 8 am. In a scenario in which all residents were present apart from those who were post-call, resident utilization generally never exceeded 100%; in scenarios in which at least 1 resident was absent owing to illness or vacation, it was common for resident utilization to approach or exceed 100%, particularly during daytime working hours. INTERPRETATION: Analyzing supply and demand on a GIM ward has allowed us to identify periods when supply and demand are not aligned and to demonstrate empirically the vulnerability of current staffing models. These data have the potential to inform and optimize scheduling on an internal medicine ward.

An electronic documentation system improves the quality of admission notes: a randomized trial.

OBJECTIVE: There are concerns that structured electronic documentation systems can limit expressivity and encourage long and unreadable notes. We assessed the impact of an electronic clinical documentation system on the quality of admission notes for patients admitted to a general medical unit. METHODS: This was a prospective randomized crossover study comparing handwritten paper notes to electronic notes on different patients by the same author, generated using a semistructured electronic admission documentation system over a 2-month period in 2014. The setting was a 4-team, 80-bed general internal medicine clinical teaching unit at a large urban academic hospital. The quality of clinical documentation was assessed using the QNOTE instrument (best possible score = 100), and word counts were assessed for free-text sections of notes. RESULTS: Twenty-one electronic-paper note pairs (42 notes) written by 21 authors were randomly drawn from a pool of 303 eligible notes. Overall note quality was significantly higher in electronic vs paper notes (mean 90 vs 69, P < .0001). The quality of free-text subsections (History of Present Illness and Impression and Plan) was significantly higher in the electronic vs paper notes (mean 93 vs 78, P < .0001; and 89 vs 77, P = .001, respectively). The History of Present Illness subsection was significantly longer in electronic vs paper notes (mean 172.4 vs 92.4 words, P = .0001). CONCLUSIONS: An electronic admission documentation system improved both the quality of free-text content and the overall quality of admission notes. Authors wrote more in the free-text sections of electronic documents as compared to paper versions.

Point-of-care ultrasound as a competency for general internists: a survey of internal medicine training programs in Canada.

BACKGROUND: Point-of-care ultrasound (POCUS) is increasingly used on General Internal Medicine (GIM) inpatient services, creating a need for defined competencies and formalized training. We evaluated the extent of training in POCUS and the clinical use of POCUS among Canadian GIM residency programs. METHOD: Internal Medicine trainees and GIM Faculty at the University of Toronto were surveyed on their clinical use of POCUS and the extent of their training. We separately surveyed Canadian IM Program Directors and Division Directors on the extent of POCUS training in their programs, barriers in the implementation of POCUS curricula, and recommendations for POCUS competencies in IM. RESULTS: A majority of IM trainees (90/118, 76%) and GIM Faculty (15/29, 52%) used POCUS clinically. However, the vast majority of resident (111/117, 95%) and GIM Faculty (18/28, 64%) had received limited training. Of the Program Leaders surveyed, half (9/17, 53%) reported POCUS clinical use by their trainees; however only one quarter (4/16, 25%) reported offering formal curricula. Most respondents agreed that POCUS training should be incorporated into IM residency curricula, specifically for procedural guidance. CONCLUSIONS: A considerable discrepancy exists between the clinical use of POCUS and the extent of formal training among Canadian IM residents and GIM Faculty. We propose that formalized POCUS training should be incorporated into IM residency programs, GIM fellowships, and Faculty development sessions, and identify POCUS skills that could be incorporated into future IM curricula.

Evaluation of standardized doctor's orders as an educational tool for undergraduate medical students: a prospective cohort study.

BACKGROUND: Standardized doctor's orders are replacing traditional order writing in teaching hospitals. The impact of this shift in practice on medical education is unknown. It is possible that preprinted orders interfere with knowledge acquisition and retention by not requiring active decision-making. The objective of the study was to evaluate the impact of standardized admission orders on disease-specific knowledge among undergraduate medical trainees. METHODS: This prospective cohort study enrolled Year 3 (n = 121) and Year 4 (n = 54) medical students at two academic hospitals in Toronto (Ontario, Canada) during their general internal medicine rotation. We used standardized orders for patient admissions for alcohol withdrawal (AW) and for acute exacerbations of chronic obstructive pulmonary disease (AECOPD) as the intervention and manual order writing as the control. Educational outcomes were assessed through end-of-rotation questionnaires assessing disease-specific knowledge of AW and AECOPD. RESULTS AND DISCUSSIONS: Of 175 students, 105 had exposure to patients with alcohol withdrawal during their rotation, and 68 students wrote admission orders. Among these 68 students, 48 used standardized orders (intervention, n = 48) and 20 used manual order writing (control, n = 20). Only 3 students used standardized orders for AECOPD, precluding analysis. There was no significant difference found in mean total score of questionnaires between those who used AW standardized orders and those who did not (11.8 vs. 11.0, p = 0.4). Students who had direct clinical experience had significantly higher mean total scores (11.6 vs. 9.0, p < 0.0001 for AW; 13.8 vs. 12.6, p = 0.02 for AECOPD) compared to students who did not. When corrected for overall knowledge, this difference only persisted for AW. CONCLUSIONS: No significant differences were found in total scores between students who used standardized admission orders and traditional manual order writing. Clinical exposure was associated with increase in disease-specific knowledge.

Electronic versus dictated hospital discharge summaries: a randomized controlled trial.

BACKGROUND: Patient care transitions are periods of enhanced risk. Discharge summaries have been used to communicate essential information between hospital-based physicians and primary care physicians (PCPs), and may reduce rates of adverse events after discharge. OBJECTIVE: To assess PCP satisfaction with an electronic discharge summary (EDS) program as compared to conventional dictated discharge summaries. DESIGN: Cluster randomized trial. PARTICIPANTS: Four medical teams of an academic general medical service. MEASUREMENTS: The primary endpoint was overall discharge summary quality, as assessed by PCPs using a 100-point visual analogue scale. Other endpoints included housestaff satisfaction (using a 100-point scale), adverse outcomes after discharge (combined endpoint of emergency department visits, readmission, and death), and patient understanding of discharge details as measured by the Care Transition Model (CTM-3) score (ranging from 0 to 100). RESULTS: 209 patient discharges were included over a 2-month period encompassing 1 housestaff rotation. Surveys were sent out for 188 of these patient discharges, and 119 were returned (63% response rate). No difference in PCP-reported overall quality was observed between the 2 methods (86.4 for EDS vs. 84.3 for dictation; P = 0.53). Housestaff found the EDS significantly easier to use than conventional dictation (86.5 for EDS vs. 49.2 for dictation; P = 0.03), but there was no difference in overall housestaff satisfaction. There was no difference between discharge methods for the combined endpoint for adverse outcomes (22 for EDS [21%] vs. 21 for dictation [20%]; P = 0.89), or for patient understanding of discharge details (CTM-3 score 80.3 for EDS vs. 81.3 for dictation; P = 0.81) CONCLUSION: An EDS program can be used by housestaff to more easily create hospital discharge summaries, and there was no difference in PCP satisfaction.

Does this patient with diabetes have osteomyelitis of the lower extremity?

CONTEXT: Osteomyelitis of the lower extremity is a commonly encountered problem in patients with diabetes and is an important cause of amputation and admission to the hospital. The diagnosis of lower limb osteomyelitis in patients with diabetes remains a challenge. OBJECTIVE: To determine the accuracy of historical features, physical examination, and laboratory and basic radiographic testing. We searched for systematic reviews of magnetic resonance imaging (MRI) in the diagnosis of lower extremity osteomyelitis in patients with diabetes to compare its performance with the reference standard. DATA SOURCES: MEDLINE search of English-language articles published between 1966 and March 2007 related to osteomyelitis in patients with diabetes. Additional articles were identified through a hand search of references from retrieved articles, previous reviews, and polling experts. STUDY SELECTION: Original studies were selected if they (1) described historical features, physical examination, laboratory investigations, or plain radiograph in the diagnosis of lower extremity osteomyelitis in patients with diabetes mellitus, (2) data could be extracted to construct 2 x 2 tables or had reported operating characteristics of the diagnostic measure, and (3) the diagnostic test was compared with a reference standard. Of 279 articles retrieved, 21 form the basis of this review. Data from a single high-quality meta-analysis were used to summarize the diagnostic characteristics of MRI in osteomyelitis. DATA EXTRACTION: Two authors independently assigned each study a quality grade using previously published criteria and abstracted operating characteristic data using a standardized instrument. DATA SYNTHESIS: The gold standard for diagnosis is bone biopsy. No studies were identified that addressed the utility of the history in the diagnosis of osteomyelitis. An ulcer area larger than 2 cm2 (positive likelihood ratio [LR], 7.2; 95% confidence interval [CI], 1.1-49; negative LR, 0.48; 95% CI, 0.31-0.76) and a positive "probe-to-bone" test result (summary positive LR, 6.4; 95% CI, 3.6-11; negative LR, 0.39; 95% CI, 0.20-0.76) were the best clinical findings. A erythrocyte sedimentation rate of more than 70 mm/h increases the probability of a diagnosis of osteomyelitis (summary LR, 11; 95% CI, 1.6-79). An abnormal plain radiograph doubles the odds of osteomyelitis (summary LR, 2.3; 95% CI, 1.6-3.3). A positive MRI result increases the likelihood of osteomyelitis (summary LR, 3.8; 95% CI, 2.5-5.8). However, a normal MRI result makes osteomyelitis much less likely (summary LR, 0.14; 95% CI, 0.08-0.26). The overall accuracy (ie, the weighted average of the sensitivity and specificity) of the MRI is 89% (95% CI, 83.0%-94.5%). CONCLUSIONS: An ulcer area larger than 2 cm2, a positive probe-to-bone test result, an erythrocyte sedimentation rate of more than 70 mm/h, and an abnormal plain radiograph result are helpful in diagnosing the presence of lower extremity osteomyelitis in patients with diabetes. A negative MRI result makes the diagnosis much less likely when all of these findings are absent. No single historical feature or physical examination reliably excludes osteomyelitis. The diagnostic utility of a combination of findings is unknown.

Clinical teaching and clinical outcomes: teaching capability and its association with patient outcomes.

BACKGROUND: There is little research on the impact of medical education on patient outcome. We studied whether teaching capability is associated with altered short-term patient outcomes. METHODS: We performed a multicentre retrospective cross-sectional study involving 40 clinician teachers who had attended on the general internal medicine services in hospitals affiliated with the University of Toronto along with the clinical outcomes of consecutive patients treated for community-acquired pneumonia, congestive heart failure, chronic obstructive pulmonary disease and gastrointestinal bleeding (n = 4377) between 1999 and 2001. Doctors were characterised by teaching effectiveness scores (n = 677) as high-rated or low-rated according to house staff ratings. RESULTS: There was no correlation between the teaching effectiveness scores and the mean length of stay for those patients treated for community-acquired pneumonia (high-rated = 10.3 versus low-rated = 8.1 days, P = 0.058), congestive heart failure (high-rated = 10.1 versus low-rated = 9.9 days, P = 0.978), chronic obstructive pulmonary disease (high-rated = 9.4 versus low-rated = 9.9 days, P = 0.419) and gastrointestinal bleeding (high-rated = 6.3 versus low-rated = 6.8 days, P = 0.741). In addition, we observed no significant correlation between teaching effectiveness scores and 7-day, 28-day and 1-year readmission rates for all pre-specified diagnoses. CONCLUSION: There is no large correlation between teaching effectiveness scores and short-term patient outcomes, suggesting that doctor teaching capabilities, as perceived by house staff, does not generally impact clinical care.

A comprehensive evidence-based approach to fever of unknown origin.

BACKGROUND: Fever of unknown origin (FUO) is defined as a temperature higher than 38.3 degrees C on several occasions and lasting longer than 3 weeks, with a diagnosis that remains uncertain after 1 week of investigation. METHODS: A systematic review was performed to develop evidence-based recommendations for the diagnostic workup of FUO. MEDLINE database was searched (January 1966 to December 2000) to identify articles related to FUO. Articles were included if the patient population met the criteria for FUO and they addressed the natural history, prognosis, or spectrum of disease or evaluated a diagnostic test in FUO. The quality of retrieved articles was rated as "good," "fair," or "poor," and sensitivity, specificity, and diagnostic yield of tests were calculated. Recommendations were made in accordance with the strength of evidence. RESULTS: The prevalence of FUO in hospitalized patients is reported to be 2.9%. Eleven studies indicate that the spectrum of disease includes "no diagnosis" (19%), infections (28%), inflammatory diseases (21%), and malignancies (17%). Deep vein thrombosis (3%) and temporal arteritis in the elderly (16%-17%) were important considerations. Four good natural history studies indicate that most patients with undiagnosed FUO recover spontaneously (51%-100%). One fair-quality study suggested a high specificity (99%) for the diagnosis of endocarditis in FUO by applying the Duke criteria. One fair-quality study showed that computed tomographic scanning of the abdomen had a diagnostic yield of 19%. Ten studies of nuclear imaging revealed that technetium was the most promising isotope, showing a high specificity (94%), albeit low sensitivity (40%-75%) (2 fair-quality studies). Two fair-quality studies showed liver biopsy to have a high diagnostic yield (14%-17%), but with risk of harm (0.009%-0.12% death). Empiric bone marrow cultures showed a low diagnostic yield of 0% to 2% (2 fair-quality articles). CONCLUSIONS: Diagnosis of FUO may be assisted by the Duke criteria for endocarditis, computed tomographic scan of the abdomen, nuclear scanning with a technetium-based isotope, and liver biopsy (fair to good evidence). Routine bone marrow cultures are not recommended.