Pediatric subspecialty telemedicine use from the patient and provider perspective.

BACKGROUND: To characterize telemedicine use among pediatric subspecialties with respect to clinical uses of telemedicine, provider experience, and patient perceptions during the COVID-19 pandemic. METHODS: We performed a mixed-methods study of telemedicine visits across pediatric endocrinology, nephrology, orthopedic surgery, and rheumatology at a large children's hospital. We used deductive analysis to review observational data from 40 video visits. Providers and patients/caregivers were surveyed around areas of satisfaction and communication. RESULTS: We found adaptations of telemedicine including shared-screen use and provider-guided parent procedures among others. All providers felt that it was safest for their patients to conduct visits by video, and 72.7% reported completing some component of a clinical exam. Patients rated the areas of being respected by the clinical staff/provider and showing care and concern highly, and the mean overall satisfaction was 86.7 ± 19.3%. CONCLUSIONS: Telemedicine has been used to deliver care to pediatric patients during the pandemic, and we found that patients were satisfied with the telemedicine visits during this stressful time and that providers were able to innovate during visits. Telemedicine is a tool that can be successfully adapted to patient and provider needs, but further studies are needed to fully explore its integration in pediatric subspecialty care. IMPACT: This study describes telemedicine use at the height of the COVID-19 pandemic from both a provider and patient perspective, in four different pediatric subspecialties. Prior to COVID-19, pediatric telehealth landscape analysis suggested that many pediatric specialty practices had pilot telehealth programs, but there are few published studies evaluating telemedicine performance through the simultaneous patient and provider experience as part of standard care. We describe novel uses and adaptations of telemedicine during a time of rapid deployment in pediatric specialty care.

Closing Disparities in Pediatric Diabetes Telehealth Care: Lessons From Telehealth Necessity During the COVID-19 Pandemic.

The coronavirus disease 2019 (COVID-19) pandemic necessitated using telehealth to bridge the clinical gap, but could increase health disparities. This article reports on a chart review of diabetes telehealth visits occurring before COVID-19, during shelter-in-place orders, and during the reopening period. Visits for children with public insurance and for those who were non-English speaking were identified. Telehealth visits for children with public insurance increased from 26.2% before COVID-19 to 37.3% during shelter-in-place orders and 34.3% during reopening. Telehealth visits for children who were non-English speaking increased from 3.5% before COVID-19 to 17.5% during shelter-in-place orders and remained at 15.0% during reopening. Pandemic-related telehealth expansion included optimization of workflows to include patients with public insurance and those who did not speak English. Increased participation by those groups persisted during the reopening phase, indicating that prioritizing inclusive telehealth workflows can reduce disparities in access to care.

Integration of Single-Center Data-Driven Vital Sign Parameters into a Modified Pediatric Early Warning System.

OBJECTIVES: Pediatric early warning systems using expert-derived vital sign parameters demonstrate limited sensitivity and specificity in identifying deterioration. We hypothesized that modified tools using data-driven vital sign parameters would improve the performance of a validated tool. DESIGN: Retrospective case control. SETTING: Quaternary-care children's hospital. PATIENTS: Hospitalized, noncritically ill patients less than 18 years old. Cases were defined as patients who experienced an emergent transfer to an ICU or out-of-ICU cardiac arrest. Controls were patients who never required intensive care. Cases and controls were split into training and testing groups. INTERVENTIONS: The Bedside Pediatric Early Warning System was modified by integrating data-driven heart rate and respiratory rate parameters (modified Bedside Pediatric Early Warning System 1 and 2). Modified Bedside Pediatric Early Warning System 1 used the 10th and 90th percentiles as normal parameters, whereas modified Bedside Pediatric Early Warning System 2 used fifth and 95th percentiles. MEASUREMENTS AND MAIN RESULTS: The training set consisted of 358 case events and 1,830 controls; the testing set had 331 case events and 1,215 controls. In the sensitivity analysis, 207 of the 331 testing set cases (62.5%) were predicted by the original tool versus 206 (62.2%; p = 0.54) with modified Bedside Pediatric Early Warning System 1 and 191 (57.7%; p < 0.001) with modified Bedside Pediatric Early Warning System 2. For specificity, 1,005 of the 1,215 testing set control patients (82.7%) were identified by original Bedside Pediatric Early Warning System versus 1,013 (83.1%; p = 0.54) with modified Bedside Pediatric Early Warning System 1 and 1,055 (86.8%; p < 0.001) with modified Bedside Pediatric Early Warning System 2. There was no net gain in sensitivity and specificity using either of the modified Bedside Pediatric Early Warning System tools. CONCLUSIONS: Integration of data-driven vital sign parameters into a validated pediatric early warning system did not significantly impact sensitivity or specificity, and all the tools showed lower than desired sensitivity and specificity at a single cutoff point. Future work is needed to develop an objective tool that can more accurately predict pediatric decompensation.

Improving Communication with Primary Care Physicians at the Time of Hospital Discharge.

BACKGROUND: Communication with primary care physicians (PCPs) at the time of a patient's hospital discharge is important to safely transition care to home. The goal of this quality improvement initiative was to increase discharge communication to PCPs at an academic children's hospital. METHODS: A multidisciplinary team at Lucile Packard Children's Hospital Stanford used Lean A3 problem solving methodology to address the problem of inadequate discharge communication with PCPs. Emphasis was placed on frontline provider (resident physicians) involvement in the improvement process, creating standards, and error proofing. Root cause analysis identified several key drivers of the problem, and successive countermeasures were implemented beginning in August 2013 aimed at achieving the target of 80% attempted verbal communication within seven days before or after (usually 24-48 hours) on the pediatric medical services. Run charts were generated tracking the outcome of PCP communication. RESULTS: On the pediatric medical services, the goal of 80% communication was met and sustained during a seven-month period starting October 2013, a statistically significant improvement. In the eight months prior to October 2013, hospitalwide PCP communication prior to discharge averaged 59.1% (n = 5,397) and improved to 76.7% (n = 4,870) in the seven months after (p <0.001). Fifteen of 19 specialty services had a significant increase in discharge communication after October 2013. CONCLUSION: Lean improvement methodology (including structured problem solving using A3 thinking), intensive frontline provider involvement, and process-oriented electronic health record work flow redesign led to increased verbal PCP communication at around the time of a patient's discharge.

Optimizing care of adults with congenital heart disease in a pediatric cardiovascular ICU using electronic clinical decision support*.

OBJECTIVES: The optimal location for postoperative cardiac care of adults with congenital heart disease is controversial. Some congenital heart surgeons operate on these adults in children's hospitals with postoperative care provided by pediatric critical care teams who may be unfamiliar with adult national performance measures. This study tested the hypothesis that Clinical Decision Support tools integrated into the clinical workflow would facilitate improved compliance with The Joint Commission Surgical Care Improvement Project performance measures in adults recovering from cardiac surgery in a children's hospital. DESIGN: Retrospective chart review comparing compliance pre- and post-Clinical Decision Support intervention for Surgical Care Improvement Project measures addressed in the critical care unit: appropriate cessation of prophylactic antibiotics; controlled blood glucose; urinary catheter removal; and reinitiation of preoperative ß-blocker when indicated. SETTING: Cardiovascular ICU in a quaternary care freestanding children's hospital. PATIENTS: The cohort included 114 adults 18-70 years old recovering from cardiac surgery in our pediatric cardiovascular ICU. INTERVENTIONS: Clinical Decision Support tools including data-triggered alerts, smart documentation forms, and order sets with conditional logic were integrated into the workflow. MEASUREMENTS AND MAIN RESULTS: Compliance with antibiotic discontinuation was 100% pre- and postintervention. Compliance rates improved for glucose control (p = 0.007) and urinary catheter removal (p = 0.05). Documentation of ß-blocker therapy (nonexistent preintervention) was 100% postintervention. Composite compliance for all measures increased from 53% to 84% (p = 0.002). There were no complications related to institution of the Surgical Care Improvement Project measures. There was no in-hospital mortality. CONCLUSIONS: Compliance with the national adult postoperative performance measures can be excellent in a children's hospital with the help of Clinical Decision Support tools. This represents an important step toward providing high-quality care to a growing population of adults with congenital heart disease who may receive care in a pediatric center.

Using Clinical Decision Support Systems to Decrease Intravenous Acetaminophen Use: Implementation and Lessons Learned.

BACKGROUND: Clinical decision support systems (CDSS) can enhance medical decision-making by providing targeted information to providers. While they have the potential to improve quality of care and reduce costs, they are not universally effective and can lead to unintended harm. OBJECTIVES: To describe the implementation of an unsuccessful interruptive CDSS that aimed to promote appropriate use of intravenous (IV) acetaminophen at an academic pediatric hospital, with an emphasis on lessons learned. METHODS: Quality improvement methodology was used to study the effect of an interruptive CDSS, which set a mandatory expiry time of 24 hours for all IV acetaminophen orders. This CDSS was implemented on April 5, 2021. The primary outcome measure was number of IV acetaminophen administrations per 1,000 patient days, measured pre- and postimplementation. Process measures were the number of IV acetaminophen orders placed per 1,000 patient days. Balancing measures were collected via survey data and included provider and nursing acceptability and unintended consequences of the CDSS. RESULTS: There was no special cause variation in hospital-wide IV acetaminophen administrations and orders after CDSS implementation, nor when the CDSS was removed. A total of 88 participants completed the survey. Nearly half (40/88) of respondents reported negative issues with the CDSS, with the majority stating that this affected patient care (39/40). Respondents cited delays in patient care and reduced efficiency as the most common negative effects. CONCLUSION: This study underscores the significance of monitoring CDSS implementations and including end user acceptability as an outcome measure. Teams should be prepared to modify or remove CDSS that do not achieve their intended goal or are associated with low end user acceptability. CDSS holds promise for improving clinical practice, but careful implementation and ongoing evaluation are crucial for maximizing their benefits and minimizing potential harm.

Structure and Funding of Clinical Informatics Fellowships: A National Survey of Program Directors.

BACKGROUND: In 2011, the American Board of Medical Specialties established clinical informatics (CI) as a subspecialty in medicine, jointly administered by the American Board of Pathology and the American Board of Preventive Medicine. Subsequently, many institutions created CI fellowship training programs to meet the growing need for informaticists. Although many programs share similar features, there is considerable variation in program funding and administrative structures. OBJECTIVES: The aim of our study was to characterize CI fellowship program features, including governance structures, funding sources, and expenses. METHODS: We created a cross-sectional online REDCap survey with 44 items requesting information on program administration, fellows, administrative support, funding sources, and expenses. We surveyed program directors of programs accredited by the Accreditation Council for Graduate Medical Education between 2014 and 2021. RESULTS: We invited 54 program directors, of which 41 (76%) completed the survey. The average administrative support received was $27,732/year. Most programs (85.4%) were accredited to have two or more fellows per year. Programs were administratively housed under six departments: Internal Medicine (17; 41.5%), Pediatrics (7; 17.1%), Pathology (6; 14.6%), Family Medicine (6; 14.6%), Emergency Medicine (4; 9.8%), and Anesthesiology (1; 2.4%). Funding sources for CI fellowship program directors included: hospital or health systems (28.3%), clinical departments (28.3%), graduate medical education office (13.2%), biomedical informatics department (9.4%), hospital information technology (9.4%), research and grants (7.5%), and other sources (3.8%) that included philanthropy and external entities. CONCLUSION: CI fellowships have been established in leading academic and community health care systems across the country. Due to their unique training requirements, these programs require significant resources for education, administration, and recruitment. There continues to be considerable heterogeneity in funding models between programs. Our survey findings reinforce the need for reformed federal funding models for informatics practice and training.

Embedding time-limited laboratory orders within computerized provider order entry reduces laboratory utilization.

OBJECTIVES: To test the hypothesis that limits on repeating laboratory studies within computerized provider order entry decrease laboratory utilization. DESIGN: Cohort study with historical controls. SETTING: A 20-bed PICU in a freestanding, quaternary care, academic children's hospital. PATIENTS: This study included all patients admitted to the pediatric ICU between January 1, 2008, and December 31, 2009. A total of 818 discharges were evaluated prior to the intervention (January 1, 2008, through December 31, 2008) and 1,021 patient discharges were evaluated postintervention (January 1, 2009, through December 31, 2009). INTERVENTION: A computerized provider order entry rule limited the ability to schedule repeating complete blood cell counts, chemistry, and coagulation studies to a 24-hour interval in the future. The time limit was designed to ensure daily evaluation of the utility of each test. MEASUREMENTS AND MAIN RESULTS: Initial analysis with t tests showed significant decreases in tests per patient day in the postintervention period (complete blood cell counts: 1.5 ± 0.1 to 1.0 ± 0.1; chemistry: 10.6 ± 0.9 to 6.9 ± 0.6; coagulation: 3.3 ± 0.4 to 1.7 ± 0.2; p < 0.01, all variables vs. preintervention period). Even after incorporating a trend toward decreasing laboratory utilization in the preintervention period into our regression analysis, the intervention decreased complete blood cell counts (p = 0.007), chemistry (p = 0.049), and coagulation (p = 0.001) tests per patient day. CONCLUSIONS: Limits on laboratory orders within the context of computerized provider order entry decreased laboratory utilization without adverse affects on mortality or length of stay. Broader application of this strategy might decrease costs, the incidence of iatrogenic anemia, and catheter-associated bloodstream infections.

Providing Online Portal Access to Families of Adolescents and Young Adults with Diminished Capacity at an Academic Children's Hospital: A Case Report.

BACKGROUND: For caregivers of adolescents and young adults with severe cognitive deficits, or "diminished capacity," access to the medical record can be critical. However, this can be a challenge when utilizing the electronic health record (EHR) as information is often restricted in order to protect adolescent confidentiality. Having enhanced access for these proxies would be expected to improve engagement with the health system for the families of these medically complex adolescents and young adults. OBJECTIVES: To describe a process for granting full EHR access to proxies of adolescents with diminished capacity and young adults who are legally conserved while respecting regulations supporting adolescent confidentiality. METHODS: The first step in this initiative was to define the "diminished capacity" access class for both adolescents and young adults. Once defined, workflows utilizing best practice alerts were developed to support clinicians in providing the appropriate documentation. In addition, processes were developed to minimize the possibility of erroneously activating the diminished capacity access class for any given patient. To enhance activation, a support tool was developed to identify patients who might meet the criteria for diminished capacity proxy access. Finally, outreach and educations were developed for providers and clinics to make them aware of this initiative. RESULTS: Since activating this workflow, proxies of 138 adolescents and young adults have been granted the diminished capacity proxy access class. Approximately 54% are between 12 and 17 years with 46% 18 years and older. Proxies for both age groups have engaged with portal functionality at higher rates when compared to institutional rates of use by proxies of the general pediatric population. CONCLUSION: With this quality improvement initiative, we were able to enhance EHR access and engagement of families of some of the most complex adolescent and young adult patients without inadvertently compromising adolescent confidentiality.

Creating a Guardrail System to Ensure Appropriate Activation of Adolescent Portal Accounts.

The parent of an adolescent patient noticed an upcoming appointment in the patient's portal account that should have remained confidential to the parent. As it turned out, this parent was directly accessing their child's adolescent patient portal account instead of using a proxy account. After investigation of this case, it was found that the adolescent account had been activated with the parent's demographic (i.e., phone/email) information. This case illustrates the challenges of using adult-centric electronic health record (EHR) systems and how our institution addressed the problem of incorrect portal account activations.Confidentiality is fundamental to providing healthcare to adolescents. To comply with the 21st Century Cures Act's information blocking rules, confidential information must be released to adolescent patients when appropriate while also remaining confidential from their guardians. While complying with this national standard, systems of care must also account for interstate variability in which services allow for confidential adolescent consent. Unfortunately, there are high rates of guardian access to adolescent portal accounts which may lead to unintended disclosure of confidential information. Therefore, measures must be taken to minimize the risk of inadvertent confidentiality breaches via adolescent patient portals.Our institution implemented a guardrail system that checks the adolescent patient's contact information against the contact information of their parent/guardian/guarantor. This guardrail reduced the rate of account activation errors after implementation. However, the guardrail can be bypassed when demographic fields are missing. Thus, ongoing efforts to create pediatric-appropriate demographic fields, clearly distinguishing patient from proxy, in the EHR and workflows for registration of proxy accounts in the patient portal are needed.

Association between Electronic Health Record Implementations and Hospital-Acquired Conditions in Pediatric Hospitals.

BACKGROUND: Implementing an electronic health record (EHR) is one of the most disruptive operational tasks a health system can undergo. Despite anecdotal reports of adverse events around the time of EHR implementations, there is limited corroborating research, particularly in pediatrics. We utilized data from Solutions for Patient Safety (SPS), a network of 145+ children's hospitals that share data and protocols to reduce harm in pediatric care delivery, to study the impact of EHR implementations on patient safety. OBJECTIVE: Determine if there is an association between the time immediately surrounding an EHR implementation and hospital-acquired conditions (HACs) rates in pediatrics. METHODS: A survey of information technology leaders at pediatric institutions identified EHR implementations occurring between 2012 and 2022. This list was cross-referenced with the SPS database to create an anonymized dataset of 27 sites comprising monthly HAC and care bundle compliance rates in the 7 months preceding and succeeding the transition. Six HACs were analyzed: central-line associated bloodstream infection (CLABSI), catheter-associated urinary tract infection (CAUTI), adverse drug events, surgical site infections (SSIs), pressure injuries (PIs), and falls, in addition to four associated care bundle compliance rates: CLABSI and CAUTI maintenance bundles, SSI bundle, and PI bundle. To determine if there was a statistically significant association with EHR implementation, the observation period was divided into three eras: "before" (months -7 to -3), "during" (months -2 to +2), and "after" go-live (months +3 to +7). Average monthly HAC and bundle compliance rates were calculated across eras. Paired t-tests were performed to compare rates between the eras. RESULTS: No statistically significant increase in HAC rates or decrease in bundle compliance rates was observed across the EHR implementation eras. CONCLUSION: This multisite study detected no significant increase in HACs and no decrease in preventive care bundle compliance in the months surrounding an EHR implementation.

A Natural Language Processing Model to Identify Confidential Content in Adolescent Clinical Notes.

BACKGROUND: The 21st Century Cures Act mandates the immediate, electronic release of health information to patients. However, in the case of adolescents, special consideration is required to ensure that confidentiality is maintained. The detection of confidential content in clinical notes may support operational efforts to preserve adolescent confidentiality while implementing information sharing. OBJECTIVES: This study aimed to determine if a natural language processing (NLP) algorithm can identify confidential content in adolescent clinical progress notes. METHODS: A total of 1,200 outpatient adolescent progress notes written between 2016 and 2019 were manually annotated to identify confidential content. Labeled sentences from this corpus were featurized and used to train a two-part logistic regression model, which provides both sentence-level and note-level probability estimates that a given text contains confidential content. This model was prospectively validated on a set of 240 progress notes written in May 2022. It was subsequently deployed in a pilot intervention to augment an ongoing operational effort to identify confidential content in progress notes. Note-level probability estimates were used to triage notes for review and sentence-level probability estimates were used to highlight high-risk portions of those notes to aid the manual reviewer. RESULTS: The prevalence of notes containing confidential content was 21% (255/1,200) and 22% (53/240) in the train/test and validation cohorts, respectively. The ensemble logistic regression model achieved an area under the receiver operating characteristic of 90 and 88% in the test and validation cohorts, respectively. Its use in a pilot intervention identified outlier documentation practices and demonstrated efficiency gains over completely manual note review. CONCLUSION: An NLP algorithm can identify confidential content in progress notes with high accuracy. Its human-in-the-loop deployment in clinical operations augmented an ongoing operational effort to identify confidential content in adolescent progress notes. These findings suggest NLP may be used to support efforts to preserve adolescent confidentiality in the wake of the information blocking mandate.

Creation and Evaluation of a Clinical Informatics Match: Initial Findings.

BACKGROUND: Clinical Informatics (CI) fellowship programs utilize the Electronic Residency Application Service (ERAS) to gather applications but until recently used an American Medical Informatics Association (AMIA) member-developed, simultaneous offer-acceptance process to match fellowship applicants to programs. In 2021, program directors collaborated with the AMIA to develop a new match to improve the process. OBJECTIVE: Describe the results of the first 2 years of the match and address opportunities for improvement. METHODS: We obtained applicant data for fellowship applicants in 2021 and 2022 from the ERAS and match data for the same years from the AMIA. We analyzed our data using descriptive statistics. RESULTS: There were 159 unique applicants over the 2-year period. Applicants submitted 2,178 applications with a median of 10 per applicant (interquartile range [IQR] 3-20). One hundred and four applicants (65.4%) participated in the match and ranked a median of seven programs (2-12). Forty-two programs in 2021 and 47 programs in 2022 offered a combined total 153 positions in the match. Participating programs ranked a median of eight applicants per year (IQR 5-11). Of participating applicants, 95 (91.3%) successfully matched and of those 66 (69.5%) received their top choice. Thirty-two programs (76.2%) matched at least one candidate in 2021 and 33 programs (70.2%) matched at least one candidate in 2022. In both years, 24 programs filled all available slots (57.1% in 2021 and 51.1% in 2022). CONCLUSION: Applicants were extremely successful in the new match, which successfully addressed most of the challenges of the simultaneous offer-acceptance process identified by program directors. However, applicant attrition resulted in a quarter of programs going unmatched. Although many programs still filled slots outside the match, fellowship slots may remain unfilled while the CI practice pathway remains open.

The Prevalence of Confidential Content in Adolescent Progress Notes Prior to the 21st Century Cures Act Information Blocking Mandate.

BACKGROUND: The 21st Century Cures Act information blocking final rule mandated the immediate and electronic release of health care data in 2020. There is anecdotal concern that a significant amount of information is documented in notes that would breach adolescent confidentiality if released electronically to a guardian. OBJECTIVES: The purpose of this study was to quantify the prevalence of confidential information, based on California laws, within progress notes for adolescent patients that would be released electronically and assess differences in prevalence across patient demographics. METHODS: This is a single-center retrospective chart review of outpatient progress notes written between January 1, 2016, and December 31, 2019, at a large suburban academic pediatric network. Notes were labeled into one of three confidential domains by five expert reviewers trained on a rubric defining confidential information for adolescents derived from California state law. Participants included a random sampling of eligible patients aged 12 to 17 years old at the time of note creation. Secondary analysis included prevalence of confidentiality across age, gender, language spoken, and patient race. RESULTS: Of 1,200 manually reviewed notes, 255 notes (21.3%) (95% confidence interval: 19-24%) contained confidential information. There was a similar distribution among gender and age and a majority of English speaking (83.9%) and white or Caucasian patients (41.2%) in the cohort. Confidential information was more likely to be found in notes for females (p < 0.05) as well as for English-speaking patients (p < 0.05). Older patients had a higher probability of notes containing confidential information (p < 0.05). CONCLUSION: This study demonstrates that there is a significant risk to breach adolescent confidentiality if historical progress notes are released electronically to proxies without further review or redaction. With increased sharing of health care data, there is a need to protect the privacy of the adolescents and prevent potential breaches of confidentiality.

Significant toxicity in a young female after low-dose tricyclic antidepressant ingestion.

BACKGROUND: Tricyclic antidepressant (TCA) ingestions are a relatively common pediatric ingestion, with significant potential for both cardiac and neurological toxicity. Previous studies on pediatric TCA ingestions have found the threshold of toxicity to be 5 mg/kg. CASE: We report a case of an 8-year-old girl who presented to the emergency department with depressed mental status and seizure-like movements. An extensive workup was pursued to evaluate the cause of her mental status, which only revealed a positive urine toxicology screen for TCA. Quantified serum levels of amitriptyline were 121 ng/mL (therapeutic range, 50-300 ng/mL) and nortriptyline were 79 ng/mL (therapeutic range 70-170 ng/mL), 18 hours after onset of symptoms. Subsequent history obtained after her mental status returned to normal revealed that she had ingested amitriptyline at a dose of 0.8 mg/kg. CONCLUSIONS: Tricyclic antidepressant ingestion has a high potential for toxicity in pediatric patients. This case suggests, contrary to previous literature, that toxicity may occur even with small doses.

Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.

OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.

The Value of OpenNotes for Pediatric Patients, Their Families and Impact on the Patient-Physician Relationship.

BACKGROUND: OpenNotes, the sharing of medical notes via a patient portal, has been extensively studied in adults but not in pediatric populations. This has been a contributing factor in the slower adoption of OpenNotes by children's hospitals. The 21st Century Cures Act Final Rule has mandated the sharing of clinical notes electronically to all patients and as health systems prepare to comply, some concerns remain particularly with OpenNotes for pediatric populations. OBJECTIVES: After a gradual implementation of OpenNotes at an academic pediatric center, we sought to better understand how pediatric patients and families perceived OpenNotes. This article presents the detailed steps of this informatics-led rollout and patient survey results with a focus on pediatric-specific concerns. METHODS: We adapted a previous OpenNotes survey used for adult populations to a pediatric outpatient setting (with parents of children <12 years old). The survey was sent to patients and families via a notification email sent as a standard practice after a clinic visit, in English or Spanish. RESULTS: Approximately 7% of patients/families with access to OpenNotes read the note during the study period, and 159 (20%) of those patients responded to the survey. Of the survey respondents, 141 (89%) of patients and families understood their notes; 126 (80%) found the notes always or usually accurate; 24 (15%) contacted their clinicians after reading a note; and 153 (97%) patients/families felt the same or better about their doctor after reading the note. CONCLUSION: Although limited by relatively low survey response rate, OpenNotes was well-received by parents of pediatric patients without untoward consequences. The main concerns pediatricians raise about OpenNotes proved to not be issues in the pediatric population. Our results demonstrate clear benefits to adoption of OpenNotes. This provides reassurance that the transition to sharing notes with pediatric patients can be successful and value additive.

Improved Medical Student Engagement with EHR Documentation following the 2018 Centers for Medicare and Medicaid Billing Changes.

BACKGROUND: Medical student note writing is an important part of the training process but has suffered in the electronic health record (EHR) era as a result of student notes being excluded from the billable encounter. The 2018 CMS billing changes allow for medical student notes to be used for billable services provided that physical presence requirements are met, and attending physicians satisfy performance requirements and verify documentation. This has the potential to improve medical student engagement and decrease physician documentation burden. METHODS: Our institution implemented medical student notes as part of the billable encounter in August 2018 with support of our compliance department. Note characteristics including number, type, length, and time in note were analyzed before and after implementation. Rotating medical students were surveyed regarding their experience following implementation. RESULTS: There was a statistically significant increase in the number of student-authored notes following implementation. Attending physicians' interactions with student notes greatly increased following the change (4% of student notes reviewed vs. 84% of student notes). Surveyed students reported that having their notes as part of the billable record made their notes more meaningful and enhanced their learning. The majority of surveyed students also agreed that they received more feedback following the change. CONCLUSION: Medical students are interested in writing notes for education and feedback. Inclusion of their notes as part of the billable record can facilitate their learning and increase their participation in the note writing process.

Quantifying Discharge Medication Reconciliation Errors at 2 Pediatric Hospitals.

INTRODUCTION: Medication reconciliation errors (MREs) are common and can lead to significant patient harm. Quality improvement efforts to identify and reduce these errors typically rely on resource-intensive chart reviews or adverse event reporting. Quantifying these errors hospital-wide is complicated and rarely done. The purpose of this study is to define a set of 6 MREs that can be easily identified across an entire healthcare organization and report their prevalence at 2 pediatric hospitals. METHODS: An algorithmic analysis of discharge medication lists and confirmation by clinician reviewers was used to find the prevalence of the 6 discharge MREs at 2 pediatric hospitals. These errors represent deviations from the standards for medication instruction completeness, clarity, and safety. The 6 error types are Duplication, Missing Route, Missing Dose, Missing Frequency, Unlisted Medication, and See Instructions errors. RESULTS: This study analyzed 67,339 discharge medications and detected MREs commonly at both hospitals. For Institution A, a total of 4,234 errors were identified, with 29.9% of discharges containing at least one error and an average of 0.7 errors per discharge. For Institution B, a total of 5,942 errors were identified, with 42.2% of discharges containing at least 1 error and an average of 1.6 errors per discharge. The most common error types were Duplication and See Instructions errors. CONCLUSION: The presented method shows these MREs to be a common finding in pediatric care. This work offers a tool to strengthen hospital-wide quality improvement efforts to reduce pediatric medication errors.

Measuring Electronic Health Record Use in the Pediatric ICU Using Audit-Logs and Screen Recordings.

BACKGROUND: Time spent in the electronic health record (EHR) has been identified as an important unit of measure for health care provider clinical activity. The lack of validation of audit-log based inpatient EHR time may have resulted in underuse of this data in studies focusing on inpatient patient outcomes, provider efficiency, provider satisfaction, etc. This has also led to a dearth of clinically relevant EHR usage metrics consistent with inpatient provider clinical activity. OBJECTIVE: The aim of our study was to validate audit-log based EHR times using observed EHR-times extracted from screen recordings of EHR usage in the inpatient setting. METHODS: This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children's Hospital Stanford between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study. Citrix session recording player was used to retrospectively review EHR session recordings that were captured as the provider interacted with the EHR. RESULTS: EHR use patterns varied by provider type. Audit-log based total EHR time correlated strongly with both observed total EHR time (r = 0.98, p < 0.001) and observed active EHR time (r = 0.95, p < 0.001). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87-1.02) minutes of observed total EHR time and 0.75 (0.67-0.83) minutes of observed active EHR time. Results were similar when stratified by provider role. CONCLUSION: Our study found inpatient audit-log based EHR time to correlate strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity and tracking outcomes of workflow improvement efforts longitudinally and across provider groups.