Using a Large Language Model to Identify Adolescent Patient Portal Account Access by Guardians.

This diagnostic/prognostic study assesses the ability of a large language model (LLM) to detect guardian authorship of messages originating from adolescent patient portals.

Evaluation of a Large Language Model to Identify Confidential Content in Adolescent Encounter Notes.

This cross-sectional study assesses the ability of a language learning model to classify whether a progress note contains confidential information and to identify the specific confidential content in the note.

Structure and Funding of Clinical Informatics Fellowships: A National Survey of Program Directors.

BACKGROUND: In 2011, the American Board of Medical Specialties established clinical informatics (CI) as a subspecialty in medicine, jointly administered by the American Board of Pathology and the American Board of Preventive Medicine. Subsequently, many institutions created CI fellowship training programs to meet the growing need for informaticists. Although many programs share similar features, there is considerable variation in program funding and administrative structures. OBJECTIVES: The aim of our study was to characterize CI fellowship program features, including governance structures, funding sources, and expenses. METHODS: We created a cross-sectional online REDCap survey with 44 items requesting information on program administration, fellows, administrative support, funding sources, and expenses. We surveyed program directors of programs accredited by the Accreditation Council for Graduate Medical Education between 2014 and 2021. RESULTS: We invited 54 program directors, of which 41 (76%) completed the survey. The average administrative support received was $27,732/year. Most programs (85.4%) were accredited to have two or more fellows per year. Programs were administratively housed under six departments: Internal Medicine (17; 41.5%), Pediatrics (7; 17.1%), Pathology (6; 14.6%), Family Medicine (6; 14.6%), Emergency Medicine (4; 9.8%), and Anesthesiology (1; 2.4%). Funding sources for CI fellowship program directors included: hospital or health systems (28.3%), clinical departments (28.3%), graduate medical education office (13.2%), biomedical informatics department (9.4%), hospital information technology (9.4%), research and grants (7.5%), and other sources (3.8%) that included philanthropy and external entities. CONCLUSION: CI fellowships have been established in leading academic and community health care systems across the country. Due to their unique training requirements, these programs require significant resources for education, administration, and recruitment. There continues to be considerable heterogeneity in funding models between programs. Our survey findings reinforce the need for reformed federal funding models for informatics practice and training.

Using Clinical Decision Support Systems to Decrease Intravenous Acetaminophen Use: Implementation and Lessons Learned.

BACKGROUND: Clinical decision support systems (CDSS) can enhance medical decision-making by providing targeted information to providers. While they have the potential to improve quality of care and reduce costs, they are not universally effective and can lead to unintended harm. OBJECTIVES: To describe the implementation of an unsuccessful interruptive CDSS that aimed to promote appropriate use of intravenous (IV) acetaminophen at an academic pediatric hospital, with an emphasis on lessons learned. METHODS: Quality improvement methodology was used to study the effect of an interruptive CDSS, which set a mandatory expiry time of 24 hours for all IV acetaminophen orders. This CDSS was implemented on April 5, 2021. The primary outcome measure was number of IV acetaminophen administrations per 1,000 patient days, measured pre- and postimplementation. Process measures were the number of IV acetaminophen orders placed per 1,000 patient days. Balancing measures were collected via survey data and included provider and nursing acceptability and unintended consequences of the CDSS. RESULTS: There was no special cause variation in hospital-wide IV acetaminophen administrations and orders after CDSS implementation, nor when the CDSS was removed. A total of 88 participants completed the survey. Nearly half (40/88) of respondents reported negative issues with the CDSS, with the majority stating that this affected patient care (39/40). Respondents cited delays in patient care and reduced efficiency as the most common negative effects. CONCLUSION: This study underscores the significance of monitoring CDSS implementations and including end user acceptability as an outcome measure. Teams should be prepared to modify or remove CDSS that do not achieve their intended goal or are associated with low end user acceptability. CDSS holds promise for improving clinical practice, but careful implementation and ongoing evaluation are crucial for maximizing their benefits and minimizing potential harm.

Creation and Evaluation of a Clinical Informatics Match: Initial Findings.

BACKGROUND: Clinical Informatics (CI) fellowship programs utilize the Electronic Residency Application Service (ERAS) to gather applications but until recently used an American Medical Informatics Association (AMIA) member-developed, simultaneous offer-acceptance process to match fellowship applicants to programs. In 2021, program directors collaborated with the AMIA to develop a new match to improve the process. OBJECTIVE: Describe the results of the first 2 years of the match and address opportunities for improvement. METHODS: We obtained applicant data for fellowship applicants in 2021 and 2022 from the ERAS and match data for the same years from the AMIA. We analyzed our data using descriptive statistics. RESULTS: There were 159 unique applicants over the 2-year period. Applicants submitted 2,178 applications with a median of 10 per applicant (interquartile range [IQR] 3-20). One hundred and four applicants (65.4%) participated in the match and ranked a median of seven programs (2-12). Forty-two programs in 2021 and 47 programs in 2022 offered a combined total 153 positions in the match. Participating programs ranked a median of eight applicants per year (IQR 5-11). Of participating applicants, 95 (91.3%) successfully matched and of those 66 (69.5%) received their top choice. Thirty-two programs (76.2%) matched at least one candidate in 2021 and 33 programs (70.2%) matched at least one candidate in 2022. In both years, 24 programs filled all available slots (57.1% in 2021 and 51.1% in 2022). CONCLUSION: Applicants were extremely successful in the new match, which successfully addressed most of the challenges of the simultaneous offer-acceptance process identified by program directors. However, applicant attrition resulted in a quarter of programs going unmatched. Although many programs still filled slots outside the match, fellowship slots may remain unfilled while the CI practice pathway remains open.

The Prevalence of Confidential Content in Adolescent Progress Notes Prior to the 21st Century Cures Act Information Blocking Mandate.

BACKGROUND: The 21st Century Cures Act information blocking final rule mandated the immediate and electronic release of health care data in 2020. There is anecdotal concern that a significant amount of information is documented in notes that would breach adolescent confidentiality if released electronically to a guardian. OBJECTIVES: The purpose of this study was to quantify the prevalence of confidential information, based on California laws, within progress notes for adolescent patients that would be released electronically and assess differences in prevalence across patient demographics. METHODS: This is a single-center retrospective chart review of outpatient progress notes written between January 1, 2016, and December 31, 2019, at a large suburban academic pediatric network. Notes were labeled into one of three confidential domains by five expert reviewers trained on a rubric defining confidential information for adolescents derived from California state law. Participants included a random sampling of eligible patients aged 12 to 17 years old at the time of note creation. Secondary analysis included prevalence of confidentiality across age, gender, language spoken, and patient race. RESULTS: Of 1,200 manually reviewed notes, 255 notes (21.3%) (95% confidence interval: 19-24%) contained confidential information. There was a similar distribution among gender and age and a majority of English speaking (83.9%) and white or Caucasian patients (41.2%) in the cohort. Confidential information was more likely to be found in notes for females (p < 0.05) as well as for English-speaking patients (p < 0.05). Older patients had a higher probability of notes containing confidential information (p < 0.05). CONCLUSION: This study demonstrates that there is a significant risk to breach adolescent confidentiality if historical progress notes are released electronically to proxies without further review or redaction. With increased sharing of health care data, there is a need to protect the privacy of the adolescents and prevent potential breaches of confidentiality.

Association between Electronic Health Record Implementations and Hospital-Acquired Conditions in Pediatric Hospitals.

BACKGROUND: Implementing an electronic health record (EHR) is one of the most disruptive operational tasks a health system can undergo. Despite anecdotal reports of adverse events around the time of EHR implementations, there is limited corroborating research, particularly in pediatrics. We utilized data from Solutions for Patient Safety (SPS), a network of 145+ children's hospitals that share data and protocols to reduce harm in pediatric care delivery, to study the impact of EHR implementations on patient safety. OBJECTIVE: Determine if there is an association between the time immediately surrounding an EHR implementation and hospital-acquired conditions (HACs) rates in pediatrics. METHODS: A survey of information technology leaders at pediatric institutions identified EHR implementations occurring between 2012 and 2022. This list was cross-referenced with the SPS database to create an anonymized dataset of 27 sites comprising monthly HAC and care bundle compliance rates in the 7 months preceding and succeeding the transition. Six HACs were analyzed: central-line associated bloodstream infection (CLABSI), catheter-associated urinary tract infection (CAUTI), adverse drug events, surgical site infections (SSIs), pressure injuries (PIs), and falls, in addition to four associated care bundle compliance rates: CLABSI and CAUTI maintenance bundles, SSI bundle, and PI bundle. To determine if there was a statistically significant association with EHR implementation, the observation period was divided into three eras: "before" (months -7 to -3), "during" (months -2 to +2), and "after" go-live (months +3 to +7). Average monthly HAC and bundle compliance rates were calculated across eras. Paired t-tests were performed to compare rates between the eras. RESULTS: No statistically significant increase in HAC rates or decrease in bundle compliance rates was observed across the EHR implementation eras. CONCLUSION: This multisite study detected no significant increase in HACs and no decrease in preventive care bundle compliance in the months surrounding an EHR implementation.

A Natural Language Processing Model to Identify Confidential Content in Adolescent Clinical Notes.

BACKGROUND: The 21st Century Cures Act mandates the immediate, electronic release of health information to patients. However, in the case of adolescents, special consideration is required to ensure that confidentiality is maintained. The detection of confidential content in clinical notes may support operational efforts to preserve adolescent confidentiality while implementing information sharing. OBJECTIVES: This study aimed to determine if a natural language processing (NLP) algorithm can identify confidential content in adolescent clinical progress notes. METHODS: A total of 1,200 outpatient adolescent progress notes written between 2016 and 2019 were manually annotated to identify confidential content. Labeled sentences from this corpus were featurized and used to train a two-part logistic regression model, which provides both sentence-level and note-level probability estimates that a given text contains confidential content. This model was prospectively validated on a set of 240 progress notes written in May 2022. It was subsequently deployed in a pilot intervention to augment an ongoing operational effort to identify confidential content in progress notes. Note-level probability estimates were used to triage notes for review and sentence-level probability estimates were used to highlight high-risk portions of those notes to aid the manual reviewer. RESULTS: The prevalence of notes containing confidential content was 21% (255/1,200) and 22% (53/240) in the train/test and validation cohorts, respectively. The ensemble logistic regression model achieved an area under the receiver operating characteristic of 90 and 88% in the test and validation cohorts, respectively. Its use in a pilot intervention identified outlier documentation practices and demonstrated efficiency gains over completely manual note review. CONCLUSION: An NLP algorithm can identify confidential content in progress notes with high accuracy. Its human-in-the-loop deployment in clinical operations augmented an ongoing operational effort to identify confidential content in adolescent progress notes. These findings suggest NLP may be used to support efforts to preserve adolescent confidentiality in the wake of the information blocking mandate.

Providing Online Portal Access to Families of Adolescents and Young Adults with Diminished Capacity at an Academic Children's Hospital: A Case Report.

BACKGROUND: For caregivers of adolescents and young adults with severe cognitive deficits, or "diminished capacity," access to the medical record can be critical. However, this can be a challenge when utilizing the electronic health record (EHR) as information is often restricted in order to protect adolescent confidentiality. Having enhanced access for these proxies would be expected to improve engagement with the health system for the families of these medically complex adolescents and young adults. OBJECTIVES: To describe a process for granting full EHR access to proxies of adolescents with diminished capacity and young adults who are legally conserved while respecting regulations supporting adolescent confidentiality. METHODS: The first step in this initiative was to define the "diminished capacity" access class for both adolescents and young adults. Once defined, workflows utilizing best practice alerts were developed to support clinicians in providing the appropriate documentation. In addition, processes were developed to minimize the possibility of erroneously activating the diminished capacity access class for any given patient. To enhance activation, a support tool was developed to identify patients who might meet the criteria for diminished capacity proxy access. Finally, outreach and educations were developed for providers and clinics to make them aware of this initiative. RESULTS: Since activating this workflow, proxies of 138 adolescents and young adults have been granted the diminished capacity proxy access class. Approximately 54% are between 12 and 17 years with 46% 18 years and older. Proxies for both age groups have engaged with portal functionality at higher rates when compared to institutional rates of use by proxies of the general pediatric population. CONCLUSION: With this quality improvement initiative, we were able to enhance EHR access and engagement of families of some of the most complex adolescent and young adult patients without inadvertently compromising adolescent confidentiality.

Creating a Guardrail System to Ensure Appropriate Activation of Adolescent Portal Accounts.

The parent of an adolescent patient noticed an upcoming appointment in the patient's portal account that should have remained confidential to the parent. As it turned out, this parent was directly accessing their child's adolescent patient portal account instead of using a proxy account. After investigation of this case, it was found that the adolescent account had been activated with the parent's demographic (i.e., phone/email) information. This case illustrates the challenges of using adult-centric electronic health record (EHR) systems and how our institution addressed the problem of incorrect portal account activations.Confidentiality is fundamental to providing healthcare to adolescents. To comply with the 21st Century Cures Act's information blocking rules, confidential information must be released to adolescent patients when appropriate while also remaining confidential from their guardians. While complying with this national standard, systems of care must also account for interstate variability in which services allow for confidential adolescent consent. Unfortunately, there are high rates of guardian access to adolescent portal accounts which may lead to unintended disclosure of confidential information. Therefore, measures must be taken to minimize the risk of inadvertent confidentiality breaches via adolescent patient portals.Our institution implemented a guardrail system that checks the adolescent patient's contact information against the contact information of their parent/guardian/guarantor. This guardrail reduced the rate of account activation errors after implementation. However, the guardrail can be bypassed when demographic fields are missing. Thus, ongoing efforts to create pediatric-appropriate demographic fields, clearly distinguishing patient from proxy, in the EHR and workflows for registration of proxy accounts in the patient portal are needed.

Closing Disparities in Pediatric Diabetes Telehealth Care: Lessons From Telehealth Necessity During the COVID-19 Pandemic.

The coronavirus disease 2019 (COVID-19) pandemic necessitated using telehealth to bridge the clinical gap, but could increase health disparities. This article reports on a chart review of diabetes telehealth visits occurring before COVID-19, during shelter-in-place orders, and during the reopening period. Visits for children with public insurance and for those who were non-English speaking were identified. Telehealth visits for children with public insurance increased from 26.2% before COVID-19 to 37.3% during shelter-in-place orders and 34.3% during reopening. Telehealth visits for children who were non-English speaking increased from 3.5% before COVID-19 to 17.5% during shelter-in-place orders and remained at 15.0% during reopening. Pandemic-related telehealth expansion included optimization of workflows to include patients with public insurance and those who did not speak English. Increased participation by those groups persisted during the reopening phase, indicating that prioritizing inclusive telehealth workflows can reduce disparities in access to care.

Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.

OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.

Pediatric subspecialty telemedicine use from the patient and provider perspective.

BACKGROUND: To characterize telemedicine use among pediatric subspecialties with respect to clinical uses of telemedicine, provider experience, and patient perceptions during the COVID-19 pandemic. METHODS: We performed a mixed-methods study of telemedicine visits across pediatric endocrinology, nephrology, orthopedic surgery, and rheumatology at a large children's hospital. We used deductive analysis to review observational data from 40 video visits. Providers and patients/caregivers were surveyed around areas of satisfaction and communication. RESULTS: We found adaptations of telemedicine including shared-screen use and provider-guided parent procedures among others. All providers felt that it was safest for their patients to conduct visits by video, and 72.7% reported completing some component of a clinical exam. Patients rated the areas of being respected by the clinical staff/provider and showing care and concern highly, and the mean overall satisfaction was 86.7 ± 19.3%. CONCLUSIONS: Telemedicine has been used to deliver care to pediatric patients during the pandemic, and we found that patients were satisfied with the telemedicine visits during this stressful time and that providers were able to innovate during visits. Telemedicine is a tool that can be successfully adapted to patient and provider needs, but further studies are needed to fully explore its integration in pediatric subspecialty care. IMPACT: This study describes telemedicine use at the height of the COVID-19 pandemic from both a provider and patient perspective, in four different pediatric subspecialties. Prior to COVID-19, pediatric telehealth landscape analysis suggested that many pediatric specialty practices had pilot telehealth programs, but there are few published studies evaluating telemedicine performance through the simultaneous patient and provider experience as part of standard care. We describe novel uses and adaptations of telemedicine during a time of rapid deployment in pediatric specialty care.

Ensuring Adolescent Patient Portal Confidentiality in the Age of the Cures Act Final Rule.

PURPOSE: Managing confidential adolescent health information in patient portals presents unique challenges. Adolescent patients and guardians electronically access medical records and communicate with providers via portals. In confidential matters like sexual health, ensuring confidentiality is crucial. A key aspect of confidential portals is ensuring that the account is registered to and utilized by the intended user. Inappropriately registered or guardian-accessed adolescent portal accounts may lead to confidentiality breaches. METHODS: We used a quality improvement framework to develop screening methodologies to flag guardian-accessible accounts. Accounts of patients aged 12-17 were flagged via manual review of account emails and natural language processing of portal messages. We implemented a reconciliation program to correct affected accounts' registered email. Clinics were notified about sign-up errors and educated on sign-up workflow. An electronic alert was created to check the adolescent's email prior to account activation. RESULTS: After initial screening, 2,307 of 3,701 (62%) adolescent accounts were flagged as registered with a guardian's email. Those accounts were notified to resolve their logins. After five notifications over 8 weeks, 266 of 2,307 accounts (12%) were corrected; the remaining 2,041 (88%) were deactivated. CONCLUSIONS: The finding that 62% of adolescent portal accounts were used/accessed by guardians has significant confidentiality implications. In the context of the Cures Act Final Rule and increased information sharing, our institution's experience with ensuring appropriate access to adolescent portal accounts is necessary, timely, and relevant. This study highlights ways to improve patient portal confidentiality and prompts institutions caring for adolescents to review their systems and processes.

The Clinical Information Systems Response to the COVID-19 Pandemic.

OBJECTIVE: The year 2020 was predominated by the coronavirus disease 2019 (COVID-19) pandemic. The objective of this article is to review the areas in which clinical information systems (CIS) can be and have been utilized to support and enhance the response of healthcare systems to pandemics, focusing on COVID-19. METHODS: PubMed/MEDLINE, Google Scholar, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies pertaining to CIS, pandemics, and COVID-19 through October 2020. The most informative and detailed studies were highlighted, while many others were referenced. RESULTS: CIS were heavily relied upon by health systems and governmental agencies worldwide in response to COVID-19. Technology-based screening tools were developed to assist rapid case identification and appropriate triaging. Clinical care was supported by utilizing the electronic health record (EHR) to onboard frontline providers to new protocols, offer clinical decision support, and improve systems for diagnostic testing. Telehealth became the most rapidly adopted medical trend in recent history and an essential strategy for allowing safe and effective access to medical care. Artificial intelligence and machine learning algorithms were developed to enhance screening, diagnostic imaging, and predictive analytics - though evidence of improved outcomes remains limited. Geographic information systems and big data enabled real-time dashboards vital for epidemic monitoring, hospital preparedness strategies, and health policy decision making. Digital contact tracing systems were implemented to assist a labor-intensive task with the aim of curbing transmission. Large scale data sharing, effective health information exchange, and interoperability of EHRs remain challenges for the informatics community with immense clinical and academic potential. CIS must be used in combination with engaged stakeholders and operational change management in order to meaningfully improve patient outcomes. CONCLUSION: Managing a pandemic requires widespread, timely, and effective distribution of reliable information. In the past year, CIS and informaticists made prominent and influential contributions in the global response to the COVID-19 pandemic.

Assessment of Prevalence of Adolescent Patient Portal Account Access by Guardians.

Importance: Patient portals can be configured to allow confidential communication for adolescents' sensitive health care information. Guardian access of adolescent patient portal accounts could compromise adolescents' confidentiality. Objective: To estimate the prevalence of guardian access to adolescent patient portals at 3 academic children's hospitals. Design, Setting, and Participants: A cross-sectional study to estimate the prevalence of guardian access to adolescent patient portal accounts was conducted at 3 academic children's hospitals. Adolescent patients (aged 13-18 years) with access to their patient portal account with at least 1 outbound message from their portal during the study period were included. A rule-based natural language processing algorithm was used to analyze all portal messages from June 1, 2014, to February 28, 2020, and identify any message sent by guardians. The sensitivity and specificity of the algorithm at each institution was estimated through manual review of a stratified subsample of patient accounts. The overall proportion of accounts with guardian access was estimated after correcting for the sensitivity and specificity of the natural language processing algorithm. Exposures: Use of patient portal. Main Outcome and Measures: Percentage of adolescent portal accounts indicating guardian access. Results: A total of 3429 eligible adolescent accounts containing 25 642 messages across 3 institutions were analyzed. A total of 1797 adolescents (52%) were female and mean (SD) age was 15.6 (1.6) years. The percentage of adolescent portal accounts with apparent guardian access ranged from 52% to 57% across the 3 institutions. After correcting for the sensitivity and specificity of the algorithm based on manual review of 200 accounts per institution, an estimated 64% (95% CI, 59%-69%) to 76% (95% CI, 73%-88%) of accounts with outbound messages were accessed by guardians across the 3 institutions. Conclusions and Relevance: In this study, more than half of adolescent accounts with outbound messages were estimated to have been accessed by guardians at least once. These findings have implications for health systems intending to rely on separate adolescent accounts to protect adolescent confidentiality.

Quantifying Discharge Medication Reconciliation Errors at 2 Pediatric Hospitals.

INTRODUCTION: Medication reconciliation errors (MREs) are common and can lead to significant patient harm. Quality improvement efforts to identify and reduce these errors typically rely on resource-intensive chart reviews or adverse event reporting. Quantifying these errors hospital-wide is complicated and rarely done. The purpose of this study is to define a set of 6 MREs that can be easily identified across an entire healthcare organization and report their prevalence at 2 pediatric hospitals. METHODS: An algorithmic analysis of discharge medication lists and confirmation by clinician reviewers was used to find the prevalence of the 6 discharge MREs at 2 pediatric hospitals. These errors represent deviations from the standards for medication instruction completeness, clarity, and safety. The 6 error types are Duplication, Missing Route, Missing Dose, Missing Frequency, Unlisted Medication, and See Instructions errors. RESULTS: This study analyzed 67,339 discharge medications and detected MREs commonly at both hospitals. For Institution A, a total of 4,234 errors were identified, with 29.9% of discharges containing at least one error and an average of 0.7 errors per discharge. For Institution B, a total of 5,942 errors were identified, with 42.2% of discharges containing at least 1 error and an average of 1.6 errors per discharge. The most common error types were Duplication and See Instructions errors. CONCLUSION: The presented method shows these MREs to be a common finding in pediatric care. This work offers a tool to strengthen hospital-wide quality improvement efforts to reduce pediatric medication errors.