Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Feasibility of Implementation of a Mobile Digital Personal Health Record to Coordinate Care for Children and Youth With Special Health Care Needs in Primary Care: Protocol for a Mixed Methods Study.

BACKGROUND: Electronic health record (EHR)-integrated digital personal health records (PHRs) via Fast Healthcare Interoperability Resources (FHIR) are promising digital health tools to support care coordination (CC) for children and youth with special health care needs but remain widely unadopted; as their adoption grows, mixed methods and implementation research could guide real-world implementation and evaluation. OBJECTIVE: This study (1) evaluates the feasibility of an FHIR-enabled digital PHR app for CC for children and youth with special health care needs, (2) characterizes determinants of implementation, and (3) explores associations between adoption and patient- or family-reported outcomes. METHODS: This nonrandomized, single-arm, prospective feasibility trial will test an FHIR-enabled digital PHR app's use among families of children and youth with special health care needs in primary care settings. Key app features are FHIR-enabled access to structured data from the child's medical record, families' abilities to longitudinally track patient- or family-centered care goals, and sharing progress toward care goals with the child's primary care provider via a clinician dashboard. We shall enroll 40 parents or caregivers of children and youth with special health care needs to use the app for 6 months. Inclusion criteria for children and youth with special health care needs are age 0-16 years; primary care at a participating site; complex needs benefiting from CC; high hospitalization risk in the next 6 months; English speaking; having requisite technology at home (internet access, Apple iOS mobile device); and an active web-based EHR patient portal account to which a parent or caregiver has full proxy access. Digital prescriptions will be used to disseminate study recruitment materials directly to eligible participants via their existing EHR patient portal accounts. We will apply an intervention mixed methods design to link quantitative and qualitative (semistructured interviews and family engagement panels with parents of children and youth with special health care needs) data and characterize implementation determinants. Two CC frameworks (Pediatric Care Coordination Framework; Patient-Centered Medical Home) and 2 evaluation frameworks (Consolidated Framework for Implementation Research; Technology Acceptance Model) provide theoretical foundations for this study. RESULTS: Participant recruitment began in fall 2022, before which we identified >300 potentially eligible patients in EHR data. A family engagement panel in fall 2021 generated formative feedback from family partners. Integrated analysis of pretrial quantitative and qualitative data informed family-centered enhancements to study procedures. CONCLUSIONS: Our findings will inform how to integrate an FHIR-enabled digital PHR app for children and youth with special health care needs into clinical care. Mixed methods and implementation research will help strengthen implementation in diverse clinical settings. The study is positioned to advance knowledge of how to use digital health innovations for improving care and outcomes for children and youth with special health care needs and their families. TRIAL REGISTRATION: ClinicalTrials.gov NCT05513235; https://clinicaltrials.gov/study/NCT05513235. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/46847.

Predictive Modeling to Identify Children With Complex Health Needs At Risk for Hospitalization.

BACKGROUND: Identifying children at high risk with complex health needs (CCHN) who have intersecting medical and social needs is challenging. This study's objectives were to (1) develop and evaluate an electronic health record (EHR)-based clinical predictive model ("model") for identifying high-risk CCHN and (2) compare the model's performance as a clinical decision support (CDS) to other CDS tools available for identifying high-risk CCHN. METHODS: This retrospective cohort study included children aged 0 to 20 years with established care within a single health system. The model development/validation cohort included 33 months (January 1, 2016-September 30, 2018) and the testing cohort included 18 months (October 1, 2018-March 31, 2020) of EHR data. Machine learning methods generated a model that predicted probability (0%-100%) for hospitalization within 6 months. Model performance measures included sensitivity, positive predictive value, area under receiver-operator curve, and area under precision-recall curve. Three CDS rules for identifying high-risk CCHN were compared: (1) hospitalization probability ≥10% (model-predicted); (2) complex chronic disease classification (using Pediatric Medical Complexity Algorithm [PMCA]); and (3) previous high hospital utilization. RESULTS: Model development and testing cohorts included 116 799 and 27 087 patients, respectively. The model demonstrated area under receiver-operator curve = 0.79 and area under precision-recall curve = 0.13. PMCA had the highest sensitivity (52.4%) and classified the most children as high risk (17.3%). Positive predictive value of the model-based CDS rule (19%) was higher than CDS based on the PMCA (1.9%) and previous hospital utilization (15%). CONCLUSIONS: A novel EHR-based predictive model was developed and validated as a population-level CDS tool for identifying CCHN at high risk for future hospitalization.

Healthcare Utilization for Medicaid-Insured Children with Medical Complexity: Differences by Sociodemographic Characteristics.

OBJECTIVE: To compare differences in healthcare utilization and costs for Medicaid-insured children with medical complexity (CMC) by race/ethnicity and rurality. METHODS: Retrospective cohort of North Carolina (NC) Medicaid claims for children 3-20 years old with 3 years continuous Medicaid coverage (10/1/2015-9/30/2018). Exposures were medical complexity, race/ethnicity, and rurality. Three medical complexity levels were: without chronic disease, non-complex chronic disease, and complex chronic disease; the latter were defined as CMC. Race/ethnicity was self-reported in claims; we defined rurality by home residence ZIP codes. Utilization and costs were summarized for 1 year (10/1/2018-9/30/2019) by complexity level classification and categorized as acute care (hospitalization, emergency [ED]), outpatient care (primary, specialty, allied health), and pharmacy. Per-complexity group utilization rates (per 1000 person-years) by race/ethnicity and rurality were compared using adjusted rate ratios (ARR). RESULTS: Among 859,166 Medicaid-insured children, 118,210 (13.8%) were CMC. Among CMC, 36% were categorized as Black non-Hispanic, 42.7% White non-Hispanic, 14.3% Hispanic, and 35% rural. Compared to White non-Hispanic CMC, Black non-Hispanic CMC had higher hospitalization (ARR = 1.12; confidence interval, CI 1.08-1.17) and ED visit (ARR = 1.17; CI 1.16-1.19) rates; Hispanic CMC had lower ED visit (ARR = 0.77; CI 0.75-0.78) and hospitalization rates (ARR = 0.79; CI 0.73-0.84). Black non-Hispanic and Hispanic CMC had lower outpatient visit rates than White non-Hispanic CMC. Rural CMC had higher ED (ARR = 1.13; CI 1.11-1.15) and lower primary care utilization rates (ARR = 0.87; CI 0.86-0.88) than urban CMC. DISCUSSION: Healthcare utilization varied by race/ethnicity and rurality for Medicaid-insured CMC. Further studies should investigate mechanisms for these variations and expand higher value, equitable care delivery for CMC.

A Tailored SMS Text Message-Based Intervention to Facilitate Patient Access to Referred Community-Based Social Needs Resources: Protocol for a Pilot Feasibility and Acceptability Study.

BACKGROUND: Health care providers are increasingly screening patients for unmet social needs (eg, food, housing, transportation, and social isolation) and referring patients to relevant community-based resources and social services. Patients' connection to referred services is often low, however, suggesting the need for additional support to facilitate engagement with resources. SMS text messaging presents an opportunity to address barriers related to contacting resources in an accessible, scalable, and low-cost manner. OBJECTIVE: In this multi-methods pilot study, we aim to develop an automated SMS text message-based intervention to promote patient connection to referred social needs resources within 2 weeks of the initial referral and to evaluate its feasibility and patient acceptability. This protocol describes the intervention, conceptual underpinnings, study design, and evaluation plan to provide a detailed illustration of how SMS technology can complement current social needs screening and referral practice patterns without disrupting care. METHODS: For this pilot prospective cohort study, this SMS text message-based intervention augments an existing social needs screening, referral, and navigation program at a federally qualified health center. Patients who received at least one referral for any identified unmet social need are sent 2 rounds of SMS messages over 2 weeks. The first round consists of 5-10 messages that deliver descriptions of and contact information for the referred resources. The second round consists of 2 messages that offer a brief reminder to contact the resources. Participants will evaluate the intervention via a survey and a semistructured interview, informed by an adapted technology acceptance model. Rapid qualitative and thematic analysis will be used to extract themes from the responses. Primary outcomes are implementation feasibility and patient acceptability. Secondary outcomes relate to intervention effectiveness: self-reported attempt to connect and successful connection to referred resources 2 weeks after the initial referral encounter. RESULTS: The study received regulatory approval in May 2021, and we anticipate enrolling 15-20 participants for this initial pilot. CONCLUSIONS: This protocol presents detailed implementation methods about a novel automated SMS intervention for social care integration within primary care. By sharing the study protocol early, we intend to facilitate the development and adoption of similar tools across different clinical settings, as more health care providers seek to address the unmet social needs of patients. Study findings will provide practical insights into the design and implementation of SMS text message-based interventions to improve social and medical care coordination. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/37316.

Identifying Priorities to Improve the System of Care for Children With Complex Health Needs in North Carolina: Process and Outcomes of Systematic Stakeholder Engagement.

OBJECTIVE: Children with complex health needs (CCHN) have both medical (eg, chronic conditions) and health-related social needs (eg, potentially adverse social determinants of health) that require ongoing health care and support from multiple community service providers. National standards developed for populations defined by health needs (CYSHCN) provide a framework for stakeholders to plan system-level improvements in care delivery for CCHN, but improvement efforts should reflect the priorities of their families and providers. This article describes a process of prioritizing system-level efforts to improve the health and well-being of CCHN and families in North Carolina (NC), using systematic stakeholder engagement and modified Delphi expert ratings. METHODS: We surveyed stakeholders with experience caring for CCHN using an open-ended, 3-item instrument to identify opportunities to improve systems of care. Using directed qualitative content analysis, we synthesized responses into a master list of potential improvement topics. Using a modified Delphi approach, a 16-member advisory committee rated all topics for importance and urgency, on 9-point Likert scales over 2 rounds; then ratings for each topic were ranked (low, medium, high) to establish relative priority. RESULTS: Forty seven individuals from 31 counties around NC provided survey responses, yielding 59 improvement topics in 10 domains. Through the modified Delphi method process, 21 topics (36%) received the highest rankings, largely representing access to community- and home-based services, equity, and enhancement of the pediatric workforce. CONCLUSIONS: Priorities identified by stakeholders will inform advocacy, policy, and improvement efforts. Next steps for the coalition include developing improvement projects to implement stakeholder-recommended actions for the highest-priority topics.

Feasibility of implementing systematic social needs assessment for children with medical complexity.

BACKGROUND: Children with medical complexity (CMC) have inter-related health and social needs; however, interventions to identify and respond to social needs have not been adapted for CMC. The objective of this study was to evaluate the feasibility of implementing social needs screening and assessment within pediatric complex care programs. METHODS: We implemented systematic social needs assessment for CMC (SSNAC) at two tertiary care centers in three phases: (1) pre-implementation, (2) implementation, and (3) implementation monitoring. We utilized a multifaceted implementation package consisting of discrete implementation strategies within each phase. In phase 1, we adapted questions from evidence-informed screening tools into a 21-item SSNAC questionnaire, and we used published frameworks to inform implementation readiness and process. In phases 2-3, clinical staff deployed the SSNAC questionnaire to parents of CMC in-person or by phone as part of usual care and adapted to local clinical workflows. Staff used shared decision-making with parents and addressed identified needs by providing information about available resources, offering direct assistance, and making referrals to community agencies. Implementation outcomes included fidelity, feasibility, acceptability, and appropriateness. RESULTS: Observations from clinical staff characterized fidelity to use of the SSNAC questionnaire, assessment template, and shared decision-making for follow-up on unmet social needs. Levels of agreement (5-point Likert scale; 1 = completely disagree; 5 = completely agree) rated by staff for key implementation outcomes were moderate to high for acceptability (mean = 4.7; range = 3-5), feasibility (mean = 4.2; range = 3-5), and appropriateness (mean = 4.6; range = 4-5). 49 SSNAC questionnaires were completed with a 91% response rate. Among participating parents, 37 (76%) reported ≥ 1 social need, including food/nutrition benefits (41%), housing (18%), and caregiver needs (29%). Staff responses included information provision (41%), direct assistance (30%), and agency referral (30%). CONCLUSIONS: It was feasible for tertiary care center-based pediatric complex care programs to implement a standardized social needs assessment for CMC to identify and address parent-reported unmet social needs.