Investigation of bias in the automated assessment of school violence.

OBJECTIVES: Natural language processing and machine learning have the potential to lead to biased predictions. We designed a novel Automated RIsk Assessment (ARIA) machine learning algorithm that assesses risk of violence and aggression in adolescents using natural language processing of transcribed student interviews. This work evaluated the possible sources of bias in the study design and the algorithm, tested how much of a prediction was explained by demographic covariates, and investigated the misclassifications based on demographic variables. METHODS: We recruited students 10-18 years of age and enrolled in middle or high schools in Ohio, Kentucky, Indiana, and Tennessee. The reference standard outcome was determined by a forensic psychiatrist as either a "high" or "low" risk level. ARIA used L2-regularized logistic regression to predict a risk level for each student using contextual and semantic features. We conducted three analyses: a PROBAST analysis of risk in study design; analysis of demographic variables as covariates; and a prediction analysis. Covariates were included in the linear regression analyses and comprised of race, sex, ethnicity, household education, annual household income, age at the time of visit, and utilization of public assistance. RESULTS: We recruited 412 students from 204 schools. ARIA performed with an AUC of 0.92, sensitivity of 71%, NPV of 77%, and specificity of 95%. Of these, 387 students with complete demographic information were included in the analysis. Individual linear regressions resulted in a coefficient of determination less than 0.08 across all demographic variables. When using all demographic variables to predict ARIA's risk assessment score, the multiple linear regression model resulted in a coefficient of determination of 0.189. ARIA performed with a lower False Negative Rate (FNR) of 15.2% (CI [0 - 40]) for the Black subgroup and 12.7%, CI [0 - 41.4] for Other races, compared to an FNR of 26.1% (CI [14.1 - 41.8]) in the White subgroup. CONCLUSIONS: Bias assessment is needed to address shortcomings within machine learning. In our work, student race, ethnicity, sex, use of public assistance, and annual household income did not explain ARIA's risk assessment score of students. ARIA will continue to be evaluated regularly with increased subject recruitment.

Machine Learning-Based Prediction of Escitalopram and Sertraline Side Effects With Pharmacokinetic Data in Children and Adolescents.

Selective serotonin reuptake inhibitors (SSRI) are the first-line pharmacologic treatment for anxiety and depressive disorders in children and adolescents. Many patients experience side effects that are difficult to predict, are associated with significant morbidity, and can lead to treatment discontinuation. Variation in SSRI pharmacokinetics could explain differences in treatment outcomes, but this is often overlooked as a contributing factor to SSRI tolerability. This study evaluated data from 288 escitalopram-treated and 255 sertraline-treated patients ≤ 18 years old to develop machine learning models to predict side effects using electronic health record data and Bayesian estimated pharmacokinetic parameters. Trained on a combined cohort of escitalopram- and sertraline-treated patients, a penalized logistic regression model achieved an area under the receiver operating characteristic curve (AUROC) of 0.77 (95% confidence interval (CI): 0.66-0.88), with 0.69 sensitivity (95% CI: 0.54-0.86), and 0.82 specificity (95% CI: 0.72-0.87). Medication exposure, clearance, and time since the last dose increase were among the top features. Individual escitalopram and sertraline models yielded an AUROC of 0.73 (95% CI: 0.65-0.81) and 0.64 (95% CI: 0.55-0.73), respectively. Post hoc analysis showed sertraline-treated patients with activation side effects had slower clearance (P = 0.01), which attenuated after accounting for age (P = 0.055). These findings raise the possibility that a machine learning approach leveraging pharmacokinetic data can predict escitalopram- and sertraline-related side effects. Clinicians may consider differences in medication pharmacokinetics, especially during dose titration and as opposed to relying on dose, when managing side effects. With further validation, application of this model to predict side effects may enhance SSRI precision dosing strategies in youth.

Early Identification of Candidates for Epilepsy Surgery: A Multicenter, Machine Learning, Prospective Validation Study.

BACKGROUND AND OBJECTIVES: Epilepsy surgery is often delayed. We previously developed machine learning (ML) models to identify candidates for resective epilepsy surgery earlier in the disease course. In this study, we report the prospective validation. METHODS: In this multicenter, prospective, longitudinal cohort study, random forest models were validated at a pediatric epilepsy center consisting of 2 hospitals and 14 outpatient neurology clinic sites and an adult epilepsy center with 2 hospitals and 27 outpatient neurology clinic sites. The models used neurology visit notes, EEG and MRI reports, visit patterns, hospitalizations, and medication, laboratory, and procedure orders to identify candidates for surgery. The models were trained on historical data up to May 10, 2019. Patients with an ICD-10 diagnosis of epilepsy who visited from May 11, 2019, to May 10, 2020, were screened by the algorithm and assigned surgical candidacy scores. The primary outcome was area under the curve (AUC), which was calculated by comparing scores from patients who underwent epilepsy surgery before November 10, 2020, against scores from nonsurgical patients. Nonsurgical patients' charts were reviewed to determine whether patients with high scores were more likely to be missed surgical candidates. Delay to surgery was defined as the time between the first visit that a surgical candidate was identified by the algorithm and the date of the surgery. RESULTS: A total of 5,285 pediatric and 5,782 adult patients were included to train the ML algorithms. During the study period, 41 children and 23 adults underwent resective epilepsy surgery. In the pediatric cohort, AUC was 0.91 (95% CI 0.87-0.94), positive predictive value (PPV) was 0.08 (0.05-0.10), and negative predictive value (NPV) was 1.00 (0.99-1.00). In the adult cohort, AUC was 0.91 (0.86-0.97), PPV was 0.07 (0.04-0.11), and NPV was 1.00 (0.99-1.00). The models first identified patients at a median of 2.1 years (interquartile range [IQR]: 1.2-4.9 years, maximum: 11.1 years) before their surgery and 1.3 years (IQR: 0.3-4.0 years, maximum: 10.1 years) before their presurgical evaluations. DISCUSSION: ML algorithms can identify surgical candidates earlier in the disease course. Even at specialized epilepsy centers, there is room to shorten the time to surgery. CLASSIFICATION OF EVIDENCE: This study provides Class II evidence that a machine learning algorithm can accurately distinguish patients with epilepsy who require resective surgery from those who do not.