CAGI6 ID-Challenge: Assessment of phenotype and variant predictions in 415 children with Neurodevelopmental Disorders (NDDs).

In the context of the Critical Assessment of the Genome Interpretation, 6th edition (CAGI6), the Genetics of Neurodevelopmental Disorders Lab in Padua proposed a new ID-challenge to give the opportunity of developing computational methods for predicting patient's phenotype and the causal variants. Eight research teams and 30 models had access to the phenotype details and real genetic data, based on the sequences of 74 genes (VCF format) in 415 pediatric patients affected by Neurodevelopmental Disorders (NDDs). NDDs are clinically and genetically heterogeneous conditions, with onset in infant age. In this study we evaluate the ability and accuracy of computational methods to predict comorbid phenotypes based on clinical features described in each patient and causal variants. Finally, we asked to develop a method to find new possible genetic causes for patients without a genetic diagnosis. As already done for the CAGI5, seven clinical features (ID, ASD, ataxia, epilepsy, microcephaly, macrocephaly, hypotonia), and variants (causative, putative pathogenic and contributing factors) were provided. Considering the overall clinical manifestation of our cohort, we give out the variant data and phenotypic traits of the 150 patients from CAGI5 ID-Challenge as training and validation for the prediction methods development.

Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine.

OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.

Evaluating construct validity of computable acute respiratory distress syndrome definitions in adults hospitalized with COVID-19: an electronic health records based approach.

BACKGROUND: Evolving ARDS epidemiology and management during COVID-19 have prompted calls to reexamine the construct validity of Berlin criteria, which have been rarely evaluated in real-world data. We developed a Berlin ARDS definition (EHR-Berlin) computable in electronic health records (EHR) to (1) assess its construct validity, and (2) assess how expanding its criteria affected validity. METHODS: We performed a retrospective cohort study at two tertiary care hospitals with one EHR, among adults hospitalized with COVID-19 February 2020-March 2021. We assessed five candidate definitions for ARDS: the EHR-Berlin definition modeled on Berlin criteria, and four alternatives informed by recent proposals to expand criteria and include patients on high-flow oxygen (EHR-Alternative 1), relax imaging criteria (EHR-Alternatives 2-3), and extend timing windows (EHR-Alternative 4). We evaluated two aspects of construct validity for the EHR-Berlin definition: (1) criterion validity: agreement with manual ARDS classification by experts, available in 175 patients; (2) predictive validity: relationships with hospital mortality, assessed by Pearson r and by area under the receiver operating curve (AUROC). We assessed predictive validity and timing of identification of EHR-Berlin definition compared to alternative definitions. RESULTS: Among 765 patients, mean (SD) age was 57 (18) years and 471 (62%) were male. The EHR-Berlin definition classified 171 (22%) patients as ARDS, which had high agreement with manual classification (kappa 0.85), and was associated with mortality (Pearson r = 0.39; AUROC 0.72, 95% CI 0.68, 0.77). In comparison, EHR-Alternative 1 classified 219 (29%) patients as ARDS, maintained similar relationships to mortality (r = 0.40; AUROC 0.74, 95% CI 0.70, 0.79, Delong test P = 0.14), and identified patients earlier in their hospitalization (median 13 vs. 15 h from admission, Wilcoxon signed-rank test P < 0.001). EHR-Alternative 3, which removed imaging criteria, had similar correlation (r = 0.41) but better discrimination for mortality (AUROC 0.76, 95% CI 0.72, 0.80; P = 0.036), and identified patients median 2 h (P < 0.001) from admission. CONCLUSIONS: The EHR-Berlin definition can enable ARDS identification with high criterion validity, supporting large-scale study and surveillance. There are opportunities to expand the Berlin criteria that preserve predictive validity and facilitate earlier identification.

Proteomic Analysis of Huntington's Disease Medium Spiny Neurons Identifies Alterations in Lipid Droplets.

Huntington's disease (HD) is a neurodegenerative disease caused by a CAG repeat expansion in the Huntingtin (HTT) gene. The resulting polyglutamine (polyQ) tract alters the function of the HTT protein. Although HTT is expressed in different tissues, the medium-spiny projection neurons (MSNs) in the striatum are particularly vulnerable in HD. Thus, we sought to define the proteome of human HD patient-derived MSNs. We differentiated HD72-induced pluripotent stem cells and isogenic controls into MSNs and carried out quantitative proteomic analysis. Using data-dependent acquisitions with FAIMS for label-free quantification on the Orbitrap Lumos mass spectrometer, we identified 6323 proteins with at least two unique peptides. Of these, 901 proteins were altered significantly more in the HD72-MSNs than in isogenic controls. Functional enrichment analysis of upregulated proteins demonstrated extracellular matrix and DNA signaling (DNA replication pathway, double-strand break repair, G1/S transition) with the highest significance. Conversely, processes associated with the downregulated proteins included neurogenesis-axogenesis, the brain-derived neurotrophic factor-signaling pathway, Ephrin-A:EphA pathway, regulation of synaptic plasticity, triglyceride homeostasis cholesterol, plasmid lipoprotein particle immune response, interferon-γ signaling, immune system major histocompatibility complex, lipid metabolism, and cellular response to stimulus. Moreover, proteins involved in the formation and maintenance of axons, dendrites, and synapses (e.g., septin protein members) were dysregulated in HD72-MSNs. Importantly, lipid metabolism pathways were altered, and using quantitative image analysis, we found that lipid droplets accumulated in the HD72-MSN, suggesting a deficit in the turnover of lipids possibly through lipophagy. Our proteomics analysis of HD72-MSNs identified relevant pathways that are altered in MSNs and confirm current and new therapeutic targets for HD.

Technology Platforms and Approaches for Building and Evaluating Machine Learning Methods in Healthcare.

BACKGROUND: Artificial intelligence (AI) methods are becoming increasingly commonly implemented in healthcare as decision support, business intelligence tools, or, in some cases, Food and Drug Administration-approved clinical decision-makers. Advanced lab-based diagnostic tools are increasingly becoming AI driven. The path from data to machine learning methods is an active area for research and quality improvement, and there are few established best practices. With data being generated at an unprecedented rate, there is a need for processes that enable data science investigation that protect patient privacy and minimize other business risks. New approaches for data sharing are being utilized that lower these risks. CONTENT: In this short review, clinical and translational AI governance is introduced along with approaches for securely building, sharing, and validating accurate and fair models. This is a constantly evolving field, and there is much interest in collecting data using standards, sharing data, building new models, evaluating models, sharing models, and, of course, implementing models into practice. SUMMARY: AI is an active area of research and development broadly for healthcare and laboratory testing. Robust data governance and machine learning methodological governance are required. New approaches for data sharing are enabling the development of models and their evaluation. Evaluation of methods is difficult, particularly when the evaluation is performed by the team developing the method, and should ideally be prospective. New technologies have enabled standardization of platforms for moving analytics and data science methods.

Predicting decompression surgery by applying multimodal deep learning to patients' structured and unstructured health data.

BACKGROUND: Low back pain (LBP) is a common condition made up of a variety of anatomic and clinical subtypes. Lumbar disc herniation (LDH) and lumbar spinal stenosis (LSS) are two subtypes highly associated with LBP. Patients with LDH/LSS are often started with non-surgical treatments and if those are not effective then go on to have decompression surgery. However, recommendation of surgery is complicated as the outcome may depend on the patient's health characteristics. We developed a deep learning (DL) model to predict decompression surgery for patients with LDH/LSS. MATERIALS AND METHOD: We used datasets of 8387 and 8620 patients from a prospective study that collected data from four healthcare systems to predict early (within 2 months) and late surgery (within 12 months after a 2 month gap), respectively. We developed a DL model to use patients' demographics, diagnosis and procedure codes, drug names, and diagnostic imaging reports to predict surgery. For each prediction task, we evaluated the model's performance using classical and generalizability evaluation. For classical evaluation, we split the data into training (80%) and testing (20%). For generalizability evaluation, we split the data based on the healthcare system. We used the area under the curve (AUC) to assess performance for each evaluation. We compared results to a benchmark model (i.e. LASSO logistic regression). RESULTS: For classical performance, the DL model outperformed the benchmark model for early surgery with an AUC of 0.725 compared to 0.597. For late surgery, the DL model outperformed the benchmark model with an AUC of 0.655 compared to 0.635. For generalizability performance, the DL model outperformed the benchmark model for early surgery. For late surgery, the benchmark model outperformed the DL model. CONCLUSIONS: For early surgery, the DL model was preferred for classical and generalizability evaluation. However, for late surgery, the benchmark and DL model had comparable performance. Depending on the prediction task, the balance of performance may shift between DL and a conventional ML method. As a result, thorough assessment is needed to quantify the value of DL, a relatively computationally expensive, time-consuming and less interpretable method.

Multi-objective prioritization of genes for high-throughput functional assays towards improved clinical variant classification.

The accurate interpretation of genetic variants is essential for clinical actionability. However, a majority of variants remain of uncertain significance. Multiplexed assays of variant effects (MAVEs), can help provide functional evidence for variants of uncertain significance (VUS) at the scale of entire genes. Although the systematic prioritization of genes for such assays has been of great interest from the clinical perspective, existing strategies have rarely emphasized this motivation. Here, we propose three objectives for quantifying the importance of genes each satisfying a specific clinical goal: (1) Movability scores to prioritize genes with the most VUS moving to non-VUS categories, (2) Correction scores to prioritize genes with the most pathogenic and/or benign variants that could be reclassified, and (3) Uncertainty scores to prioritize genes with VUS for which variant pathogenicity predictors used in clinical classification exhibit the greatest uncertainty. We demonstrate that existing approaches are sub-optimal when considering these explicit clinical objectives. We also propose a combined weighted score that optimizes the three objectives simultaneously and finds optimal weights to improve over existing approaches. Our strategy generally results in better performance than existing knowledge-driven and data-driven strategies and yields gene sets that are clinically relevant. Our work has implications for systematic efforts that aim to iterate between predictor development, experimentation and translation to the clinic.

A Multifaceted benchmarking of synthetic electronic health record generation models.

Synthetic health data have the potential to mitigate privacy concerns in supporting biomedical research and healthcare applications. Modern approaches for data generation continue to evolve and demonstrate remarkable potential. Yet there is a lack of a systematic assessment framework to benchmark methods as they emerge and determine which methods are most appropriate for which use cases. In this work, we introduce a systematic benchmarking framework to appraise key characteristics with respect to utility and privacy metrics. We apply the framework to evaluate synthetic data generation methods for electronic health records data from two large academic medical centers with respect to several use cases. The results illustrate that there is a utility-privacy tradeoff for sharing synthetic health data and further indicate that no method is unequivocally the best on all criteria in each use case, which makes it evident why synthetic data generation methods need to be assessed in context.

Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria.

Recommendations from the American College of Medical Genetics and Genomics and the Association for Molecular Pathology (ACMG/AMP) for interpreting sequence variants specify the use of computational predictors as "supporting" level of evidence for pathogenicity or benignity using criteria PP3 and BP4, respectively. However, score intervals defined by tool developers, and ACMG/AMP recommendations that require the consensus of multiple predictors, lack quantitative support. Previously, we described a probabilistic framework that quantified the strengths of evidence (supporting, moderate, strong, very strong) within ACMG/AMP recommendations. We have extended this framework to computational predictors and introduce a new standard that converts a tool's scores to PP3 and BP4 evidence strengths. Our approach is based on estimating the local positive predictive value and can calibrate any computational tool or other continuous-scale evidence on any variant type. We estimate thresholds (score intervals) corresponding to each strength of evidence for pathogenicity and benignity for thirteen missense variant interpretation tools, using carefully assembled independent data sets. Most tools achieved supporting evidence level for both pathogenic and benign classification using newly established thresholds. Multiple tools reached score thresholds justifying moderate and several reached strong evidence levels. One tool reached very strong evidence level for benign classification on some variants. Based on these findings, we provide recommendations for evidence-based revisions of the PP3 and BP4 ACMG/AMP criteria using individual tools and future assessment of computational methods for clinical interpretation.

Postnatal Conditional Deletion of Bcl11b in Striatal Projection Neurons Mimics the Transcriptional Signature of Huntington's Disease.

The dysregulation of striatal gene expression and function is linked to multiple diseases, including Huntington's disease (HD), Parkinson's disease, X-linked dystonia-parkinsonism (XDP), addiction, autism, and schizophrenia. Striatal medium spiny neurons (MSNs) make up 90% of the neurons in the striatum and are critical to motor control. The transcription factor, Bcl11b (also known as Ctip2), is required for striatal development, but the function of Bcl11b in adult MSNs in vivo has not been investigated. We conditionally deleted Bcl11b specifically in postnatal MSNs and performed a transcriptomic and behavioral analysis on these mice. Multiple enrichment analyses showed that the D9-Cre-Bcl11btm1.1Leid transcriptional profile was similar to the HD gene expression in mouse and human data sets. A Gene Ontology enrichment analysis linked D9-Cre-Bcl11btm1.1Leid to calcium, synapse organization, specifically including the dopaminergic synapse, protein dephosphorylation, and HDAC-signaling, commonly dysregulated pathways in HD. D9-Cre-Bcl11btm1.1Leid mice had decreased DARPP-32/Ppp1r1b in MSNs and behavioral deficits, demonstrating the dysregulation of a subtype of the dopamine D2 receptor expressing MSNs. Finally, in human HD isogenic MSNs, the mislocalization of BCL11B into nuclear aggregates points to a mechanism for BCL11B loss of function in HD. Our results suggest that BCL11B is important for the function and maintenance of mature MSNs and Bcl11b loss of function drives, in part, the transcriptomic and functional changes in HD.

A Comparison of Natural Language Processing Methods for the Classification of Lumbar Spine Imaging Findings Related to Lower Back Pain.

RATIONALE AND OBJECTIVES: The use of natural language processing (NLP) in radiology provides an opportunity to assist clinicians with phenotyping patients. However, the performance and generalizability of NLP across healthcare systems is uncertain. We assessed the performance within and generalizability across four healthcare systems of different NLP representational methods, coupled with elastic-net logistic regression to classify lower back pain-related findings from lumbar spine imaging reports. MATERIALS AND METHODS: We used a dataset of 871 X-ray and magnetic resonance imaging reports sampled from a prospective study across four healthcare systems between October 2013 and September 2016. We annotated each report for 26 findings potentially related to lower back pain. Our framework applied four different NLP methods to convert text into feature sets (representations). For each representation, our framework used an elastic-net logistic regression model for each finding (i.e., 26 binary or "one-vs.-rest" classification models). For performance evaluation, we split data into training (80%, 697/871) and testing (20%, 174/871). In the training set, we used cross validation to identify the optimal hyperparameter value and then retrained on the full training set. We then assessed performance based on area under the curve (AUC) for the test set. We repeated this process 25 times with each repeat using a different random train/test split of the data, so that we could estimate 95% confidence intervals, and assess significant difference in performance between representations. For generalizability evaluation, we trained models on data from three healthcare systems with cross validation and then tested on the fourth. We repeated this process for each system, then calculated mean and standard deviation (SD) of AUC across the systems. RESULTS: For individual representations, n-grams had the best average performance across all 26 findings (AUC: 0.960). For generalizability, document embeddings had the most consistent average performance across systems (SD: 0.010). Out of these 26 findings, we considered eight as potentially clinically important (any stenosis, central stenosis, lateral stenosis, foraminal stenosis, disc extrusion, nerve root displacement compression, endplate edema, and listhesis grade 2) since they have a relatively greater association with a history of lower back pain compared to the remaining 18 classes. We found a similar pattern for these eight in which n-grams and document embeddings had the best average performance (AUC: 0.954) and generalizability (SD: 0.007), respectively. CONCLUSION: Based on performance assessment, we found that n-grams is the preferred method if classifier development and deployment occur at the same system. However, for deployment at multiple systems outside of the development system, or potentially if physician behavior changes within a system, one should consider document embeddings since embeddings appear to have the most consistent performance across systems.

Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research.

BACKGROUND: Suicide is the 10th leading cause of death in the United States, with >47,000 deaths in 2019. Most people who died by suicide had contact with the health care system in the year before their death. Health care provider training is a top research priority identified by the National Action Alliance for Suicide Prevention; however, evidence-based approaches that target skill-building are resource intensive and difficult to implement. Advances in artificial intelligence technology hold promise for improving the scalability and sustainability of training methods, as it is now possible for computers to assess the intervention delivery skills of trainees and provide feedback to guide skill improvements. Much remains to be known about how best to integrate these novel technologies into continuing education for health care providers. OBJECTIVE: In Project WISE (Workplace Integrated Support and Education), we aim to develop e-learning training in suicide safety planning, enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The research aims include identifying strategies for the implementation and workflow integration of both the training and safety planning with patients, adapting 2 existing technologies to enhance general counseling skills for use in suicide safety planning (a conversational agent and an artificial intelligence-based feedback system), observing training acceptability and nurse engagement with the training components, and assessing the feasibility of recruitment, retention, and collection of longitudinal self-report and electronic health record data for patients identified as at risk of suicide. METHODS: Our developmental research includes qualitative and observational methods to explore the implementation context and technology usability, formative evaluation of the training paradigm, and pilot research to assess the feasibility of conducting a future cluster randomized pragmatic trial. The trial will examine whether patients hospitalized for medical or surgical reasons or traumatic injury who are at risk of suicide have better suicide-related postdischarge outcomes when admitted to a unit with nurses trained using the skill-building technology than those admitted to a unit with untrained nurses. The research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center. RESULTS: Project WISE was funded in July 2019. As of September 2021, we have completed focus groups and usability testing with 27 acute care and 3 acute and intensive care nurses. We began data collection for research aims 3 and 4 in November 2021. All research has been approved by the University of Washington institutional review board. CONCLUSIONS: Project WISE aims to further the national agenda to improve suicide prevention in health care settings by training nurses in suicide prevention with medically hospitalized patients using novel e-learning technologies. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/33695.

Unbiased identification of novel transcription factors in striatal compartmentation and striosome maturation.

Many diseases are linked to dysregulation of the striatum. Striatal function depends on neuronal compartmentation into striosomes and matrix. Striatal projection neurons are GABAergic medium spiny neurons (MSNs), subtyped by selective expression of receptors, neuropeptides, and other gene families. Neurogenesis of the striosome and matrix occurs in separate waves, but the factors regulating compartmentation and neuronal differentiation are largely unidentified. We performed RNA- and ATAC-seq on sorted striosome and matrix cells at postnatal day 3, using the Nr4a1-EGFP striosome reporter mouse. Focusing on the striosome, we validated the localization and/or role of Irx1, Foxf2, Olig2, and Stat1/2 in the developing striosome and the in vivo enhancer function of a striosome-specific open chromatin region 4.4 Kb downstream of Olig2. These data provide novel tools to dissect and manipulate the networks regulating MSN compartmentation and differentiation, including in human iPSC-derived striatal neurons for disease modeling and drug discovery.

From the clinic to the community: Can health system data accurately estimate population obesity prevalence?

OBJECTIVE: Health system data were assessed for how well they can estimate obesity prevalence in census tracts. METHODS: Clinical visit data were available from two large health systems (Kaiser Permanente Washington and University of Washington Medicine) in King County, Washington, as were census tract-level obesity prevalence estimates from the Behavioral Risk Factor Surveillance System (BRFSS). The health system data were geocoded to identify each patient's tract of residence, and the cross-sectional concordance between census tract-level obesity prevalence estimates computed from the two health systems in 2005 to 2006 and the concordance between University of Washington Medicine and BRFSS from 2012 to 2016 were assessed. RESULTS: The spatial distribution of obesity was similar between the health systems (Spearman r = 0.63). The University of Washington Medicine estimates of rank order correlated well with BRFSS estimates (Spearman r = 0.85), though prevalence estimates from BRFSS were lower (mean obesity prevalence = 26% for University of Washington Medicine versus 20% for BRFSS, Wilcoxon rank sum test p < 0.001). Across all data sources, obesity was more prevalent in tracts with less educational attainment. CONCLUSIONS: Health system clinical weight data can reliably replicate census tract-level spatial patterns in the ranking of obesity prevalence. Health system data may be an efficient resource for geographic obesity surveillance.

Development and Validation of ARC, a Model for Anticipating Acute Respiratory Failure in Coronavirus Disease 2019 Patients.

OBJECTIVES: To evaluate factors predictive of clinical progression among coronavirus disease 2019 patients following admission, and whether continuous, automated assessments of patient status may contribute to optimal monitoring and management. DESIGN: Retrospective cohort for algorithm training, testing, and validation. SETTING: Eight hospitals across two geographically distinct regions. PATIENTS: Two-thousand fifteen hospitalized coronavirus disease 2019-positive patients. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Anticipating Respiratory failure in Coronavirus disease (ARC), a clinically interpretable, continuously monitoring prognostic model of acute respiratory failure in hospitalized coronavirus disease 2019 patients, was developed and validated. An analysis of the most important clinical predictors aligns with key risk factors identified by other investigators but contributes new insights regarding the time at which key factors first begin to exhibit aberrency and distinguishes features predictive of acute respiratory failure in coronavirus disease 2019 versus pneumonia caused by other types of infection. Departing from prior work, ARC was designed to update continuously over time as new observations (vitals and laboratory test results) are recorded in the electronic health record. Validation against data from two geographically distinct health systems showed that the proposed model achieved 75% specificity and 77% sensitivity and predicted acute respiratory failure at a median time of 32 hours prior to onset. Over 80% of true-positive alerts occurred in non-ICU settings. CONCLUSIONS: Patients admitted to non-ICU environments with coronavirus disease 2019 are at ongoing risk of clinical progression to severe disease, yet it is challenging to anticipate which patients will develop acute respiratory failure. A continuously monitoring prognostic model has potential to facilitate anticipatory rather than reactive approaches to escalation of care (e.g., earlier initiation of treatments for severe disease or structured monitoring and therapeutic interventions for high-risk patients).

Forecasting Future Asthma Hospital Encounters of Patients With Asthma in an Academic Health Care System: Predictive Model Development and Secondary Analysis Study.

BACKGROUND: Asthma affects a large proportion of the population and leads to many hospital encounters involving both hospitalizations and emergency department visits every year. To lower the number of such encounters, many health care systems and health plans deploy predictive models to prospectively identify patients at high risk and offer them care management services for preventive care. However, the previous models do not have sufficient accuracy for serving this purpose well. Embracing the modeling strategy of examining many candidate features, we built a new machine learning model to forecast future asthma hospital encounters of patients with asthma at Intermountain Healthcare, a nonacademic health care system. This model is more accurate than the previously published models. However, it is unclear how well our modeling strategy generalizes to academic health care systems, whose patient composition differs from that of Intermountain Healthcare. OBJECTIVE: This study aims to evaluate the generalizability of our modeling strategy to the University of Washington Medicine (UWM), an academic health care system. METHODS: All adult patients with asthma who visited UWM facilities between 2011 and 2018 served as the patient cohort. We considered 234 candidate features. Through a secondary analysis of 82,888 UWM data instances from 2011 to 2018, we built a machine learning model to forecast asthma hospital encounters of patients with asthma in the subsequent 12 months. RESULTS: Our UWM model yielded an area under the receiver operating characteristic curve (AUC) of 0.902. When placing the cutoff point for making binary classification at the top 10% (1464/14,644) of patients with asthma with the largest forecasted risk, our UWM model yielded an accuracy of 90.6% (13,268/14,644), a sensitivity of 70.2% (153/218), and a specificity of 90.91% (13,115/14,426). CONCLUSIONS: Our modeling strategy showed excellent generalizability to the UWM, leading to a model with an AUC that is higher than all of the AUCs previously reported in the literature for forecasting asthma hospital encounters. After further optimization, our model could be used to facilitate the efficient and effective allocation of asthma care management resources to improve outcomes. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/resprot.5039.

Establishing the reliability of algorithms.

As rich biomedical data streams are accumulating across people and time, they provide a powerful opportunity to address limitations in our existing scientific knowledge and to overcome operational challenges in healthcare and life sciences. Yet the relative weighting of insights vs. methodologies in our current research ecosystem tends to skew the computational community away from algorithm evaluation and operationalization, resulting in a well-reported trend towards the proliferation of scientific outcomes of unknown reliability. Algorithm selection and use is hindered by several problems that persist across our field. One is the impact of the self-assessment bias, which can lead to mis-representations in the accuracy of research results. A second challenge is the impact of data context on algorithm performance. Biology and medicine are dynamic and heterogeneous. Data is collected under varying conditions. For algorithms, this means that performance is not universal - and need to be evaluated across a range of contexts. These issues are increasingly difficult as algorithms are trained and used on data collected in the real-world, outside of the traditional clinical research lab. In these cases, data collection is neither supervised nor well controlled and data access may be limited by privacy or proprietary reasons. Therefore, there is a risk that algorithms will be applied to data that are outside of the scope of the intent of the original training data provided. This workshop will focus on approaches that are emerging across the researcher community to quantify the accuracy of algorithms and the reliability of their outputs.

Rationale and design of the Kidney Precision Medicine Project.

Chronic kidney disease (CKD) and acute kidney injury (AKI) are common, heterogeneous, and morbid diseases. Mechanistic characterization of CKD and AKI in patients may facilitate a precision-medicine approach to prevention, diagnosis, and treatment. The Kidney Precision Medicine Project aims to ethically and safely obtain kidney biopsies from participants with CKD or AKI, create a reference kidney atlas, and characterize disease subgroups to stratify patients based on molecular features of disease, clinical characteristics, and associated outcomes. An additional aim is to identify critical cells, pathways, and targets for novel therapies and preventive strategies. This project is a multicenter prospective cohort study of adults with CKD or AKI who undergo a protocol kidney biopsy for research purposes. This investigation focuses on kidney diseases that are most prevalent and therefore substantially burden the public health, including CKD attributed to diabetes or hypertension and AKI attributed to ischemic and toxic injuries. Reference kidney tissues (for example, living-donor kidney biopsies) will also be evaluated. Traditional and digital pathology will be combined with transcriptomic, proteomic, and metabolomic analysis of the kidney tissue as well as deep clinical phenotyping for supervised and unsupervised subgroup analysis and systems biology analysis. Participants will be followed prospectively for 10 years to ascertain clinical outcomes. Cell types, locations, and functions will be characterized in health and disease in an open, searchable, online kidney tissue atlas. All data from the Kidney Precision Medicine Project will be made readily available for broad use by scientists, clinicians, and patients.