PheValuator 2.0: Methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation.

PURPOSE: Phenotype algorithms are central to performing analyses using observational data. These algorithms translate the clinical idea of a health condition into an executable set of rules allowing for queries of data elements from a database. PheValuator, a software package in the Observational Health Data Sciences and Informatics (OHDSI) tool stack, provides a method to assess the performance characteristics of these algorithms, namely, sensitivity, specificity, and positive and negative predictive value. It uses machine learning to develop predictive models for determining a probabilistic gold standard of subjects for assessment of cases and non-cases of health conditions. PheValuator was developed to complement or even replace the traditional approach of algorithm validation, i.e., by expert assessment of subject records through chart review. Results in our first PheValuator paper suggest a systematic underestimation of the PPV compared to previous results using chart review. In this paper we evaluate modifications made to the method designed to improve its performance. METHODS: The major changes to PheValuator included allowing all diagnostic conditions, clinical observations, drug prescriptions, and laboratory measurements to be included as predictors within the modeling process whereas in the prior version there were significant restrictions on the included predictors. We also have allowed for the inclusion of the temporal relationships of the predictors in the model. To evaluate the performance of the new method, we compared the results from the new and original methods against results found from the literature using traditional validation of algorithms for 19 phenotypes. We performed these tests using data from five commercial databases. RESULTS: In the assessment aggregating all phenotype algorithms, the median difference between the PheValuator estimate and the gold standard estimate for PPV was reduced from -21 (IQR -34, -3) in Version 1.0 to 4 (IQR -3, 15) using Version 2.0. We found a median difference in specificity of 3 (IQR 1, 4.25) for Version 1.0 and 3 (IQR 1, 4) for Version 2.0. The median difference between the two versions of PheValuator and the gold standard for estimates of sensitivity was reduced from -39 (-51, -20) to -16 (-34, -6). CONCLUSION: PheValuator 2.0 produces estimates for the performance characteristics for phenotype algorithms that are significantly closer to estimates from traditional validation through chart review compared to version 1.0. With this tool in researcher's toolkits, methods, such as quantitative bias analysis, may now be used to improve the reliability and reproducibility of research studies using observational data.

Phenotype Algorithms to Identify Hidradenitis Suppurativa Using Real-World Data: Development and Validation Study.

BACKGROUND: Hidradenitis suppurativa (HS) is a potentially debilitating, chronic, recurring inflammatory disease. Observational databases provide opportunities to study the epidemiology of HS. OBJECTIVE: This study's objective was to develop phenotype algorithms for HS suitable for epidemiological studies based on a network of observational databases. METHODS: A data-driven approach was used to develop 4 HS algorithms. A literature search identified prior HS algorithms. Standardized databases from the Observational Medical Outcomes Partnership (n=9) were used to develop 2 incident and 2 prevalent HS phenotype algorithms. Two open-source diagnostic tools, CohortDiagnostics and PheValuator, were used to evaluate and generate phenotype performance metric estimates, including sensitivity, specificity, positive predictive value (PPV), and negative predictive value. RESULTS: We developed 2 prevalent and 2 incident HS algorithms. Validation showed that PPV estimates were highest (mean 86%) for the prevalent HS algorithm requiring at least two HS diagnosis codes. Sensitivity estimates were highest (mean 58%) for the prevalent HS algorithm requiring at least one HS code. CONCLUSIONS: This study illustrates the evaluation process and provides performance metrics for 2 incident and 2 prevalent HS algorithms across 9 observational databases. The use of a rigorous data-driven approach applied to a large number of databases provides confidence that the HS algorithms can correctly identify HS subjects.

Autism: Comorbidities and Treatment Patterns in the Real World, a Retrospective Cohort Study Among Children, Adolescents and Adults Newly Diagnosed with Autism.

We used real world data to summarize comorbidities and interventions among patients newly diagnosed with autism spectrum disorder (ASD). Data were derived from two claims-based US healthcare databases; Medicaid and Optum to construct a retrospective cohort of 36,000 patients. Attention-Deficit-Hyperactivity-Disorder (ADHD) was the most common co-morbidity (Medicaid: 50.09%; Optum: 44.16%), followed by mood disorder (Medicaid: 16.56% and Optum: 17.47%). Most patients received at least one type of treatment. Behavioral therapy was common (74.64% in Medicaid and 71.97% in Optum). More than half the cohorts received at least 1 pharmacotherapy. However, pharmacotherapies were diverse. Combination therapy and therapy switching was common. Understanding the clinical diversity and complexity of patients with ASD is an important first step in understanding unmet therapeutic needs.

Acute care of pediatric patients with sickle cell disease: a simulation performance assessment.

BACKGROUND: Sickle cell disease (SCD) is a rare disorder with cardinal features including hospitalization for vaso-occlusive pain episodes, acute pulmonary injury, and increased infection rates. For physician-trainees, learning optimal SCD management is challenging because of limited exposure to life threatening complications requiring timely interventions. PROCEDURE: To create, demonstrate reliability, and validate simulation-based, acute care SCD scenarios for physician-trainees, seven scenarios were derived from SCD patient cases. For each scenario, participants had 5 minutes to complete diagnostic and treatment interventions. Participants were divided into two groups based on clinical experience: interns or residents/fellows. Two raters scored performances using diagnostic and therapeutic checklists--indicating whether specific actions were performed and a global, 1 (poor) to 9 (excellent), rating. Scenario scores were calculated by averaging rater scores on each metric. Reliability was defined through uniformity in rater scoring and consistency of participant performance over scenarios. Validity was demonstrated by the performance gradient where the more experienced trainees outperform those early in training. RESULTS: Twenty-eight pediatric residents and hematology fellows took part in the study. Reliability for assessment scores overall was moderate. Performance on all but one scenario was moderately predictive of overall performance. Senior resident/fellows performed significantly better than interns. Positive associations existed between overall performance scores (P < 0.01) and months of postgraduate training (P < 0.01). CONCLUSIONS: Mannequin-based simulation is a novel method for teaching pediatric residents SCD-specific acute care skills. The assessment provided reliable and valid measures of trainees' performance. Further studies are needed to determine simulation's utility in education and evaluation.

Precision and accuracy of DXA and pQCT for densitometry of the rat femur.

Measurements of bone mineral density and bone mineral content are key data in the study of osteoporosis and pathologic skeletal disease. Dual-energy X-ray absorptiometry and peripheral quantitative computed tomography are used in human and small animal studies. The purpose of this study was to evaluate the precision, accuracy, and systematic bias of measurement of the rat femur. Comparing machine-measured parameters with standard, nonradiographic measurements, we assessed validation of relative and absolute accuracy. Regression analysis and calculations of percent difference from standard values were used to determine the accuracy of each densitometry technique. Machine-specific and subject-specific precision was evaluated for each densitometer using repeated scans to calculate coefficients of variation. Each of the methods of densitometry examined in this study produced comparable results and was sensitive to small changes following experimental stimuli. Further, our assessment of the precision and accuracy observed between methods of scanning excised rat femurs validates our data acquisition method and serves as a foundation for future densitometry studies.