Validity of forensic cartridge-case comparisons.

This article presents key findings from a research project that evaluated the validity and probative value of cartridge-case comparisons under field-based conditions. Decisions provided by 228 trained firearm examiners across the US showed that forensic cartridge-case comparison is characterized by low error rates. However, inconclusive decisions constituted over one-fifth of all decisions rendered, complicating evaluation of the technique's ability to yield unambiguously correct decisions. Specifically, restricting evaluation to only the conclusive decisions of identification and elimination yielded true-positive and true-negative rates exceeding 99%, but incorporating inconclusives caused these values to drop to 93.4% and 63.5%, respectively. The asymmetric effect on the two rates occurred because inconclusive decisions were rendered six times more frequently for different-source than same-source comparisons. Considering probative value, which is a decision's usefulness for determining a comparison's ground-truth state, conclusive decisions predicted their corresponding ground-truth states with near perfection. Likelihood ratios (LRs) further showed that conclusive decisions greatly increase the odds of a comparison's ground-truth state matching the ground-truth state asserted by the decision. Inconclusive decisions also possessed probative value, predicting different-source status and having a LR indicating that they increase the odds of different-source status. The study also manipulated comparison difficulty by using two firearm models that produce dissimilar cartridge-case markings. The model chosen for being more difficult received more inconclusive decisions for same-source comparisons, resulting in a lower true-positive rate compared to the less difficult model. Relatedly, inconclusive decisions for the less difficult model exhibited more probative value, being more strongly predictive of different-source status.

Universal adaptability: Target-independent inference that competes with propensity scoring.

The gold-standard approaches for gleaning statistically valid conclusions from data involve random sampling from the population. Collecting properly randomized data, however, can be challenging, so modern statistical methods, including propensity score reweighting, aim to enable valid inferences when random sampling is not feasible. We put forth an approach for making inferences based on available data from a source population that may differ in composition in unknown ways from an eventual target population. Whereas propensity scoring requires a separate estimation procedure for each different target population, we show how to build a single estimator, based on source data alone, that allows for efficient and accurate estimates on any downstream target data. We demonstrate, theoretically and empirically, that our target-independent approach to inference, which we dub "universal adaptability," is competitive with target-specific approaches that rely on propensity scoring. Our approach builds on a surprising connection between the problem of inferences in unspecified target populations and the multicalibration problem, studied in the burgeoning field of algorithmic fairness. We show how the multicalibration framework can be employed to yield valid inferences from a single source population across a diverse set of target populations.

The scientific value of numerical measures of human feelings.

Human feelings measured in integers (my happiness is an 8 out of 10, my pain 2 out of 6) have no objective scientific basis. They are "made-up" numbers on a scale that does not exist. Yet such data are extensively collected-despite criticism from, especially, economists-by governments and international organizations. We examine this paradox. We draw upon longitudinal information on the feelings and decisions of tens of thousands of randomly sampled citizens followed through time over four decades in three countries (n = 700,000 approximately). First, we show that a single feelings integer has greater predictive power than does a combined set of economic and social variables. Second, there is a clear inverse relationship between feelings integers and subsequent get-me-out-of-here actions (in the domain of neighborhoods, partners, jobs, and hospital visits). Third, this feelings-to-actions relationship takes a generic form, is consistently replicable, and is fairly close to linear in structure. Therefore, it seems that human beings can successfully operationalize an integer scale for feelings even though there is no true scale. How individuals are able to achieve this is not currently known. The implied scientific puzzle-an inherently cross-disciplinary one-demands attention.

Physiological Microenvironment Dependent Self-Cross-Linking of Multifunctional Nanohybrid for Prolonged Antibacterial Therapy via Synergistic Chemodynamic-Photothermal-Biological Processes.

Herein, a multifunctional nanohybrid (PL@HPFTM nanoparticles) was fabricated to perform the integration of chemodynamic therapy, photothermal therapy, and biological therapy over the long term at a designed location for continuous antibacterial applications. The PL@HPFTM nanoparticles consisted of a polydopamine/hemoglobin/Fe2+ nanocomplex with comodification of tetrazole/alkene groups on the surface as well as coloading of antimicrobial peptides and luminol in the core. During therapy, the PL@HPFTM nanoparticles would selectively cross-link to surrounding bacteria via tetrazole/alkene cycloaddition under chemiluminescence produced by the reaction between luminol and overexpressed H2O2 at the infected area. The resulting PL@HPFTM network not only significantly damaged bacteria by Fe2+-catalyzed ROS production, effective photothermal conversion, and sustained release of antimicrobial peptides but dramatically enhanced the retention time of these therapeutic agents for prolonged antibacterial therapy. Both in vitro and in vivo results have shown that our PL@HPFTM nanoparticles have much higher bactericidal efficiency and remarkably longer periods of validity than free antibacterial nanoparticles.

External Validation of the Discriminative Validity of the ReSVinet Score and Development of Simplified ReSVinet Scores in Secondary Care.

BACKGROUND: There is no consensus on how to best quantify disease severity in infants with respiratory syncytial virus (RSV) and/or bronchiolitis; this lack of a sufficiently validated score complicates the provision of clinical care and, the evaluation of trials of therapeutics and vaccines. The ReSVinet score appears to be one of the most promising; however, it is too time consuming to be incorporated into routine clinical care. We aimed to develop and externally validate simplified versions of this score. METHODS: Data from a multinational (the Netherlands, Spain, and United Kingdom) multicenter case-control study of infants with RSV were used to develop simplified versions of the ReSVinet score by conducting a grid search to determine the best combination of equally weighted parameters to maximize for the discriminative ability (measured by area under the receiver operating characteristic curve [AUROC]) across a range of outcomes (hospitalization, intensive care unit admission, ventilation requirement). Subsequently discriminative validity of the score for a range of secondary care outcomes was externally validated by secondary analysis of datasets from Rwanda and Colombia. RESULTS: Three candidate simplified scores were identified using the development dataset; they were excellent (AUROC >0.9) at discriminating for a range of outcomes, and their performance was not significantly different from the original ReSVinet score despite having fewer parameters. In the external validation datasets, the simplified scores were moderate to excellent (AUROC, 0.7-1) across a range of outcomes. In all outcomes, except in a single dataset for predicting admission to the high-dependency unit, they performed at least as well as the original ReSVinet score. CONCLUSIONS: The candidate simplified scores developed require further external validation in larger datasets, ideally from resource-limited settings before any recommendation regarding their use.

Validity of Clinical Severity Scores for Respiratory Syncytial Virus: A Systematic Review.

BACKGROUND: Respiratory syncytial virus (RSV) is a widespread respiratory pathogen, and RSV-related acute lower respiratory tract infections are the most common cause of respiratory hospitalization in children <2 years of age. Over the last 2 decades, a number of severity scores have been proposed to quantify disease severity for RSV in children, yet there remains no overall consensus on the most clinically useful score. METHODS: We conducted a systematic review of English-language publications in peer-reviewed journals published since January 2000 assessing the validity of severity scores for children (≤24 months of age) with RSV and/or bronchiolitis, and identified the most promising scores. For included articles, (1) validity data were extracted, (2) quality of reporting was assessed using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis checklist (TRIPOD), and (3) quality was assessed using the Prediction Model Risk Of Bias Assessment Tool (PROBAST). To guide the assessment of the validity data, standardized cutoffs were employed, and an explicit definition of what we required to determine a score was sufficiently validated. RESULTS: Our searches identified 8541 results, of which 1779 were excluded as duplicates. After title and abstract screening, 6670 references were excluded. Following full-text screening and snowballing, 32 articles, including 31 scores, were included. The most frequently assessed scores were the modified Tal score and the Wang Bronchiolitis Severity Score; none of the scores were found to be sufficiently validated according to our definition. The reporting and/or design of all the included studies was poor. The best validated score was the Bronchiolitis Score of Sant Joan de Déu, and a number of other promising scores were identified. CONCLUSIONS: No scores were found to be sufficiently validated. Further work is warranted to validate the existing scores, ideally in much larger datasets.

Comparison of predictive models using logistic regression, classification tree, and structural equation model for severe dengue.

BACKGROUND: The aim of this study was to compare the predictive performance of three statistical models-logistic regression, classification tree, and structural equation model (SEM)-in predicting severe dengue illness. METHODS/FINDINGS: We adopted modified classification of dengue illness severity based on WHO 1997 guideline. Predictive models were constructed using demographic factors and laboratory indicators on the day of fever occurrence. We developed statistical predictive models using data from two hospital cohorts in Thailand, consisting of 257 Thai children. Different predictive models for each category of severe dengue illness were developed employing logistic regression, classification tree, and SEM. The probability of discrimination of each model for severe output of disease was analyzed with external validation data sets from 55 and 700 patients not used in model development. From external validation using predictors on the day of presentation to the hospital, the area under the receiver operating characteristic curve was between 0.65 and 0.84 for the regression model. It was between 0.73 and 0.85 for SEM models. Classification tree models showed good results of sensitivity, ranging from 0.95 to 0.99. However, they showed poor specificity ranging from 0.10 to 0.44. CONCLUSIONS: Our study showed that SEM is comparable to logistic regression or classification tree, which was widely used for more severe form of dengue prediction.

Chronic ecologically relevant stress effects on reverse-translated touchscreen assays of reward responsivity and attentional processes in male rats: Implications for depression.

Stagnation in the development of novel therapeutic strategies for treatment-resistant depression has encouraged continued interest in improving preclinical methods. One tactic prioritizes the reverse translation of behavioral tasks developed to objectively quantify depressive phenotypes in patient populations for their use in laboratory animals via touchscreen technology. After cross-species concordance in task outcomes under healthy conditions is confirmed, construct validity can be further enhanced by identifying environmental stressors that reliably produce deficits in task performance that resemble those in depressive participants. The present studies characterized in male rats the ability of two chronic ecologically relevant stressors, inescapable ice water or isolated restraint, to produce depressive-like behavioral phenotypes in the Probabilistic Reward Task (PRT) and Psychomotor Vigilance Task (PVT). These tasks previously have been reverse-translated using touchscreen technology for rodents and nonhuman primates to objectively quantify, respectively, reward responsivity (anhedonia) and attentional processes (impaired cognitive function), each of which are core features of major depressive disorder. In the PRT, both inescapable ice water and isolated restraint produced persistent anhedonic phenotypes compared to non-stressed control performance (i.e., significantly blunted response bias for the richly rewarded stimulus). In the PVT, both chronic stressors impaired attentional processing, revealed by increases in titrated reaction times; however, these deficits largely subsided by the end of the chronic condition. Taken together, these findings confirm the ability of reverse-translated touchscreen tasks to effectively generate behavioral phenotypes that exhibit expected deficits in performance outcomes following exposure to chronic ecologically relevant stress. In turn, this approach is well positioned to appraise the ability of candidate therapeutics to attenuate or reverse such behavioral deficits and, thereby, contribute to preclinical medications development for treatment-resistant depression.

Standardizing to specific target populations in distributed networks and multisite pharmacoepidemiologic studies.

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology.

Variable selection when estimating effects in external target populations.

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (ie, "transport"), some methods require that one account for all effect measure modifiers (EMMs). However, little is known about how including other variables that are not EMMs (ie, non-EMMs) in adjustment sets affects estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing the impacts of covariates that (1) differ (or not) between the trial and the target, (2) are associated with the outcome (or not), and (3) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Inclusion of variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omission of necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations.

Validity and Reproducibility of a Semiquantitative Food Frequency Questionnaire for Measuring Intakes of Foods and Food Groups.

We evaluated the validity and reproducibility of a semiquantitative food frequency questionnaire (FFQ) for measuring intakes of 149 foods and 25 food groups among 736 participants of the Women's Lifestyle Validation Study (WLVS, 2010-2012) and 649 participants of the Men's Lifestyle Validation Study (MLVS, 2011-2013). Validity of the FFQ compared with two 7-day dietary records measured 6 months apart and the reproducibility between 2 FFQs administered 1 year apart (FFQ1 and FFQ2) were assessed using Spearman correlations and intraclass correlation coefficients. The average 1-year reproducibility of FFQ-measured foods was 0.64 in both the WLVS and MLVS. Reproducibility of the food groups (mean = 0.71 among women and 0.72 among men) was generally higher than that for individual foods. Among women, the average validity correlation for individual foods was 0.59 when comparing FFQ2 with the 7-day dietary records. Among men, the corresponding average validity correlation was 0.61. Compared with individual foods, food groups had slightly higher validity correlations in both women (range, 0.45-0.92; mean = 0.61) and men (range, 0.46-0.88; mean = 0.65). This study reaffirms that the FFQ performs well in measuring most foods and food groups and provides data to adjust for measurement errors in epidemiologic studies of foods and food groups.

Comparison of caffeine consumption behavior with plasma caffeine levels as exposure measures in drug-target Mendelian randomization.

Mendelian randomization is an epidemiological technique that can explore the potential effect of perturbing a pharmacological target. Plasma caffeine levels can be used as a biomarker to measure the pharmacological effects of caffeine. Alternatively, this can be assessed using a behavioral proxy, such as average number of caffeinated drinks consumed per day. Either variable can be used as the exposure in a Mendelian randomization investigation, and to select which genetic variants to use as instrumental variables. Another possibility is to choose variants in gene regions with known biological relevance to caffeine level regulation. These choices affect the causal question that is being addressed by the analysis, and the validity of the analysis assumptions. Further, even when using the same genetic variants, the sign of Mendelian randomization estimates (positive or negative) can change depending on the choice of exposure. Some genetic variants that decrease caffeine metabolism associate with higher levels of plasma caffeine, but lower levels of caffeine consumption, as individuals with these variants require less caffeine consumption for the same physiological effect. We explore Mendelian randomization estimates for the effect of caffeine on body mass index, and discuss implications for variant and exposure choice in drug target Mendelian randomization investigations.

Validation of algorithms in studies based on routinely collected health data: general principles.

Clinicians, researchers, regulators, and other decision-makers increasingly rely on evidence from real-world data (RWD), including data routinely accumulating in health and administrative databases. RWD studies often rely on algorithms to operationalize variable definitions. An algorithm is a combination of codes or concepts used to identify persons with a specific health condition or characteristic. Establishing the validity of algorithms is a prerequisite for generating valid study findings that can ultimately inform evidence-based health care. This paper aims to systematize terminology, methods, and practical considerations relevant to the conduct of validation studies of RWD-based algorithms. We discuss measures of algorithm accuracy; gold/reference standard; study size; prioritizing accuracy measures; algorithm portability; and implication for interpretation. Information bias is common in epidemiologic studies, underscoring the importance of transparency in decisions regarding choice and prioritizing measures of algorithm validity. The validity of an algorithm should be judged in the context of a data source, and one size does not fit all. Prioritizing validity measures within a given data source depends on the role of a given variable in the analysis (eligibility criterion, exposure, outcome or covariate). Validation work should be part of routine maintenance of RWD sources.

Thinking computationally in translational psychiatry. A commentary on Neville et al. (2024).

There is a growing focus on the computational aspects of psychiatric disorders in humans. This idea also is gaining traction in nonhuman animal studies. Commenting on a new comprehensive overview of the benefits of applying this approach in translational research by Neville et al. (Cognitive Affective & Behavioral Neuroscience 1-14, 2024), we discuss the implications for translational model validity within this framework. We argue that thinking computationally in translational psychiatry calls for a change in the way that we evaluate animal models of human psychiatric processes, with a shift in focus towards symptom-producing computations rather than the symptoms themselves. Further, in line with Neville et al.'s adoption of the reinforcement learning framework to model animal behaviour, we illustrate how this approach can be applied beyond simple decision-making paradigms to model more naturalistic behaviours.

Test-retest reliability of the play-or-pass version of the Iowa Gambling Task.

The Iowa Gambling Task (IGT) is used to assess decision-making in clinical populations. The original IGT does not disambiguate reward and punishment learning; however, an adaptation of the task, the "play-or-pass" IGT, was developed to better distinguish between reward and punishment learning. We evaluated the test-retest reliability of measures of reward and punishment learning from the play-or-pass IGT and examined associations with self-reported measures of reward/punishment sensitivity and internalizing symptoms. Participants completed the task across two sessions, and we calculated mean-level differences and rank-order stability of behavioral measures across the two sessions using traditional scoring, involving session-wide choice proportions, and computational modeling, involving estimates of different aspects of trial-level learning. Measures using both approaches were reliable; however, computational modeling provided more insights regarding between-session changes in performance, and how performance related to self-reported measures of reward/punishment sensitivity and internalizing symptoms. Our results show promise in using the play-or-pass IGT to assess decision-making; however, further work is still necessary to validate the play-or-pass IGT.

Mouse models of sickle cell disease: Imperfect and yet very informative.

The root cause of sickle cell disease (SCD) has been known for nearly a century, however, few therapies to treat the disease are available. Over several decades of work, with advances in gene editing technology and after several iterations of mice with differing genotype/phenotype relationships, researchers have developed humanized SCD mouse models. However, while a large body of preclinical studies has led to huge gains in basic science knowledge about SCD in mice, this knowledge has not led to the development of effective therapies to treat SCD-related complications in humans, thus leading to frustration with the paucity of translational progress in the SCD field. The use of mouse models to study human diseases is based on the genetic and phenotypic similarities between mouse and humans (face validity). The Berkeley and Townes SCD mice express only human globin chains and no mouse hemoglobin. With this genetic composition, these models present many phenotypic similarities, but also significant discrepancies that should be considered when interpreting preclinical studies results. Reviewing genetic and phenotypic similarities and discrepancies and examining studies that have translated to humans and those that have not, offer a better perspective of construct, face, and predictive validities of humanized SCD mouse models.

Accuracy of patient race and ethnicity data in a central cancer registry.

PURPOSE: Race and Hispanic ethnicity data can be challenging for central cancer registries to collect. We evaluated the accuracy of the race and Hispanic ethnicity variables collected by the Utah Cancer Registry compared to self-report. METHODS: Participants were 3,162 cancer survivors who completed questionnaires administered in 2015-2022 by the Utah Cancer Registry. Each survey included separate questions collecting race and Hispanic ethnicity, respectively. Registry-collected race and Hispanic ethnicity were compared to self-reported values for the same individuals. We calculated sensitivity and specificity for each race category and Hispanic ethnicity separately. RESULTS: Survey participants included 323 (10.2%) survivors identifying as Hispanic, a lower proportion Hispanic than the 12.1% in the registry Hispanic variable (sensitivity 88.2%, specificity 96.5%). For race, 43 participants (1.4%) self-identified as American Indian or Alaska Native (AIAN), 32 (1.0%) as Asian, 23 (0.7%) as Black or African American, 16 (0.5%) Pacific Islander (PI), and 2994 (94.7%) as White. The registry race variable classified a smaller proportion of survivors as members of each of these race groups except White. Sensitivity for classification of race as AIAN was 9.3%, Asian 40.6%, Black 60.9%, PI 25.0%, and specificity for each of these groups was > 99%. Sensitivity and specificity for White were 98.8% and 47.4%. CONCLUSION: Cancer registry race and Hispanic ethnicity data often did not match the individual's self-identification. Of particular concern is the high proportion of AIAN individuals whose race is misclassified. Continued attention should be directed to the accurate capture of race and ethnicity data by hospitals.

Health utility assessments in individuals undergoing diagnostic and surveillance colonoscopy: improved discrimination with a cancer-specific scale.

PURPOSE: To compare the sensitivity and discriminant validity of generic and cancer-specific measures for assessing health-related quality of life (HRQoL) for individuals undergoing diagnostic or surveillance colonoscopy for colorectal cancer. METHODS: HRQoL was assessed using EQ-5D-5L (generic), and EORTC QLQ-C30 (cancer-specific) scales, 14 days after (baseline) and one-year following colonoscopy (follow-up). Utility scores were calculated by mapping EORTC-QLQ-C30 onto QLU-C10D. Differences between participants with different indications for colonoscopy (positive faecal occult blood test (FOBT), surveillance, or symptoms) and colonoscopy findings (no polyps, polyps, or cancer) were tested using Wilcoxon-Mann-Whitney and Kruskal-Wallis H tests. Sensitivity was assessed by calculating the ceiling effects (proportion reporting the best possible level). RESULTS: 246 adults completed the survey, including those undergoing colonoscopy for symptoms (n = 87), positive FOBT (n = 92) or surveillance (n = 67). Those with symptoms had the lowest HRQoL at both baseline and follow-up, with differences observed within the HRQoL domains/areas of role function, appetite loss and bowel function on the QLU-C10D. No differences were found in HRQoL when stratified by findings at colonoscopy with both measures or when comparing baseline and follow-up responses. Participants reporting full health with EQ-5D-5L (21% at baseline and 16% at follow-up) still had problems on the QLU-C10D, with fatigue and sleep at baseline and with role function and fatigue at follow-up. CONCLUSION: Patients undergoing colonoscopy for symptoms had lower HRQoL compared to surveillance or positive FOBT. The cancer-specific QLU-C10D was more sensitive and had greater discriminant ability between patients undergoing colonoscopy for different indications.

Evaluation of the diagnostic accuracy of exome sequencing and its impact on diagnostic thinking for patients with rare disease in a publicly funded health care system: A prospective cohort study.

PURPOSE: To evaluate the diagnostic utility of publicly funded clinical exome sequencing (ES) for patients with suspected rare genetic diseases. METHODS: We prospectively enrolled 297 probands who met eligibility criteria and received ES across 5 sites in Ontario, Canada, and extracted data from medical records and clinician surveys. Using the Fryback and Thornbury Efficacy Framework, we assessed diagnostic accuracy by examining laboratory interpretation of results and assessed diagnostic thinking by examining the clinical interpretation of results and whether clinical-molecular diagnoses would have been achieved via alternative hypothetical molecular tests. RESULTS: Laboratories reported 105 molecular diagnoses and 165 uncertain results in known and novel genes. Of these, clinicians interpreted 102 of 105 (97%) molecular diagnoses and 6 of 165 (4%) uncertain results as clinical-molecular diagnoses. The 108 clinical-molecular diagnoses were in 104 families (35% diagnostic yield). Each eligibility criteria resulted in diagnostic yields of 30% to 40%, and higher yields were achieved when >2 eligibility criteria were met (up to 45%). Hypothetical tests would have identified 61% of clinical-molecular diagnoses. CONCLUSION: We demonstrate robustness in eligibility criteria and high clinical validity of laboratory results from ES testing. The importance of ES was highlighted by the potential 40% of patients that would have gone undiagnosed without this test.