Validation of algorithms in studies based on routinely collected health data: general principles.

Clinicians, researchers, regulators, and other decision-makers increasingly rely on evidence from real-world data (RWD), including data routinely accumulating in health and administrative databases. RWD studies often rely on algorithms to operationalize variable definitions. An algorithm is a combination of codes or concepts used to identify persons with a specific health condition or characteristic. Establishing the validity of algorithms is a prerequisite for generating valid study findings that can ultimately inform evidence-based health care. This paper aims to systematize terminology, methods, and practical considerations relevant to the conduct of validation studies of RWD-based algorithms. We discuss measures of algorithm accuracy; gold/reference standard; study size; prioritizing accuracy measures; algorithm portability; and implication for interpretation. Information bias is common in epidemiologic studies, underscoring the importance of transparency in decisions regarding choice and prioritizing measures of algorithm validity. The validity of an algorithm should be judged in the context of a data source, and one size does not fit all. Prioritizing validity measures within a given data source depends on the role of a given variable in the analysis (eligibility criterion, exposure, outcome or covariate). Validation work should be part of routine maintenance of RWD sources.

On the value of advanced information about delayed rewards.

In a variety of laboratory preparations, several animal species prefer signaled over unsignaled outcomes. Here we examine whether pigeons prefer options that signal the delay to reward over options that do not and how this preference changes with the ratio of the delays. We offered pigeons repeated choices between two alternatives leading to a short or a long delay to reward. For one alternative (informative), the short and long delays were reliably signaled by different stimuli (e.g., SS for short delays, SL for long delays). For the other (non-informative), the delays were not reliably signaled by the stimuli presented (S1 and S2). Across conditions, we varied the durations of the short and long delays, hence their ratio, while keeping the average delay to reward constant. Pigeons preferred the informative over the non-informative option and this preference became stronger as the ratio of the long to the short delay increased. A modified version of the Δ-Σ hypothesis (González et al., J Exp Anal Behav 113(3):591-608. https://doi.org/10.1002/jeab.595 , 2020a) incorporating a contrast-like process between the immediacies to reward signaled by each stimulus accounted well for our findings. Functionally, we argue that a preference for signaled delays hinges on the potential instrumental advantage typically conveyed by information.

Nondifferential Treatment Misclassification Biases Toward the Null? Not a Safe Bet for Active Comparator Studies.

Active comparator studies are increasingly common, particularly in pharmacoepidemiology. In such studies, the parameter of interest is a contrast (difference or ratio) in the outcome risks between the treatment of interest and the selected active comparator. While it may appear treatment is dichotomous, treatment is actually polytomous as there are at least 3 levels: no treatment, the treatment of interest, and the active comparator. Because misclassification may occur between any of these groups, independent nondifferential treatment misclassification may not be toward the null (as expected with a dichotomous treatment). In this work, we describe bias from independent nondifferential treatment misclassification in active comparator studies with a focus on misclassification that occurs between each active treatment and no treatment. We derive equations for bias in the estimated outcome risks, risk difference, and risk ratio, and we provide bias correction equations that produce unbiased estimates, in expectation. Using data obtained from US insurance claims data, we present a hypothetical comparative safety study of antibiotic treatment to illustrate factors that influence bias and provide an example probabilistic bias analysis using our derived bias correction equations.

Misconceptions About the Direction of Bias From Nondifferential Misclassification.

Measurement error is pervasive in epidemiologic research. Epidemiologists often assume that mismeasurement of study variables is nondifferential with respect to other analytical variables and then rely on the heuristic that "nondifferential misclassification will bias estimates towards the null." However, there are many exceptions to the heuristic for which bias towards the null cannot be assumed. In this paper, we compile and characterize 7 exceptions to this rule and encourage analysts to take a more critical and nuanced approach to evaluating and discussing bias from nondifferential mismeasurement.

Variability of biomarkers used for the classification of metabolic syndrome: A repeated measurements study.

BACKGROUND AND AIMS: The definition of the metabolic syndrome consists of five components. The underlying measurements are subject to intra-individual variability. This repeated measurements study investigated the impact of intra-individual measurement variability on the stability of the diagnosis of metabolic syndrome over 12 months. METHODS AND RESULTS: Twenty-five employees of the University Medicine Greifswald aged 22-70 years were examined once a month over one year. Examinations included blood sampling and anthropometric and blood pressure measurements. Laboratory measurements included glucose, cholesterol (high-density lipoprotein [HDL], and low-density lipoprotein [LDL]), and triglycerides. The metabolic syndrome was defined according to the International Diabetes Federation modified for non-fasting blood samples. Variations in continuous metabolic markers were assessed using coefficients of variation (CV) and intra-class correlation coefficients (ICC). Overall eight participants (32%) were categorized at least once within 12 months as having a metabolic syndrome; in none of those metabolic syndrome was found consistently over the study follow-ups. The Cohen's Kappa for metabolic syndrome was 0.57. CV was highest for triglycerides (27.5%) followed by glucose (10.1%), LDL- (9.5%), and HDL-cholesterol (8.6%). ICC's were lowest for glucose (0.51), triglycerides (0.65), systolic (0.68), and diastolic blood pressure (0.69). CONCLUSION: We showed that the measurement of biomarkers defining the metabolic syndrome is a time-varying condition with implications for the concept of the metabolic syndrome. To account for this uncertainty in prevalence studies we propose to identify uncertain cases according to the current definition of the metabolic syndrome. For analysing associations we recommend to apply probabilistic sensitivity analyses.

Smoking intensity and urinary nicotine metabolites by socioeconomic status in the Heinz Nixdorf Recall study.

BACKGROUND: Smoking intensity, which is generally based on self-reported average cigarettes per day (CPD), is a major behavioural risk factor and strongly related to socioeconomic status (SES). To assess the validity of the CPD measure, correlations with objective markers of tobacco smoke exposure - such as urinary nicotine metabolites - were examined. Yet, it remains unclear, whether this correlation is affected by SES, which may indicate imprecise or biased self-reports of smoking intensity. METHODS: We investigated the role of SES in the association between CPD and nicotine metabolites in current smokers among the participants of the population-based, prospective Heinz Nixdorf Recall Study. We determined urinary cotinine and additionally trans-3'-hydroxy-cotinine. SES was assessed by the International Socio-Economic Index of occupational status, and education. We calculated correlations (Pearson's r) between logarithmised CPD and cotinine in subgroups of SES and analysed SES and further predictors of cotinine in multiple linear regression models separately by gender. RESULTS: Median reported smoking intensity was 20 CPD in male and 19 CPD in female smokers. Men showed higher cotinine concentrations (median 3652 µg/L, interquartile range (IQR) 2279-5422 µg/L) than women (3127 µg/L, IQR 1692-4920 µg/L). Logarithmised CPD correlated moderately with cotinine in both, men and women (Pearson's r 0.4), but correlations were weaker in smokers with lower SES: Pearson's r for low, intermediate, and high occupational SES was 0.35, 0.39, and 0.48 in men, and 0.28, 0.43, and 0.47 in women, respectively. Logarithmised CPD and urinary creatinine were main predictors of cotinine in multiple regression models, whereas SES showed a weak negative association in women. Results were similar for trans-3'-hydroxy-cotinine. CONCLUSIONS: Decreasing precision of self-reported CPD was indicated for low SES in men and women. We found no strong evidence for biased self-reports of smoking intensity by SES.

Validation of cardiovascular outcomes and risk factors in the Clinical Practice Research Datalink in the United Kingdom.

PURPOSE: Strategies to identify and validate acute myocardial infarction (AMI) and stroke in primary-care electronic records may impact effect measures, but to an unknown extent. Additionally, the validity of cardiovascular risk factors that could act as confounders in studies on those endpoints has not been thoroughly assessed in the United Kingdom Clinical Practice Research Datalink's (CPRD's) GOLD database. We explored the validity of algorithms to identify cardiovascular outcomes and risk factors and evaluated different outcome-identification strategies using these algorithms for estimation of adjusted incidence rate ratios (IRRs). METHODS: First, we identified AMI, stroke, smoking, obesity, and menopausal status in a cohort treated for overactive bladder by applying computerized algorithms to primary care medical records (2004-2012). We validated these cardiovascular outcomes and risk factors with physician questionnaires (gold standard for this analysis). Second, we estimated IRRs for AMI and stroke using algorithm-identified and questionnaire-confirmed cases, comparing these with IRRs from cases identified through linkage with hospitalization/mortality data (best estimate). RESULTS: For AMI, the algorithm's positive predictive value (PPV) was >90%. Initial algorithms for stroke performed less well because of inclusion of codes for prevalent stroke; algorithm refinement increased PPV to 80% but decreased sensitivity by 20%. Algorithms for smoking and obesity were considered valid. IRRs based on questionnaire-confirmed cases only were closer to IRRs estimated from hospitalization/mortality data than IRRs from algorithm-identified cases. CONCLUSIONS: AMI, stroke, smoking, obesity, and postmenopausal status can be accurately identified in CPRD. Physician questionnaire-validated AMI and stroke cases yield IRRs closest to the best estimate.

Effects of social interactions and information bias on the willingness to pay for transboundary basin ecosystem services.

Payments for watershed ecosystem services are the most important forms of global water environmental protection. Transboundary basin ecological compensation policies in China are mainly based on the central government's appropriation to local governments or transfer payments between local governments. However, watershed ecosystem services face many problems such as the lack of interprovincial horizontal compensation policies and insufficient public participation. Most of China's rivers are distributed in vast rural areas, and the livelihoods of farmers living in these areas are highly dependent on the water environment. Since a watershed usually spans multiple administrative regions, the inconsistency between the natural and administrative boundaries of the river affects the completeness of the ecosystem services' information exchange between the service providers and payers. To promote interprovincial government water management cooperation and spark the farmers' enthusiasm for participating in the payments for watershed ecosystem services, this study examines the mechanism by which social interactions can affect farmers' willingness to pay (WTP) by mitigating the information bias. The results show that information bias plays a mediating role in the effect of social interactions on WTP. Additionally, the cadres/associations' and village-level interactions can effectively reduce the information bias of farmers, thereby increasing their WTP for transboundary basin ecosystem services. Moreover, the intensity of the psychological ownership of the watershed and government credibility have a significant moderating effect on the above-mentioned mechanisms. This study suggests that it is necessary to broaden the source channels of farmers' information on upstream ecological governance, improve the completeness of farmers' information, and curb the negative impact of information bias on WTP. Simultaneously, it is necessary to improve the government credibility and cultivate the farmers' sense of belonging and responsibility toward the watershed.

A semiparametric model for wearable sensor-based physical activity monitoring data with informative device wear.

Wearable sensors provide an exceptional opportunity in collecting real-time behavioral data in free living conditions. However, wearable sensor data from observational studies often suffer from information bias, since participants' willingness to wear the monitoring devices may be associated with the underlying behavior of interest. The aim of this study was to introduce a semiparametric statistical approach for modeling wearable sensor-based physical activity monitoring data with informative device wear. Our simulation study indicated that estimates from the generalized estimating equations showed ignorable bias when device wear patterns were independent of the participants physical activity process, but incrementally more biased when the patterns of device non-wear times were increasingly associated with the physical activity process. The estimates from the proposed semiparametric modeling approach were unbiased both when the device wear patterns were (i) independent or (ii) dependent to the underlying physical activity process. We demonstrate an application of this method using data from the 2003-2004 National Health and Nutrition Examination Survey ($N=4518$), to examine gender differences in physical activity measured using accelerometers. The semiparametric model can be implemented using our R package acc, free software developed for reading, processing, simulating, visualizing, and analyzing accelerometer data, publicly available at the Comprehensive R Archive Network.

Maternal effects impact decision-making in a viviparous lizard.

Stressful conditions experienced during early development can have deleterious effects on offspring morphology, physiology and behaviour. However, few studies have examined how developmental stress influences an individual's cognitive phenotype. Using a viviparous lizard, we show that the availability of food resources to a mother during gestation influences a key component of her offspring's cognitive phenotype: their decision-making. Offspring from females who experienced low resource availability during gestation did better in an anti-predatory task that relied on spatial associations to guide their decisions, whereas offspring from females who experienced high resource availability during gestation did better in a foraging task that relied on colour associations to inform their decisions. This shows that the prenatal environment can influence decision-making in animals, a cognitive trait with functional implications later in life.

Information bias in epidemiological studies with a special focus on obstetrics and gynecology.

Information bias occurs when any information used in a study is either measured or recorded inaccurately. This paper describes some of the most common types of information bias, using examples from obstetrics and gynecology, and describes how information bias may affect results of observational studies. Non-differential misclassification occurs when the degree of misclassification of exposure status among those with and those without the disease is the same; in cohort studies, this type of bias is most likely and will bias estimates toward no association when exposure is dichotomized. Non-differential underreporting of an exposure with more than two categories may mask a true threshold effect as a dose-response relation and, if a true threshold effect exists, the threshold will be set at too low a level, if the exposure is underreported. Differential misclassification may cause bias in either direction and is particularly likely, when exposure status is reported after the outcome occurred. Misclassification of confounders is an issue that needs special attention by researchers, as failure to measure accurately one or more (strong) confounders may seriously bias the observed results. Misclassification of disease status may also cause bias of estimates of association in either direction. Information bias is probably best prevented during planning of data collection, as there are few and insufficient methods available for correcting inaccurate information.

Information bias of social gradients in sickness absence: a comparison of self-report data in the Norwegian Mother and Child Cohort Study (MoBa) and data in national registries.

BACKGROUND: Measurement error in self-report questionnaires is a common source of bias in epidemiologic studies. The study aim was to assess information bias of the educational gradient in sickness absence among participants in the Norwegian Mother and Child Cohort Study (MoBa), comparing self-report data with national register data. METHODS: MoBa is a national prospective cohort study. The present study included 49,637 participants, born 1967-1976, who gave birth 2000-2009. The highest completed education level was recorded in categories and as educational years. Sickness absence was defined as one or more spell lasting more than 16 days between pregnancy weeks 13 and 30. We computed sickness absence risk in mid-pregnancy in strata of education level. Associations between completed educational years and sickness absence were estimated as risk differences in binomial regression and compared between self-report and register data. In additional analyses, we aimed to explain discrepancies between estimates from the two data sources. RESULTS: The overall registry-based sickness absence risk was 0.478 and decreased for increasingly higher education in a consistent fashion, yielding an additive risk difference in association with one additional education year of - 0.032 (95% confidence interval - 0.035 to - 0.030). The self-report risk was lower (0.307) with a corresponding risk difference of only - 0.013 (95% confidence interval - 0.015 to - 0.011). The main explanation of the lower risk difference in the self-report data was a tendency for mothers in low education categories to omit reporting sickness absence in the questionnaire. CONCLUSIONS: A plausible explanation for the biased self-report association is complexity of the sickness absence question and a resulting educational gradient in non-response. As shown for sickness absence in mid-pregnancy in the present study, national registries could be a preferred alternative to self-report questionnaires.

The health impacts of semiconductor production: an epidemiologic review.

BACKGROUND: Despite concerns over the harmful health effects of semiconductor production, epidemiological studies have shown mixed results. OBJECTIVES: We aim to critically appraise epidemiologic studies to date, and to suggest future research and actions to protect workers in semiconductor industry. METHODS: Epidemiologic studies were identified through electronic database searches, review of reference lists of relevant published works, and expert consultations, and were narratively reviewed. RESULTS: Most evidence suggests reproductive risks from fabrication jobs, including spontaneous abortion (SAB), congenital malformation, and reduced fertility. Although chemicals have been suspected as causal agents, knowledge of the likely contribution(s) from specific exposures is still limited. Evidence of cancer risk seems to be equivocal. However, the available studies had serious limitations including healthy worker effects (HWEs), information bias, and insufficient power, all of which are associated with underestimation. Nevertheless, excess risks for non-Hodgkin's lymphoma (NHL), leukemia, brain tumor, and breast cancer were observed. CONCLUSIONS: Monitoring and innovative research based on international collaboration with a focus on sentinel events are required.

Quantitative bias analysis methods for summary level epidemiologic data in the peer-reviewed literature: a systematic review.

OBJECTIVE: Quantitative bias analysis (QBA) methods evaluate the impact of biases arising from systematic errors on observational study results. This systematic review aimed to summarize the range and characteristics of quantitative bias analysis (QBA) methods for summary level data published in the peer-reviewed literature. STUDY DESIGN AND SETTING: We searched MEDLINE, Embase, Scopus, and Web of Science for English-language articles describing QBA methods. For each QBA method, we recorded key characteristics, including applicable study designs, bias(es) addressed; bias parameters, and publicly available software. The study protocol was pre-registered on the Open Science Framework (https://osf.io/ue6vm/). RESULTS: Our search identified 10,249 records, of which 53 were articles describing 57 QBA methods for summary level data. Of the 57 QBA methods, 53 (93%) were explicitly designed for observational studies, and 4 (7%) for meta-analyses. There were 29 (51%) QBA methods that addressed unmeasured confounding, 19 (33%) misclassification bias, 6 (11%) selection bias, and 3 (5%) multiple biases. 38 (67%) QBA methods were designed to generate bias-adjusted effect estimates and 18 (32%) were designed to describe how bias could explain away observed findings. 22 (39%) articles provided code or online tools to implement the QBA methods. CONCLUSION: In this systematic review, we identified a total of 57 QBA methods for summary level epidemiologic data published in the peer-reviewed literature. Future investigators can use this systematic review to identify different QBA methods for summary level epidemiologic data.

Informed consent for national registration of COVID-19 vaccination caused information bias of vaccine effectiveness estimates mostly in older adults: a bias correction study.

OBJECTIVES: Registration in the Dutch national COVID-19 vaccination register requires consent from the vaccinee. This causes misclassification of nonconsenting vaccinated persons as being unvaccinated. We quantified and corrected the resulting information bias in vaccine effectiveness (VE) estimates. STUDY DESIGN AND SETTING: National data were used for the period dominated by the SARS-CoV-2 Delta variant (July 11 to November 15, 2021). VE ((1-relative risk)∗100%) against COVID-19 hospitalization and intensive care unit (ICU) admission was estimated for individuals 12 to 49, 50 to 69, and ≥70 years of age using negative binomial regression. Anonymous data on vaccinations administered by the Municipal Health Services were used to determine informed consent percentages and estimate corrected VEs by iteratively imputing corrected vaccination status. Absolute bias was calculated as the absolute change in VE; relative bias as uncorrected/corrected relative risk. RESULTS: A total of 8804 COVID-19 hospitalizations and 1692 COVID-19 ICU admissions were observed. The bias was largest in the 70+ age group where the nonconsent proportion was 7.0% and observed vaccination coverage was 87%: VE of primary vaccination against hospitalization changed from 75.5% (95% CI 73.5-77.4) before to 85.9% (95% CI 84.7-87.1) after correction (absolute bias -10.4 percentage point, relative bias 1.74). VE against ICU admission in this group was 88.7% (95% CI 86.2-90.8) before and 93.7% (95% CI 92.2-94.9) after correction (absolute bias -5.0 percentage point, relative bias 1.79). CONCLUSION: VE estimates can be substantially biased with modest nonconsent percentages for vaccination data registration. Data on covariate-specific nonconsent percentages should be available to correct this bias.

Does recall bias explain the association of mood disorders with workplace harassment?

Purpose: To determine the contribution of recall bias to the observed excess in mental ill-health in those reporting harassment at work. Methods: A prospective cohort of 1885 workers in welding and electrical trades was contacted every six months for up to 5 years, asking whether they were currently anxious or depressed and whether this was made worse by work. Only at the end of the study did we ask about any workplace harassment they had experienced at work. We elicited sensitivity and specificity of self-reported bullying from published reliability studies and formulated priors that reflect the possibility of over-reporting of workplace harassment (exposure) by those whose anxiety or depression was reported to be made worse by work (cases). We applied the resulting misclassification models to probabilistic bias analysis (PBA) of relative risks. Results: We observe that PBA implies that it is unlikely that biased misclassification due to the study subjects' states of mind could have caused the entire observed association. Indeed, the results demonstrated that doubling of risk of anxiety or depression following workplace harassment is plausible, with the unadjusted relative risk attenuated with understated uncertainty. Conclusions: It seems unlikely that risk of anxiety or depression following workplace harassment can be explained by the form of recall bias that we proposed.

Simultaneous adjustment of uncontrolled confounding, selection bias and misclassification in multiple-bias modelling.

BACKGROUND: Adjusting for multiple biases usually involves adjusting for one bias at a time, with careful attention to the order in which these biases are adjusted. A novel, alternative approach to multiple-bias adjustment involves the simultaneous adjustment of all biases via imputation and/or regression weighting. The imputed value or weight corresponds to the probability of the missing data and serves to 'reconstruct' the unbiased data that would be observed based on the provided assumptions of the degree of bias. METHODS: We motivate and describe the steps necessary to implement this method. We also demonstrate the validity of this method through a simulation study with an exposure-outcome relationship that is biased by uncontrolled confounding, exposure misclassification, and selection bias. RESULTS: The study revealed that a non-biased effect estimate can be obtained when correct bias parameters are applied. It also found that incorrect specification of every bias parameter by +/-25% still produced an effect estimate with less bias than the observed, biased effect. CONCLUSIONS: Simultaneous multi-bias analysis is a useful way of investigating and understanding how multiple sources of bias may affect naive effect estimates. This new method can be used to enhance the validity and transparency of real-world evidence obtained from observational, longitudinal studies.

An algorithm to predict data completeness in oncology electronic medical records for comparative effectiveness research.

INTRODUCTION: Electronic health record (EHR) discontinuity (missing out-of-network encounters) can lead to information bias. We sought to construct an algorithm that identifies high EHR-continuity among oncology patients. METHODS: Using a linked Medicare-EHR database and regression, we sought to 1) measure how often Medicare claims for outpatient encounters were substantiated by visits recorded in the EHR, and 2) predict continuity ratio, defined as the yearly proportion of outpatient encounters reported to Medicare that were captured by EHR data. The prediction model...s performance was evaluated with the coefficient of determination and Spearman...s correlation. We quantified variable misclassification by decile of continuity ratio using standardized difference and sensitivity. RESULTS: A total of 79,678 subjects met all eligibility criteria. Predicted and observed continuity was highly correlated (σSpearman=0.86). On average across all variables measured, MSD was reduced by a factor of 1/7th and sensitivity was improved 35-fold comparing subjects in the highest vs. lowest decile of CR. CONCLUSION: In the oncology population, restricting EHR-based study cohorts to subjects with high continuity may reduce misclassification without greatly impacting representativeness. Further work is needed to elucidate the best manner of implementing continuity prediction rules in cohort studies.

Advancing an Algorithm for the Identification of Patients with High Data-Continuity in Electronic Health Records.

Background: Identifying high data-continuity patients in an electronic health record (EHR) system may facilitate selecting cohorts with a lower degree of variable misclassification and promote study validity. We updated a previously developed algorithm for identifying patients with high EHR data-completeness by adding demographic and health utilization factors to improve adaptability to networks serving patients of diverse backgrounds. We also expanded the algorithm to accommodate data in the ICD-10 era. Methods: We used Medicare claims linked with EHR data to identify individuals aged ≥65 years. EHR-continuity was defined as the proportion of encounters captured in EHR data relative to claims. We compared the model with additional demographic factors and their interaction terms with other predictors with the original algorithm and assessed the performance by area under the ROC curve (AUC) and net reclassification index (NRI). Results: The study cohort consisted of 264,099 subjects. The updated prediction model had an AUC of 0.93 in the validation set. Compared to the previous model, the new model had an NRI of 37.4% (p<0.001) for EHR-continuity classification. Interaction terms between demographic variables and other predictors did not improve the performance. Patients within the top 20% of predicted EHR-continuity had four times less misclassification of key variables compared to the remaining population. Conclusion: Adding demographic and healthcare utilization variables significantly improved the model performance. Patients with high predicted EHR-continuity had less misclassification of study variables compared to the remaining population in both ICD-9 and 10 eras.