ABSTRACT
The self-controlled case-series (SCCS) research design is increasingly used in pharmacoepidemiologic studies of drug-drug interactions (DDIs), with the target of inference being the incidence rate ratio (IRR) associated with concomitant exposure to the object plus precipitant drug versus the object drug alone. While day-level drug exposure can be inferred from dispensing claims, these inferences may be inaccurate, leading to biased IRRs. Grace periods (periods assuming continued treatment impact after days' supply exhaustion) are frequently used by researchers, but the impact of grace period decisions on bias from exposure misclassification remains unclear. Motivated by an SCCS study examining the potential DDI between clopidogrel (object) and warfarin (precipitant), we investigated bias due to precipitant or object exposure misclassification using simulations. We show that misclassified precipitant treatment always biases the estimated IRR toward the null, whereas misclassified object treatment may lead to bias in either direction or no bias, depending on the scenario. Further, including a grace period for each object dispensing may unintentionally increase the risk of misclassification bias. To minimize such bias, we recommend 1) avoiding the use of grace periods when specifying object drug exposure episodes; and 2) including a washout period following each precipitant exposed period.
ABSTRACT
OBJECTIVES: We assessed reporting misclassification for 12 critical congenital heart defects (CCHDs) identified through administrative diagnosis codes within a passive surveillance system. We measured the effect of misclassification on prevalence estimation. Lastly, we investigated a sample-based review strategy to estimate surveillance misclassification resulting from administrative diagnosis codes for case detection. METHODS: We received 419 reports of CCHDs between 2007 and 2018; 414 were clinically reviewed. We calculated confirmation probabilities to assess misclassification and adjust prevalence estimates. Random samples of reported cases were taken at proportions between 20% and 90% for each condition to assess sample bias. Sampling was repeated 1000 times to measure sample-estimate variability. RESULTS: Misclassification ranged from a low of 19% (n = 4/21) to a high of 84% (n = 21/25). Unconfirmed prevalence rates ranged between one and six cases per 10,000 live births, with some conditions significantly higher than national estimates. However, confirmed rates were either lower or comparable to national estimates. CONCLUSION: Passive birth defect surveillance programs that rely on administrative diagnosis codes for case identification of CCHDs are subject to misclassification that bias prevalence estimates. We showed that a sample-based review could improve the prevalence estimates of 12 cardiovascular conditions relative to their unconfirmed prevalence rates.
Subject(s)
Heart Defects, Congenital , Humans , Heart Defects, Congenital/epidemiology , Prevalence , Population Surveillance/methods , Bias , Female , Male , Infant, NewbornABSTRACT
Stool has multiple components, which include undigested food material, plant, animal products, normal intestinal microbiome, and parasites. Due to the existence of all the elements, stool parasite examination is cumbersome, especially with identification of the eggs of Ascaris lumbricoides. We examined 650 stool samples of pregnant women before anti-helminthic treatment. We found that the prevalence of Ascaris lumbricoides was 5.4% (95% CI 3.8-7.4, n = 35) by a single observer in microscopy, and the majority (33/35) were identified as decorticated fertilized eggs. The prevalence of Ascaris by molecular methods was 2.6% (95% CI 1.5-4.2%, n = 17). Five samples were positive by both methods. The prevalence of structures resembling Ascaris was 4.6% (95% CI 3.1-6.5, n = 30). Three of the positive samples were confirmed with sequencing. With the subjective nature of microscopy along with the naked eye examination, errors can happen. Hence adequate training and confirmation with molecular techniques for identification of Ascaris lumbricoides are advisable.
ABSTRACT
OBJECTIVE: Phenotypic misclassification in genetic association analyses can impact the accuracy of PRS-based prediction models. The bias reduction method proposed by Tong et al. (2019) has demonstrated its efficacy in reducing the effects of bias on the estimation of association parameters between genotype and phenotype while minimizing variance by employing chart reviews on a subset of the data for validating phenotypes, however its improvement of subsequent PRS prediction accuracy remains unclear. Our study aims to fill this gap by assessing the performance of simulated PRS models and estimating the optimal number of chart reviews needed for validation. METHODS: To comprehensively assess the efficacy of the bias reduction method proposed by Tong et al. in enhancing the accuracy of PRS-based prediction models, we simulated each phenotype under different correlation structures (an independent model, a weakly correlated model, a strongly correlated model) and introduced error-prone phenotypes using two distinct error mechanisms (differential and non-differential phenotyping errors). To facilitate this, we used genotype and phenotype data from 12 case-control datasets in the Alzheimer's Disease Genetics Consortium (ADGC) to produce simulated phenotypes. The evaluation included analyses across various misclassification rates of original phenotypes as well as quantities of validation set. Additionally, we determined the median threshold, identifying the minimal validation size required for a meaningful improvement in the accuracy of PRS-based predictions across a broad spectrum. RESULTS: This simulation study demonstrated that incorporating chart review does not universally guarantee enhanced performance of PRS-based prediction models. Specifically, in scenarios with minimal misclassification rates and limited validation sizes, PRS models utilizing debiased regression coefficients demonstrated inferior predictive capabilities compared to models using error-prone phenotypes. Put differently, the effectiveness of the bias reduction method is contingent upon the misclassification rates of phenotypes and the size of the validation set employed during chart reviews. Notably, when dealing with datasets featuring higher misclassification rates, the advantages of utilizing this bias reduction method become more evident, requiring a smaller validation set to achieve better performance. CONCLUSION: This study highlights the importance of choosing an appropriate validation set size to balance between the efforts of chart review and the gain in PRS prediction accuracy. Consequently, our study establishes a valuable guidance for validation planning, across a diverse array of sensitivity and specificity combinations.
ABSTRACT
The test-negative design (TND) is a popular method for evaluating vaccine effectiveness (VE). A "classical" TND study includes symptomatic individuals tested for the disease targeted by the vaccine to estimate VE against symptomatic infection. However, recent applications of the TND have attempted to estimate VE against infection by including all tested individuals, regardless of their symptoms. In this article, we use directed acyclic graphs and simulations to investigate potential biases in TND studies of COVID-19 VE arising from the use of this "alternative" approach, particularly when applied during periods of widespread testing. We show that the inclusion of asymptomatic individuals can potentially lead to collider stratification bias, uncontrolled confounding by health and healthcare-seeking behaviors (HSBs), and differential outcome misclassification. While our focus is on the COVID-19 setting, the issues discussed here may also be relevant in the context of other infectious diseases. This may be particularly true in scenarios where there is either a high baseline prevalence of infection, a strong correlation between HSBs and vaccination, different testing practices for vaccinated and unvaccinated individuals, or settings where both the vaccine under study attenuates symptoms of infection and diagnostic accuracy is modified by the presence of symptoms.
ABSTRACT
Mortality statistics are critical to determine the burden of disease. Certain causes of death are prone to being misclassified on cause of death certificates. This poses a serious risk for public health and safety, as accurate death certificates form the basis for mortality statistics, which in turn are crucial for research, funding allocation and health interventions. This study uses generalised estimating equations and regression modelling to investigate for which cause of death categories suicide and accident deaths are misclassified as. National mortality statistics and autopsy rates from North America and Europe covering the past forty years were analysed to determine the associations between the different causes of death in cross-sectional and longitudinal models. We find that suicides and deaths by accidents are frequently mutually misclassified. We also find that suicides are frequently misclassified as drug use disorder deaths, in contrast to accident deaths, which are not misclassified as drug use disorder deaths. Furthermore, suicides do not seem to be misclassified as undetermined deaths or ill-defined deaths. The frequency of misclassification shows that the quality of death certificates should be improved, and autopsies may be used systematically to control the quality of death certificates.
ABSTRACT
Gender is an observed effect modifier of the association between loneliness and memory aging. However, this effect modification may be a result of information bias due to differential loneliness under-reporting by gender. We applied probabilistic bias analyses to examine whether effect modification of the loneliness-memory decline relationship by gender is retained under three simulation scenarios with various magnitudes of differential loneliness under-reporting between men and women. Data were from biennial interviews with adults aged 50+ in the US Health and Retirement Study from 1996-2016 (5,646 women and 3,386 men). Loneliness status (yes vs. no) was measured from 1996-2004 using the CES-D loneliness item and memory was measured from 2004-2016. Simulated sensitivity and specificity of the loneliness measure were informed by a validation study using the UCLA Loneliness Scale as a gold standard. The likelihood of observing effect modification by gender was higher than 90% in all simulations, although the likelihood reduced with an increasing difference in magnitude of the loneliness under-reporting between men and women. The gender difference in loneliness under-reporting did not meaningfully affect the observed effect modification by gender in our simulations. Our simulation approach may be promising to quantify potential information bias in effect modification analyses.
ABSTRACT
Introduction: Misclassification of American Indian and Alaska Native (AI/AN) peoples exists across various databases in research and clinical practice. Oral health is associated with cancer incidence and survival; however, misclassification adds another layer of complexity to understanding the impact of poor oral health. The objective of this literature review was to systematically evaluate and analyze publications focused on racial misclassification of AI/AN racial identities among cancer surveillance data. Methods: The PRISMA Statement and the CONSIDER Statement were used for this systematic literature review. Studies involving the racial misclassification of AI/AN identity among cancer surveillance data were screened for eligibility. Data were analyzed in terms of the discussion of racial misclassification, methods to reduce this error, and the reporting of research involving Indigenous peoples. Results: A total of 66 articles were included with publication years ranging from 1972 to 2022. A total of 55 (83%) of the 66 articles discussed racial misclassification. The most common method of addressing racial misclassification among these articles was linkage with the Indian Health Service or tribal clinic records (45 articles or 82%). The average number of CONSIDER checklist domains was three, with a range of zero to eight domains included. The domain most often identified was Prioritization (60), followed by Governance (47), Methodologies (31), Dissemination (27), Relationships (22), Participation (9), Capacity (9), and Analysis and Findings (8). Conclusion: To ensure equitable representation of AI/AN communities, and thwart further oppression of minorities, specifically AI/AN peoples, is through accurate data collection and reporting processes.
ABSTRACT
The problems of point estimation and classification under the assumption that the training data follow a Lindley distribution are considered. Bayes estimators are derived for the parameter of the Lindley distribution applying the Markov chain Monte Carlo (MCMC), and Tierney and Kadane's [Tierney and Kadane, Accurate approximations for posterior moments and marginal densities, J. Amer. Statist. Assoc. 81 (1986), pp. 82-86] methods. In the sequel, we prove that the Bayes estimators using Tierney and Kadane's approximation and Lindley's approximation both converge to the maximum likelihood estimator (MLE), as n â ∞ , where n is the sample size. The performances of all the proposed estimators are compared with some of the existing ones using bias and mean squared error (MSE), numerically. It has been noticed from our simulation study that the proposed estimators perform better than some of the existing ones. Applying these estimators, we construct several plug-in type classification rules and a rule that uses the likelihood accordance function. The performances of each of the rules are numerically evaluated using the expected probability of misclassification (EPM). Two real-life examples related to COVID-19 disease are considered for illustrative purposes.
ABSTRACT
OBJECTIVES: Registration in the Dutch national COVID-19 vaccination register requires consent from the vaccinee. This causes misclassification of nonconsenting vaccinated persons as being unvaccinated. We quantified and corrected the resulting information bias in vaccine effectiveness (VE) estimates. STUDY DESIGN AND SETTING: National data were used for the period dominated by the SARS-CoV-2 Delta variant (July 11 to November 15, 2021). VE ((1-relative risk)∗100%) against COVID-19 hospitalization and intensive care unit (ICU) admission was estimated for individuals 12 to 49, 50 to 69, and ≥70 years of age using negative binomial regression. Anonymous data on vaccinations administered by the Municipal Health Services were used to determine informed consent percentages and estimate corrected VEs by iteratively imputing corrected vaccination status. Absolute bias was calculated as the absolute change in VE; relative bias as uncorrected/corrected relative risk. RESULTS: A total of 8804 COVID-19 hospitalizations and 1692 COVID-19 ICU admissions were observed. The bias was largest in the 70+ age group where the nonconsent proportion was 7.0% and observed vaccination coverage was 87%: VE of primary vaccination against hospitalization changed from 75.5% (95% CI 73.5-77.4) before to 85.9% (95% CI 84.7-87.1) after correction (absolute bias -10.4 percentage point, relative bias 1.74). VE against ICU admission in this group was 88.7% (95% CI 86.2-90.8) before and 93.7% (95% CI 92.2-94.9) after correction (absolute bias -5.0 percentage point, relative bias 1.79). CONCLUSION: VE estimates can be substantially biased with modest nonconsent percentages for vaccination data registration. Data on covariate-specific nonconsent percentages should be available to correct this bias.
ABSTRACT
Inappropriate therapy is a frequent adverse consequence of implantable cardioverter-defibrillator. Inappropriate therapy often occurs due to the misinterpretation of sinus tachycardia or atrial fibrillation/flutter with rapid atrioventricular conduction by the device. Current implantable cardioverter-defibrillator (ICD) mechanisms integrate various discriminators into algorithms to differentiate supraventricular tachycardia (SVT) from ventricular tachycardia (VT), to prevent such occurrences. A 40-year-old man suffered seizures and cardiac arrest abruptly, without prior complaints of chest pain. Without delay, he initiated cardiopulmonary resuscitation (CPR), resulting in the regaining of spontaneous circulation. The patient had previously received a single-chamber ICD due to recurring VT and a prior episode of cardiac arrest. The patient had a medical background of coronary artery disease with complete revascularisation and no previous occurrence of SVT. Interrogating the ICD revealed captured non-sustained ventricular tachycardia (NSVT) and SVT events but no VT episode or shock therapy. During the specified time period, the patient underwent an electrophysiological study, and no SVT was induced with the normal function of the atrioventricular and sinoatrial nodes. Various causes can lead to errors in morphology discrimination criteria in single-chamber ICDs. Extending the detection interval is highly recommended to avoid misclassification of ICDs. LEARNING POINTS: This highlights the crucial significance of precise classification of supraventricular tachycardia (SVT) and ventricular tachycardia (VT) using a single-chamber implantable cardioverter-defibrillator (ICD) discriminator to guarantee prompt and appropriate therapy delivery.The morphology criterion used in single-chamber ICDs may have potential limits and inaccuracies, which might result in the misdiagnosis of VT as SVT.Further study and enhancement of differentiation algorithms, paired with precise programming and prolonged detection durations are essential to reduce such misclassifications and improve patient outcomes.
ABSTRACT
BACKGROUND: Several measures of occupational exposure to pesticides have been used to study associations between exposure to pesticides and neurobehavioral outcomes. This study assessed the impact of different exposure measures for glyphosate and mancozeb on the association with neurobehavioral outcomes based on original and recalled self-reported data with 246 smallholder farmers in Uganda. METHODS: The association between the 6 exposure measures and 6 selected neurobehavioral test scores was investigated using linear multivariable regression models. Exposure measures included original exposure measures for the previous year in 2017: (i) application status (yes/no), (ii) number of application days, (iii) average exposure-intensity scores (EIS) of an application and (iv) number of EIS-weighted application days. Two additional measures were collected in 2019: (v) recalled application status and (vi) recalled EIS for the respective periods in 2017. RESULTS: Recalled applicator status and EIS were between 1.2 and 1.4 times more frequent and higher for both pesticides than the original application status and EIS. Adverse associations between the different original measures of exposure to glyphosate and 4 neurobehavioral tests were observed. Glyphosate exposure based on recalled information and all mancozeb exposure measures were not associated with the neurobehavioral outcomes. CONCLUSIONS: The relation between the different original self-reported glyphosate exposure measures and neurobehavioral test scores appeared to be robust. When based on recalled exposure measures, associations observed with the original exposure measures were no longer present. Therefore, future epidemiological studies on self-reported exposure should critically evaluate the potential bias towards the null in observed exposure-response associations.
Subject(s)
Glycine , Glyphosate , Occupational Exposure , Pesticides , Zineb , Humans , Occupational Exposure/adverse effects , Occupational Exposure/analysis , Pesticides/adverse effects , Male , Adult , Female , Glycine/analogs & derivatives , Glycine/adverse effects , Uganda , Farmers , Maneb , Middle Aged , Neuropsychological Tests/statistics & numerical data , Self ReportABSTRACT
We spend a great deal of time on confounding in our teaching, in our methods development and in our assessment of study results. This may give the impression that uncontrolled confounding is the biggest problem that observational epidemiology faces, when in fact, other sources of bias such as selection bias, measurement error, missing data, and misalignment of zero time may often (especially if they are all present in a single study) lead to a stronger deviation from the truth. Compared to the amount of time we spend teaching how to address confounding in a data analysis, we spend relatively little time teaching methods for simulating confounding (and other sources of bias) to learn their impact and develop plans to mitigate or quantify the bias. We review a paper by Desai et al that uses simulation methods to quantify the impact of an unmeasured confounder when it is completely missing or when a proxy of the confounder is measured. We use this article to discuss how we can use simulations of sources of bias to ensure we generate better and more valid study estimates, and we discuss the importance of simulating realistic datasets with plausible bias structures to guide data collection. If an advanced life form exists outside of our current universe and they came to earth with the goal of scouring the published epidemiologic literature to understand what the biggest problem epidemiologists have, they would quickly discover that the limitations section of publications would provide them with all the information they needed. And most likely what they would conclude is that the biggest problem that we face is uncontrolled confounding. It seems to be an obsession of ours.
ABSTRACT
Clinicians, researchers, regulators, and other decision-makers increasingly rely on evidence from real-world data (RWD), including data routinely accumulating in health and administrative databases. RWD studies often rely on algorithms to operationalize variable definitions. An algorithm is a combination of codes or concepts used to identify persons with a specific health condition or characteristic. Establishing the validity of algorithms is a prerequisite for generating valid study findings that can ultimately inform evidence-based health care. This paper aims to systematize terminology, methods, and practical considerations relevant to the conduct of validation studies of RWD-based algorithms. We discuss measures of algorithm accuracy; gold/reference standard; study size; prioritizing accuracy measures; algorithm portability; and implication for interpretation. Information bias is common in epidemiologic studies, underscoring the importance of transparency in decisions regarding choice and prioritizing measures of algorithm validity. The validity of an algorithm should be judged in the context of a data source, and one size does not fit all. Prioritizing validity measures within a given data source depends on the role of a given variable in the analysis (eligibility criterion, exposure, outcome or covariate). Validation work should be part of routine maintenance of RWD sources.
ABSTRACT
Purpose: To determine the contribution of recall bias to the observed excess in mental ill-health in those reporting harassment at work. Methods: A prospective cohort of 1885 workers in welding and electrical trades was contacted every six months for up to 5 years, asking whether they were currently anxious or depressed and whether this was made worse by work. Only at the end of the study did we ask about any workplace harassment they had experienced at work. We elicited sensitivity and specificity of self-reported bullying from published reliability studies and formulated priors that reflect the possibility of over-reporting of workplace harassment (exposure) by those whose anxiety or depression was reported to be made worse by work (cases). We applied the resulting misclassification models to probabilistic bias analysis (PBA) of relative risks. Results: We observe that PBA implies that it is unlikely that biased misclassification due to the study subjects' states of mind could have caused the entire observed association. Indeed, the results demonstrated that doubling of risk of anxiety or depression following workplace harassment is plausible, with the unadjusted relative risk attenuated with understated uncertainty. Conclusions: It seems unlikely that risk of anxiety or depression following workplace harassment can be explained by the form of recall bias that we proposed.
ABSTRACT
Postpartum haemorrhage (PPH) is a significant cause of maternal morbidity and mortality worldwide, particularly in low-resource settings. This study aimed to develop a predictive model for PPH using early risk factors and rank their importance in terms of predictive ability. The dataset was obtained from an observational case-control study in northern Rwanda. Various statistical models and machine learning techniques were evaluated, including logistic regression, logistic regression with elastic-net regularisation, Random Forests, Extremely Randomised Trees, and gradient-boosted trees with XGBoost. The Random Forest model, with an average sensitivity of 80.7%, specificity of 71.3%, and a misclassification rate of 12.19%, outperformed the other models, demonstrating its potential as a reliable tool for predicting PPH. The important predictors identified in this study were haemoglobin level during labour and maternal age. However, there were differences in PPH risk factor importance in different data partitions, highlighting the need for further investigation. These findings contribute to understanding PPH risk factors, highlight the importance of considering different data partitions and implementing cross-validation in predictive modelling, and emphasise the value of identifying the appropriate prediction model for the application. Effective PPH prediction models are essential for improving maternal health outcomes on a global scale. This study provides valuable insights for healthcare providers to develop predictive models for PPH to identify high-risk women and implement targeted interventions.
Subject(s)
Machine Learning , Models, Statistical , Postpartum Hemorrhage , Humans , Female , Postpartum Hemorrhage/epidemiology , Risk Factors , Adult , Case-Control Studies , Pregnancy , Rwanda/epidemiology , Young Adult , Logistic ModelsABSTRACT
BACKGROUND: The Kato-Katz method is a commonly used diagnostic tool for helminth infections, particularly in field studies. This method can yield inaccurate results when samples contain eggs that are similar in appearance, such as Minute Intestinal Fluke (MIF) and Opisthorchis viverrini (OV) eggs. The close resemblance of eggs can be problematic and raises the possibility of false diagnoses. The objectives were to compare the diagnostic performance of the Kato-Katz method for accurately identifying MIF and OV and to provide evidence of possible misclassification. Methods: Based on questionnaire responses from 15 (young parasitologists and public health staff), the test comprised 50 MIF egg images and 50 OV egg images, for a total of 100 Google Form questionnaires. RESULTS: The morphology of MIF and OV eggs found size and shape similarity and found that the shoulder rims were small, while the OV egg found the knobs had disappeared. The opercular conjunction was apparent, the shoulder rims and miricidium were prominent. The average percentage of correctly classified infections was 61.6 ± 12.1%. The accuracy percentages for both public health staff and young parasitologists in identifying were found to be 59.0 ± 14.8 and 66.8 ± 2.8, respectively. There was no significant difference observed in both groups. CONCLUSION: These findings highlight the need for improving the accuracy of parasite identification. Preserving stool samples before the Kato-Katz method can help mitigate the potential degradation or distortion of parasite eggs. The incorrect classification of both eggs had an impact on treatment plans and the policy of parasite control programs.
Subject(s)
Feces , Opisthorchiasis , Opisthorchis , Parasite Egg Count , Animals , Humans , Opisthorchiasis/parasitology , Feces/parasitology , Parasite Egg Count/methods , Ovum , Fasciola hepatica/isolation & purification , Surveys and QuestionnairesABSTRACT
The objective of this cross-sectional study was to estimate the validity of laboratory culture, Petrifilm and Tri-Plate on-farm culture systems, as well as luminometry to correctly identify IMI at dry-off in dairy cows, considering all tests to be imperfect. From September 2020 until December 2021, we collected composite milk samples from cows before dry-off and divided them into 4 aliquots for luminometry, Petrifilm (aerobic count), Tri-Plate, and laboratory culture tests. We assessed multiple thresholds of relative light units (RLU) for luminometry, and we used thresholds of ≥100 cfu/mL for the laboratory culture, ≥50 cfu/mL for Petrifilm, and ≥1 cfu for Tri-Plate tests. We fitted Bayesian latent class analysis models to estimate the sensitivity (Se) and specificity (Sp) for each test to identify IMI, with 95% credibility interval (BCI). Using different prevalence measures (0.30, 0.50, and 0.70), we calculated the predictive values (PV) and misclassification cost terms (MCT) at different false negative-to-false-positive ratios (FN:FP). A total of 333 cows were enrolled in the study from one commercial Holstein herd. The validity of the luminometry was poor for all thresholds, with an Se of 0.51 (95% BCI = 0.43-0.59) and Sp of 0.38 (95% BCI = 0.26-0.50) when using a threshold of ≥150 RLU. The laboratory culture had an Se of 0.93 (95% BCI = 0.85-0.98) and Sp of 0.69 (95% BCI = 0.49-0.89); the Petrifilm had an Se of 0.91 (95% BCI = 0.80-0.98) and Sp of 0.71 (95% BCI = 0.51-0.90); and the Tri-Plate had an Se of 0.65 (95% BCI = 0.53-0.82) and Sp of 0.85 (95% BCI = 0.66-0.97). Bacteriological tests had good PV, with comparable positive PV for all 3 tests, but lower negative PV for the Tri-Plate compared with the laboratory culture and the Petrifilm. For a prevalence of IMI of 0.30, all 3 tests had similar MCT, but for prevalence of 0.50 and 0.70, the Tri-Plate had higher MCT in scenarios where leaving a cow with IMI untreated is considered to have greater detrimental effects than treating a healthy cow (i.e., FN:FP of 3:1). Our results showed that the bacteriological tests have adequate validity to diagnose IMI at dry-off, but luminometry does not. We concluded that although luminometry is not useful to identify IMI at dry-off, the Petrifilm and Tri-Plate tests performed similarly to laboratory culture, depending on the prevalence and importance of the FP and FN results.