Accuracy and reproducibility of forensic tire examination decisions.

Tire impression evidence can be a valuable tool during a crime scene investigation-it can link vehicles to scenes or secondary locations, and reveal information about the series of events surrounding a crime. The interpretation of tire impression evidence relies on the expertise of forensic tire examiners. To date, there have not been any studies published that empirically evaluate the accuracy and reproducibility of decisions made by tire impression examiners. This paper presents the results of a study in which 17 tire impression examiners and trainees conducted 238 comparisons on 77 distinct questioned impression-known tire comparison sets (QKsets). This study was conducted digitally and addressed examinations based solely upon the characteristics of the tire impression images provided. The quality and characteristics of the impressions were selected to be broadly representative of those encountered in casework. Participants reported their decisions using a multi-level conclusion scale: 68% of responses were class associations (Association of Class Characteristics or Limited Association of Class), 21% were definitive decisions (ID or Exclusion), 8% were probable decisions (High Degree of Association or Indications of Non-Association), and 3% were neutral responses (Not Suitable or Inconclusive). Although class associations were the most reported response type, when definitive decisions were reported, they were often correct: 96% of IDs and 89% of Exclusions were consistent with ground truth regarding the source of the known tire in the QKset. Overall, we observed 4 erroneous definitive decisions (3 Exclusions on mated QKsets; 1 ID on a nonmated QKset) and 1 incorrect probable decision (Indications of Non-Association on a mated QKset). Decision rates were notably associated with both quality (lower quality questioned impressions were more likely to result in class associations) and dimensionality (2D questioned impressions were more likely to result in definitive decisions), which were correlated factors. Although the study size limits the precision of the measured rates, the results of this study remain valuable to the forensic science and legal communities and provide empirical data regarding examiner performance for a discipline that previously did not have any such estimates.

Variation in assessments of suitability and number of contributors for DNA mixtures.

The interpretation of a DNA mixture (a sample that contains DNA from two or more people) depends on a laboratory/analyst's assessment of the suitability of the sample for comparison/analysis, and an assessment of the number of contributors (NoC) present in the sample. In this study, 134 participants from 67 forensic laboratories provided a total of 2272 assessments of 29 DNA mixtures (provided as electropherograms). The laboratories' responses were evaluated in terms of the variability of suitability assessments, and the accuracy and variability of NoC assessments. Policies and procedures related to suitability and NoC varied notably among labs. We observed notable variation in whether labs would assess a given mixture as suitable or not, predominantly due to differences in lab policies: if two labs following their standard operating procedures (SOPs) were given the same mixture, they agreed on whether the mixture was suitable for comparison 66% of the time. Differences in suitability assessments have a direct effect on variability in interpretations among labs, since mixtures assessed as not suitable would not result in reported interpretations. For labs following their SOPs, 79% of assessments of NoC were correct. When two different labs provided NoC responses, 63% of the time both labs were correct, and 7% of the time both labs were incorrect. Incorrect NoC assessments have been shown to affect statistical analyses in some cases, but do not necessarily imply inaccurate interpretations or conclusions. Most incorrect NoC estimates were overestimates, which previous research has shown have less of an effect on likelihood ratios than underestimates.

DNAmix 2021: Laboratory policies, procedures, and casework scenarios summary and dataset.

DNAmix 2021 is a large-scale study conducted to evaluate the extent of consistency and variation among forensic laboratories in the interpretation of DNA mixtures, and to assess the effects of various potential sources of variability. This study utilized a multi-phasic approach designed to collect information about participating laboratories, laboratory policies, and their standard operating procedures (SOPs). It also characterizes the degree of variation in assessments of suitability and number of contributors as well as in comparisons and statistical analyses of DNA mixture profiles. This paper specifically details the study design and the data collected in the first two phases of the study: the Policies & Procedures (P&P) Questionnaire and the Casework Scenarios Questionnaire (CSQ). We report on the variation in policies and SOPs for 86 forensic laboratories-including information about their DNA workflows, systems, and type of statistics reported. We also provide details regarding various case-scenario specific decisions and the nature of mixture casework for 83 forensic laboratories. The data discussed in this article provide insight into the state of the field for forensic DNA mixture interpretation policies and SOPs at the time of the study (2021-2022).

Accuracy and reliability of forensic handwriting comparisons.

Forensic handwriting examination involves the comparison of writing samples by forensic document examiners (FDEs) to determine whether or not they were written by the same person. Here we report the results of a large-scale study conducted to assess the accuracy and reliability of handwriting comparison conclusions. Eighty-six practicing FDEs each conducted up to 100 handwriting comparisons, resulting in 7,196 conclusions on 180 distinct comparison sets, using a five-level conclusion scale. Erroneous "written by" conclusions (false positives) were reached in 3.1% of the nonmated comparisons, while 1.1% of the mated comparisons yielded erroneous "not written by" conclusions (false negatives). False positive rates were markedly higher for nonmated samples written by twins (8.7%) compared to nontwins (2.5%). Notable associations between training and performance were observed: FDEs with less than 2 y of formal training generally had higher error rates, but they also had higher true positive and true negative rates because they tended to provide more definitive conclusions; FDEs with at least 2 y of formal training were less likely to make definitive conclusions, but those definitive conclusions they made were more likely to be correct (higher positive predictive and negative predictive values). We did not observe any association between writing style (cursive vs. printing) and rates of errors or incorrect conclusions. This report also provides details on the repeatability and reproducibility of conclusions, and reports how conclusions are affected by the quantity of writing and the similarity of content.

Accuracy and reproducibility of conclusions by forensic bloodstain pattern analysts.

Although the analysis of bloodstain pattern evidence left at crime scenes relies on the expert opinions of bloodstain pattern analysts, the accuracy and reproducibility of these conclusions have never been rigorously evaluated at a large scale. We investigated conclusions made by 75 practicing bloodstain pattern analysts on 192 bloodstain patterns selected to be broadly representative of operational casework, resulting in 33,005 responses to prompts and 1760 short text responses. Our results show that conclusions were often erroneous and often contradicted other analysts. On samples with known causes, 11.2% of responses were erroneous. The results show limited reproducibility of conclusions: 7.8% of responses contradicted other analysts. The disagreements with respect to the meaning and usage of BPA terminology and classifications suggest a need for improved standards. Both semantic differences and contradictory interpretations contributed to errors and disagreements, which could have serious implications if they occurred in casework.

Characterizing missed identifications and errors in latent fingerprint comparisons using eye-tracking data.

Latent fingerprint examiners sometimes come to different conclusions when comparing fingerprints, and eye-gaze behavior may help explain these outcomes. missed identifications (missed IDs) are inconclusive, exclusion, or No Value determinations reached when the consensus of other examiners is an identification. To determine the relation between examiner behavior and missed IDs, we collected eye-gaze data from 121 latent print examiners as they completed a total 1444 difficult (latent-exemplar) comparisons. We extracted metrics from the gaze data that serve as proxies for underlying perceptual and cognitive capacities. We used these metrics to characterize potential mechanisms of missed IDs: Cursory Comparison and Mislocalization. We find that missed IDs are associated with shorter comparison times, fewer regions visited, and fewer attempted correspondences between the compared images. Latent print comparisons resulting in erroneous exclusions (a subset of missed IDs) are also more likely to have fixations in different regions and less accurate correspondence attempts than those comparisons resulting in identifications. We also use our derived metrics to describe one atypical examiner who made six erroneous identifications, four of which were on comparisons intended to be straightforward exclusions. The present work helps identify the degree to which missed IDs can be explained using eye-gaze behavior, and the extent to which missed IDs depend on cognitive and decision-making factors outside the domain of eye-tracking methodologies.

Why do latent fingerprint examiners differ in their conclusions?

Forensic latent print examiners usually but do not always reproduce each other's conclusions. Using data from tests of experts conducting fingerprint comparisons, we show the extent to which differing conclusions can be explained in terms of the images, and in terms of the examiners. Some images are particularly prone to disagreements or erroneous conclusions; the highest and lowest quality images generally result in unanimous conclusions. The variability among examiners can be seen as the effect of implicit individual decision thresholds, which we demonstrate are measurable and differ substantially among examiners; this variation may reflect differences in skill, risk tolerance, or bias. Much of the remaining variability relates to inconsistency of the examiners themselves: borderline conclusions (i.e., close to individual decision thresholds) often were not repeated by the examiners themselves, and tended to be completed more slowly and rated difficult. A few examiners have significantly higher error rates than most: aggregate error rates of many examiners are not necessarily representative of individual examiners. The use of a three-level conclusion scale does not precisely represent the underlying agreements and disagreements among examiners. We propose a new method of quantifying examiner skill that would be appropriate for use in proficiency tests. These findings are operationally relevant to staffing, quality assurance, and disagreements among experts in court.

Gaze behavior and cognitive states during fingerprint target group localization.

BACKGROUND: The comparison of fingerprints by expert latent print examiners generally involves repeating a process in which the examiner selects a small area of distinctive features in one print (a target group), and searches for it in the other print. In order to isolate this key element of fingerprint comparison, we use eye-tracking data to describe the behavior of latent fingerprint examiners on a narrowly defined "find the target" task. Participants were shown a fingerprint image with a target group indicated and asked to find the corresponding area of ridge detail in a second impression of the same finger and state when they found the target location. Target groups were presented on latent and plain exemplar fingerprint images, and as small areas cropped from the plain exemplars, to assess how image quality and the lack of surrounding visual context affected task performance and eye behavior. One hundred and seventeen participants completed a total of 675 trials. RESULTS: The presence or absence of context notably affected the areas viewed and time spent in comparison; differences between latent and plain exemplar tasks were much less significant. In virtually all trials, examiners repeatedly looked back and forth between the images, suggesting constraints on the capacity of visual working memory. On most trials where context was provided, examiners looked immediately at the corresponding location: with context, median time to find the corresponding location was less than 0.3 s (second fixation); however, without context, median time was 1.9 s (five fixations). A few trials resulted in errors in which the examiner did not find the correct target location. Basic gaze measures of overt behaviors, such as speed, areas visited, and back-and-forth behavior, were used in conjunction with the known target area to infer the underlying cognitive state of the examiner. CONCLUSIONS: Visual context has a significant effect on the eye behavior of latent print examiners. Localization errors suggest how errors may occur in real comparisons: examiners sometimes compare an incorrect but similar target group and do not continue to search for a better candidate target group. The analytic methods and predictive models developed here can be used to describe the more complex behavior involved in actual fingerprint comparisons.

Factors associated with latent fingerprint exclusion determinations.

Exclusion is the determination by a latent print examiner that two friction ridge impressions did not originate from the same source. The concept and terminology of exclusion vary among agencies. Much of the literature on latent print examination focuses on individualization, and much less attention has been paid to exclusion. This experimental study assesses the associations between a variety of factors and exclusion determinations. Although erroneous exclusions are more likely to occur on some images and for some examiners, they were widely distributed among images and examiners. Measurable factors found to be associated with exclusion rates include the quality of the latent, value determinations, analysis minutia count, comparison difficulty, and the presence of cores or deltas. An understanding of these associations will help explain the circumstances under which errors are more likely to occur and when determinations are less likely to be reproduced by other examiners; the results should also lead to improved effectiveness and efficiency of training and casework quality assurance. This research is intended to assist examiners in improving the examination process and provide information to the broader community regarding the accuracy, reliability, and implications of exclusion decisions.

Data on the interexaminer variation of minutia markup on latent fingerprints.

The data in this article supports the research paper entitled "Interexaminer variation of minutia markup on latent fingerprints" [1]. The data in this article describes the variability in minutia markup during both analysis of the latents and comparison between latents and exemplars. The data was collected in the "White Box Latent Print Examiner Study," in which each of 170 volunteer latent print examiners provided detailed markup documenting their examinations of latent-exemplar pairs of prints randomly assigned from a pool of 320 pairs. Each examiner examined 22 latent-exemplar pairs; an average of 12 examiners marked each latent.

Interexaminer variation of minutia markup on latent fingerprints.

Latent print examiners often differ in the number of minutiae they mark during analysis of a latent, and also during comparison of a latent with an exemplar. Differences in minutia counts understate interexaminer variability: examiners' markups may have similar minutia counts but differ greatly in which specific minutiae were marked. We assessed variability in minutia markup among 170 volunteer latent print examiners. Each provided detailed markup documenting their examinations of 22 latent-exemplar pairs of prints randomly assigned from a pool of 320 pairs. An average of 12 examiners marked each latent. The primary factors associated with minutia reproducibility were clarity, which regions of the prints examiners chose to mark, and agreement on value or comparison determinations. In clear areas (where the examiner was "certain of the location, presence, and absence of all minutiae"), median reproducibility was 82%; in unclear areas, median reproducibility was 46%. Differing interpretations regarding which regions should be marked (e.g., when there is ambiguity in the continuity of a print) contributed to variability in minutia markup: especially in unclear areas, marked minutiae were often far from the nearest minutia marked by a majority of examiners. Low reproducibility was also associated with differences in value or comparison determinations. Lack of standardization in minutia markup and unfamiliarity with test procedures presumably contribute to the variability we observed. We have identified factors accounting for interexaminer variability; implementing standards for detailed markup as part of documentation and focusing future training efforts on these factors may help to facilitate transparency and reduce subjectivity in the examination process.

Changes in latent fingerprint examiners' markup between analysis and comparison.

After the initial analysis of a latent print, an examiner will sometimes revise the assessment during comparison with an exemplar. Changes between analysis and comparison may indicate that the initial analysis of the latent was inadequate, or that confirmation bias may have affected the comparison. 170 volunteer latent print examiners, each randomly assigned 22 pairs of prints from a pool of 320 total pairs, provided detailed markup documenting their interpretations of the prints and the bases for their comparison conclusions. We describe changes in value assessments and markup of features and clarity. When examiners individualized, they almost always added or deleted minutiae (90.3% of individualizations); every examiner revised at least some markups. For inconclusive and exclusion determinations, changes were less common, and features were added more frequently when the image pair was mated (same source). Even when individualizations were based on eight or fewer corresponding minutiae, in most cases some of those minutiae had been added during comparison. One erroneous individualization was observed: the markup changes were notably extreme, and almost all of the corresponding minutiae had been added during comparison. Latents assessed to be of value for exclusion only (VEO) during analysis were often individualized when compared to a mated exemplar (26%); in our previous work, where examiners were not required to provide markup of features, VEO individualizations were much less common (1.8%).

Measuring what latent fingerprint examiners consider sufficient information for individualization determinations.

Latent print examiners use their expertise to determine whether the information present in a comparison of two fingerprints (or palmprints) is sufficient to conclude that the prints were from the same source (individualization). When fingerprint evidence is presented in court, it is the examiner's determination--not an objective metric--that is presented. This study was designed to ascertain the factors that explain examiners' determinations of sufficiency for individualization. Volunteer latent print examiners (n = 170) were each assigned 22 pairs of latent and exemplar prints for examination, and annotated features, correspondence of features, and clarity. The 320 image pairs were selected specifically to control clarity and quantity of features. The predominant factor differentiating annotations associated with individualization and inconclusive determinations is the count of corresponding minutiae; other factors such as clarity provided minimal additional discriminative value. Examiners' counts of corresponding minutiae were strongly associated with their own determinations; however, due to substantial variation of both annotations and determinations among examiners, one examiner's annotation and determination on a given comparison is a relatively weak predictor of whether another examiner would individualize. The extensive variability in annotations also means that we must treat any individual examiner's minutia counts as interpretations of the (unknowable) information content of the prints: saying "the prints had N corresponding minutiae marked" is not the same as "the prints had N corresponding minutiae." More consistency in annotations, which could be achieved through standardization and training, should lead to process improvements and provide greater transparency in casework.

On relative distortion in fingerprint comparison.

When fingerprints are deposited, non-uniform pressure in conjunction with the inherent elasticity of friction ridge skin often causes linear and non-linear distortions in the ridge and valley structure. The effects of these distortions must be considered during analysis of fingerprint images. Even when individual prints are not notably distorted, relative distortion between two prints can have a serious impact on comparison. In this paper we discuss several metrics for quantifying and visualizing linear and non-linear fingerprint deformations, and software tools to assist examiners in accounting for distortion in fingerprint comparisons.

Understanding the sufficiency of information for latent fingerprint value determinations.

A latent print examiner's assessment of the value, or suitability, of a latent impression is the process of determining whether the impression has sufficient information to make a comparison. A "no value" determination preemptively states that no individualization or exclusion determination could be made using the impression, regardless of quality of the comparison prints. Factors contributing to a value determination include clarity and the types, quantity, and relationships of features. These assessments are made subjectively by individual examiners and may vary among examiners. We modeled the relationships between value determinations and feature annotations made by 21 certified latent print examiners on 1850 latent impressions. Minutia count was strongly associated with value determinations. None of the models resulted in a stronger intraexaminer association with "value for individualization" determinations than minutia count alone. The association between examiner annotation and value determinations is greatly limited by the lack of reproducibility of both annotation and value determinations.

Assessing the clarity of friction ridge impressions.

The ability of friction ridge examiners to correctly discern and make use of the ridges and associated features in finger or palm impressions is limited by clarity. The clarity of an impression relates to the examiner's confidence that the presence, absence, and attributes of features can be correctly discerned. Despite the importance of clarity in the examination process, there have not previously been standard methods for assessing clarity in friction ridge impressions. We introduce a process for annotation, analysis, and interchange of friction ridge clarity information that can be applied to latent or exemplar impressions. This paper: (1) describes a method for evaluating the clarity of friction ridge impressions by using color-coded annotations that can be used by examiners or automated systems; (2) discusses algorithms for overall clarity metrics based on manual or automated clarity annotation; and (3) defines a method of quantifying the correspondence of clarity when comparing a pair of friction ridge images, based on clarity annotation and resulting metrics. Different uses of this approach include examiner interchange of data, quality assurance, metrics, and as an aid in automated fingerprint matching.

Repeatability and reproducibility of decisions by latent fingerprint examiners.

The interpretation of forensic fingerprint evidence relies on the expertise of latent print examiners. We tested latent print examiners on the extent to which they reached consistent decisions. This study assessed intra-examiner repeatability by retesting 72 examiners on comparisons of latent and exemplar fingerprints, after an interval of approximately seven months; each examiner was reassigned 25 image pairs for comparison, out of total pool of 744 image pairs. We compare these repeatability results with reproducibility (inter-examiner) results derived from our previous study. Examiners repeated 89.1% of their individualization decisions, and 90.1% of their exclusion decisions; most of the changed decisions resulted in inconclusive decisions. Repeatability of comparison decisions (individualization, exclusion, inconclusive) was 90.0% for mated pairs, and 85.9% for nonmated pairs. Repeatability and reproducibility were notably lower for comparisons assessed by the examiners as "difficult" than for "easy" or "moderate" comparisons, indicating that examiners' assessments of difficulty may be useful for quality assurance. No false positive errors were repeated (n = 4); 30% of false negative errors were repeated. One percent of latent value decisions were completely reversed (no value even for exclusion vs. of value for individualization). Most of the inter- and intra-examiner variability concerned whether the examiners considered the information available to be sufficient to reach a conclusion; this variability was concentrated on specific image pairs such that repeatability and reproducibility were very high on some comparisons and very low on others. Much of the variability appears to be due to making categorical decisions in borderline cases.

Accuracy and reliability of forensic latent fingerprint decisions.

The interpretation of forensic fingerprint evidence relies on the expertise of latent print examiners. The National Research Council of the National Academies and the legal and forensic sciences communities have called for research to measure the accuracy and reliability of latent print examiners' decisions, a challenging and complex problem in need of systematic analysis. Our research is focused on the development of empirical approaches to studying this problem. Here, we report on the first large-scale study of the accuracy and reliability of latent print examiners' decisions, in which 169 latent print examiners each compared approximately 100 pairs of latent and exemplar fingerprints from a pool of 744 pairs. The fingerprints were selected to include a range of attributes and quality encountered in forensic casework, and to be comparable to searches of an automated fingerprint identification system containing more than 58 million subjects. This study evaluated examiners on key decision points in the fingerprint examination process; procedures used operationally include additional safeguards designed to minimize errors. Five examiners made false positive errors for an overall false positive rate of 0.1%. Eighty-five percent of examiners made at least one false negative error for an overall false negative rate of 7.5%. Independent examination of the same comparisons by different participants (analogous to blind verification) was found to detect all false positive errors and the majority of false negative errors in this study. Examiners frequently differed on whether fingerprints were suitable for reaching a conclusion.