A study of examiner accuracy in cartridge case comparisons. Part 1: Examiner error rates.

This report describes a study undertaken to estimate examiner (not laboratory) error rates for false Identifications and false Eliminations when comparing an unknown to a collection of three known cartridge cases. Volunteer active examiners with Association of Firearm and Toolmark Examiners (AFTE) membership or working in laboratories that participate in the Association of Crime Laboratory Directors (ASCLD) were provided with 15 sets of three known and one questioned cartridge cases fired from a collection of 25 new Ruger SR9 handguns. Remington 9-mm Luger (manufacturer designation L9MM3) ammunition was used and comparison sets were made up of cartridge cases fired within 100 cartridges of each other for each gun. Examiners were provided with a background survey, an answer sheet allowing for the AFTE Range of Conclusions, and return shipping materials. In addition to determining whether the known and questioned cartridge cases were fired with the same handgun, examiners were also asked to assess how many of the three knowns in each set were suitable for comparison, providing an estimated rate of how often each firearm used in the study produces useable, quality marks. The participating examiners were provided with both same-source and different-source comparison sets allowing the study to assess both error rates. Responses were received from 218 participating examiners. The overall rate of false Eliminations was estimated as 0.367% from comparisons known to be from the same firearm but reported as Eliminations. The overall rate of false Identifications was estimated as 1.01% from comparisons known to be from different firearms but reported as Identifications. The rates are not uniform across the sample population with a few examiners providing most of the false Identification responses. Rates of poor-quality mark production varied across the 25 sample handguns; those rates were 2.3% ( ± 1.4%). Both false Elimination and false Identification rates are comparable to or lower than the rate of production of poor-quality marks by the firearms used in this study.

A study of examiner accuracy in cartridge case comparisons. Part 2: Examiner use of the AFTE range of conclusions.

This report describes an analysis of how examiners used the Association of Firearm and Toolmark Examiners (AFTE) Range of Conclusions in a controlled study undertaken to estimate examiner error rates in comparing cartridge cases. Results of the error rate analysis are reported in [1]; this paper focuses on a broader analysis of how the entire collection of classification categories, especially those in the Inconclusive range, were used by the participating examiners. Volunteer active examiners with AFTE membership or working in laboratories that participate in Association of Crime Laboratory Directors (ASCLD) were provided with 15 sets of three known and one questioned cartridge cases fired from a collection of 25 new Ruger SR9 handguns. Remington 9-mm Luger (manufacturer designation L9MM3) ammunition was used and comparison sets were made up of cartridge cases fired within 100 cartridges of each other for each gun. Examiners were provided with a background survey, an answer sheet allowing for the AFTE Range of Conclusions, and return shipping materials. The participating examiners were provided with both same-source and different-source comparison sets allowing the study to assess both error rates. Responses were received from 218 participating examiners. The overall rate of false-negatives was estimated as 0.367 % from comparisons known to be from the same firearm but reported as eliminations. The overall rate of false-positives was estimated as 1.01 % from comparisons known to be from different firearms but reported as identifications. In the case of true different-source examinations, it is clear that the three Inconclusive categories and the Elimination category are not used consistently by all examiners. We identify five different apparent patterns of use of the AFTE Range of Conclusions scale, and discuss possible reasons for and implications of these differences.

Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners.

This paper describes design and logistical aspects of a decision analysis study to assess the performance of qualified firearms examiners working in accredited laboratories in the United States in terms of accuracy (error rate), repeatability, and reproducibility of decisions involving comparisons of fired bullets and cartridge cases. The purpose of the study was to validate current practice of the forensic discipline of firearms/toolmarks (F/T) examination. It elicited error rate data by counting the number of false positive and false negative conclusions. Preceded by the experimental design, decisions, and logistics described herein, testing was ultimately administered 173 qualified, practicing F/T examiners in public and private crime laboratories. The first round of testing evaluated accuracy, while two subsequent rounds evaluated repeatability and reproducibility of examiner conclusions. This project expands on previous studies by involving many F/T examiners in challenging comparisons and by executing the study in the recommended double-blind format.

Observation of large photoacoustic signal phase changes during a diffusion process.

The phase of the photoacoustic signal is known to be a sensitive and accurate means to investigate, both qualitatively and quantitatively, static multilayer heterogeneous systems. According to theory, the maximum phase delay for a very weakly absorbing homogeneous sample should be within 45 degrees of a very strongly absorbing sample, while for heterogeneous samples the phase delay can be greater than 45 degrees. Here we report the observation of photoacoustic phase delays greater than 350 degrees by extending the use of step-scan phase modulation photoacoustic spectroscopy to study a non-repetitive dynamic system in situ, in real time. These large phase delays correspond to sampling several thermal diffusion lengths into the sample. The model system used in this study consisted of a hydrocarbon grease diffusing through a porous Teflon film. The progress of the diffusion was tracked by monitoring both the photoacoustic signal magnitude and the phase of the hydrocarbon grease after isolation from the Teflon film signal contributions at two different phase modulation frequencies.

Analysis of glass fragments by laser ablation-inductively coupled plasma-mass spectrometry and principal component analysis.

Laser ablation-inductively coupled plasma-mass spectrometry (LA-ICP-MS) is used to differentiate glass samples with similar optical and physical properties based on trace elemental composition. Laser ablation increases the number of elements that can be used for differentiation by eliminating problems commonly associated with dissolution and contamination. In this study, standard residential window and tempered glass samples that could not be differentiated by refractive index or density were successfully differentiated by LA-ICP-MS. The primary analysis approach used is Principal Component Analysis (PCA) of the complete mass spectrum. PCA, a multivariate analysis technique, provides rapid analysis of samples without time-consuming pair-wise comparison of calibrated analyses or prior knowledge of the elements present in the samples. Probabilities for positive association of the individual samples are derived from PCA. Utilization of the Q-statistic with PCA allowed us to distinguish all samples within the set to a certainty greater than the 99% confidence interval.

Multivariate pattern matching of trace elements in solids by laser ablation inductively coupled plasma-mass spectrometry: source attribution and preliminary diagnosis of fractionation.

Laser ablation inductively coupled plasma mass spectrometry (LA-ICPMS) is used with two variations of principal components analysis (PCA) for objective, routine comparisons of forensic materials without time-consuming and destructive sample dissolution. The relative concentrations of trace elements in a solid sample are examined to provide a "fingerprint" composition that can be used for identification and source matching of the material. Residue samples are matched to bulk materials using PCA. Variation of laser focus and PCA are also used to diagnose the severity of elemental fractionation in two metal samples that are prone to fractionation, brass and steel. Such fractionation remains the most significant limitation for accurate quantitative analyses by LA-ICPMS.