Interpretable algorithmic forensics.

One of the most troubling trends in criminal investigations is the growing use of "black box" technology, in which law enforcement rely on artificial intelligence (AI) models or algorithms that are either too complex for people to understand or they simply conceal how it functions. In criminal cases, black box systems have proliferated in forensic areas such as DNA mixture interpretation, facial recognition, and recidivism risk assessments. The champions and critics of AI argue, mistakenly, that we face a catch 22: While black box AI is not understandable by people, they assume that it produces more accurate forensic evidence. In this Article, we question this assertion, which has so powerfully affected judges, policymakers, and academics. We describe a mature body of computer science research showing how "glass box" AI-designed to be interpretable-can be more accurate than black box alternatives. Indeed, black box AI performs predictably worse in settings like the criminal system. Debunking the black box performance myth has implications for forensic evidence, constitutional criminal procedure rights, and legislative policy. Absent some compelling-or even credible-government interest in keeping AI as a black box, and given the constitutional rights and public safety interests at stake, we argue that a substantial burden rests on the government to justify black box AI in criminal cases. We conclude by calling for judicial rulings and legislation to safeguard a right to interpretable forensic AI.

Convicting with confidence? Why we should not over-rely on eyewitness confidence.

Eyewitness memory researchers have recently devoted considerable attention to eyewitness confidence. While there is strong consensus that courtroom confidence is problematic, we now recognise that an eyewitness's initial confidence in their first identification - in certain contexts - can be of value. A few psychological scientists, however, have confidently, but erroneously claimed that in real-world cases, eyewitness initial confidence is the most important indicator of eyewitness accuracy, trumping all other factors that might exist in a case. This claim accompanies an exaggeration of the role of eyewitnesses' "initial confidence" in the DNA exoneration cases. Still worse, overstated claims about the confidence-accuracy relationship, and eyewitness memory, have reached our top scientific journals, news articles, and criminal cases. To set the record straight, we review what we actually know and do not know about the "initial confidence" of eyewitnesses in the DNA exoneration cases. Further reasons for skepticism about the value of the confidence-accuracy relationship in real-world cases come from new analyses of a separate database, the National Registry of Exonerations. Finally, we review new research that reveals numerous conditions wherein eyewitnesses with high initial confidence end up being wrong.

Eyewitness confidence may not be ready for the courts: a reply to Wixted et al.

Wixted et al. (in press. Doing right by the eyewitness evidence: A response to Berkowitz et al. Memory) remind us that they are aware of some conditions in which confidence does not trump all but suggest that initial high-confidence errors should be rare. In this reply, we draw attention to new lab research that continues to cast doubt on the value of an initial eyewitness identification made with high confidence. Additional data from field studies of police lineups lead us to conclude that it is far too risky in real-world cases to assume that eyewitnesses who have high initial confidence are also highly accurate. As a final point, we dispute Wixted et al.'s interpretation of "initial low confidence" in the DNA exoneration cases.

Open risk assessment.

As criminal justice actors increasingly seek to rely on more evidence-informed practices, including risk assessment instruments, they often lack adequate information about the evidence that informed the development of the practice or the tool. Open science practices, including making scientific research and data accessible and public, have not typically been followed in the development of tools designed for law enforcement, judges, probation, and others. This is in contrast to other government agencies, which often open their processes to public notice and comment. Lack of transparency has become pressing in the area of risk assessment, as entire judicial systems have adopted some type of risk assessment scheme. While the types of information used in a risk tool may be made public, often the underlying methods, validation data, and studies are not - nor are the assumptions behind how a level of risk gets categorized as "high" or "low." We discuss why those concerns are relevant and important to the new risk assessment tool now being used in federal prisons, as part of the First Step Act. We conclude that a number of key assumptions and policy choices made in the design of that tool are not verifiable or are inadequately supported, including the choice of risk thresholds and the validation data itself. Unfortunately, as a result, the federal risk assessment effort has not been the hoped-for model for open risk assessment.

Risk assessment in sentencing and plea bargaining: The roles of prosecutors and defense attorneys.

Research on risk assessment in sentencing has focused heavily on the role of judges. Ignoring the role of other courtroom actors in the sentencing process, however, leaves unexamined the potentially significant effects on judicial decision making of arguments made by prosecutors and defense attorneys at sentencing hearings. Unduly focusing on judges also overlooks the vast majority of sentences arrived at through negotiated guilty pleas. We explored the extent to which considerations of risk are made among prosecutors and defense attorneys when advocating for given sentences in open court or during plea negotiations. We surveyed all prosecutors and defense attorneys in 14 judicial circuits in Virginia and found that most prosecutors and defense attorneys at least "sometimes" explicitly invoked actuarial risk estimates both at sentencing hearings and during plea negotiations. However, defense attorneys were much more likely than prosecutors to be averse to the use of risk assessment in either form of case disposition.

Mock jurors' evaluation of firearm examiner testimony.

OBJECTIVES: Firearms experts traditionally have testified that a weapon leaves "unique" toolmarks, so bullets or cartridge casings can be visually examined and conclusively matched to a particular firearm. Recently, due to scientific critiques, Department of Justice policy, and judges' rulings, firearms experts have tempered their conclusions. In two experiments, we tested whether this ostensibly more cautious language has its intended effect on jurors (Experiment 1), and whether cross-examination impacts jurors' perception of firearm testimony (Experiment 2). HYPOTHESES: Four hypotheses were tested. First, jurors will accord significant weight to firearm testimony that declares a "match" compared to testimony that does not (Experiments 1 and 2). Second, variations to "match" language will not affect guilty verdicts (Experiment 1). Third, only the most cautious language ("cannot exclude the gun") would lower guilty verdicts (Experiment 1). Fourth, cross-examination will reduce guilty verdicts depending on specific language used (Experiment 2). METHOD: In two preregistered, high-powered experiments with 200 mock jurors per cell, participants recruited from Qualtrics Panels were presented with a criminal case containing firearms evidence, which varied the wording of the examiner's conclusion and whether cross-examination was present. These variations include conclusion language used by practitioners, language advised by government organizations, and language required by judges in several cases. Participants gave a verdict, rated the evidence and expert in all conditions. RESULTS: Guilty verdicts significantly increased when a match was declared compared to when a match was not declared. Variation in conclusion language did not affect guilty verdicts nor did it affect jurors' estimates of the likelihood the defendant's gun fired the bullet recovered at the crime scene. In contrast, however, a more cautious conclusion that an examiner "cannot exclude the defendant's gun" did significantly reduce guilty verdicts and likelihood estimates alike. The presence of cross-examination did not affect these findings. CONCLUSION: Apart from the most limited language ("cannot exclude the defendant's gun"), judicial intervention to limit firearms conclusion language is not likely to produce its intended effect. Moreover, cross-examination does not appear to affect perceptions or individual juror verdicts. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

The impact of proficiency testing information and error aversions on the weight given to fingerprint evidence.

Fingerprint examiners regularly participate in tests designed to assess their proficiency. These tests provide information relevant to the weight of fingerprint evidence, but no prior research has directly examined how jurors react to proficiency testing information. Using a nationally representative sample of American adults, we examined the impact of proficiency testing information on the weight given to the opinions of fingerprint examiners by mock jurors considering a hypothetical criminal case. The fingerprint examiner's level of performance on a proficiency test (high, medium, low, or very low), but not the type of error committed on the test (false positive identifications, false negative identifications, or a mix of both types of error), affected the weight that jury-eligible adults gave to an examiner's opinion that latent fingerprints recovered from a crime scene matched the defendant's fingerprints, which in turn affected judgments about the defendant's guilt. Jurors who had no information about proficiency gave similar weight to the testimony as jurors exposed to highly proficient examiners, suggesting that jurors assume fingerprint examiners perform at high levels of proficiency unless informed otherwise. We also found that a plurality of Americans deems false acquittals just as aversive as false convictions and a significant minority deems false acquittals more serious. These differences in error aversions predicted differences in evidentiary assessments, suggesting that error aversions of jurors may play an important role in criminal trials.

Judicial appraisals of risk assessment in sentencing.

The assessment of an offender's risk of recidivism is emerging as a key consideration in sentencing policy in many US jurisdictions. However, little information is available on how actual sentencing judges view this development. This study surveys the views of a population sample of judges in Virginia, the state that has gone further than any other in legislatively mandating risk assessment for certain drug and property offenders. Results indicate that a strong majority of judges endorse the principle that sentencing eligible offenders should include a consideration of recidivism risk. However, a strong majority also report the availability of alternatives to imprisonment in their jurisdictions to be inadequate at best. Finally, most judges oppose the adoption of a policy requiring them to provide a written reason for declining to impose alternative interventions on "low-risk" offenders.

New Insights on Expert Opinion About Eyewitness Memory Research.

Experimental psychologists investigating eyewitness memory have periodically gathered their thoughts on a variety of eyewitness memory phenomena. Courts and other stakeholders of eyewitness research rely on the expert opinions reflected in these surveys to make informed decisions. However, the last survey of this sort was published more than 20 years ago, and the science of eyewitness memory has developed since that time. Stakeholders need a current database of expert opinions to make informed decisions. In this article, we provide that update. We surveyed 76 scientists for their opinions on eyewitness memory phenomena. We compared these current expert opinions to expert opinions from the past several decades. We found that experts today share many of the same opinions as experts in the past and have more nuanced thoughts about two issues. Experts in the past endorsed the idea that confidence is weakly related to accuracy, but experts today acknowledge the potential diagnostic value of initial confidence collected from a properly administered lineup. In addition, experts in the past may have favored sequential over simultaneous lineup presentation, but experts today are divided on this issue. We believe this new survey will prove useful to the court and to other stakeholders of eyewitness research.

Surveying practicing firearm examiners.

A sample (n = 79) of practicing firearm and toolmark examiners was queried about casework as well as their views about the potential role that statistics might play in future firearm examinations and expert witness testimony. Principal findings include: The modal response for time spent conducting bullet examinations is 2-4 hours, and the modal response for cartridge casings is 1-2 hours. The average participant (median) makes an identification in 65% of casework, makes an elimination in 12% of casework, and reports that the examination was inconclusive in 20% of casework calls. The vast majority of examiners work at laboratories that permit eliminations when class characteristics agree. The reported industry-wide false positive error rate is 1%, though very few participants could name a study or give a citation for their reported estimate. Qualitative responses about the potential role of statistics were mixed.

A practical tool for information management in forensic decisions: Using Linear Sequential Unmasking-Expanded (LSU-E) in casework.

Forensic analysts often receive information from a multitude of sources. Empirical work clearly demonstrates that biasing information can affect analysts' decisions, and that the order in which task-relevant information is received impacts human cognition and decision-making. Linear Sequential Unmasking (LSU; Dror et al., 2015) and LSU-Expanded (LSU-E; Dror & Kukucka, 2021) are examples of research-based procedural frameworks to guide laboratories' and analysts' consideration and evaluation of case information. These frameworks identify parameters-such as objectivity, relevance, and biasing power-to prioritize and optimally sequence information for forensic analyses. Moreover, the LSU-E framework can be practically incorporated into any forensic discipline to improve decision quality by increasing the repeatability, reproducibility, and transparency of forensic analysts' decisions, as well as reduce bias. Future implementation of LSU and LSU-E in actual forensic casework can be facilitated by concrete guidance. We present here a practical worksheet designed to bridge the gap between research and practice by facilitating the implementation of LSU-E.

Judges and forensic science education: A national survey.

In criminal cases, forensic science reports and expert testimony play an increasingly important role in adjudication. More states now follow a federal reliability standard, which calls upon judges to assess the reliability and validity of scientific evidence. Little is known about how judges view their own background in forensic scientific evidence, and what types of specialized training they receive on it. In this study, we surveyed 164 judges from 39 different U.S. states, who attended past trainings at the National Judicial College. We asked these judges about their background in forensic science, their views concerning the reliability of common forensic disciplines, and their needs to better evaluate forensic science evidence. We discovered that judges held views regarding the scientific support for different forensic science disciplines that were fairly consistent with available literature; their error rate estimates were more supported by research than many estimates by laypersons, who often assume forensic methods are nearly infallible. We did not find any association between how judges rate forensic reliability and prior training. We did, however, find that training corresponded with judges' views that they should, and do in fact, take on a more active gatekeeping role regarding forensics. Regarding the tools judges need to vet forensic experts and properly evaluate forensic science evidence, they reported having very different backgrounds in relevant scientific concepts and having forensic science education needs. Judges reported needs in accessing better material concerning reliability of forensic science methods. These results support new efforts to expand scientific evidence education in the judiciary.

Test a Witness's Memory of a Suspect Only Once.

Eyewitness misidentifications are almost always made with high confidence in the courtroom. The courtroom is where eyewitnesses make their last identification of defendants suspected of (and charged with) committing a crime. But what did those same eyewitnesses do on the first identification test, conducted early in a police investigation? Despite testifying with high confidence in court, many eyewitnesses also testified that they had initially identified the suspect with low confidence or failed to identify the suspect at all. Presenting a lineup leaves the eyewitness with a memory trace of the faces in the lineup, including that of the suspect. As a result, the memory signal generated by the face of that suspect will be stronger on a later test involving the same witness, even if the suspect is innocent. In that sense, testing memory contaminates memory. These considerations underscore the importance of a newly proposed recommendation for conducting eyewitness identifications: Avoid repeated identification procedures with the same witness and suspect. This recommendation applies not only to additional tests conducted by police investigators but also to the final test conducted in the courtroom, in front of the judge and jury.

North Carolina Specialty Courts, Treatment Access, and the Substance Use Crisis: A Promising but Underfunded Model.

Treatment courts aim to reduce criminal recidivism by addressing the behavioral health care needs of persons with psychiatric or substance use disorders that contribute to their offending. Stable funding and access to behavioral health providers are crucial elements of success for the treatment court model. What happens when courts lose state funding and must rely on local initiatives and resources? In this study, a survey of North Carolina treatment court professionals identified resource gaps and unmet needs. The authors argue that continuing state investment could make treatment courts more viable and effective. Medicaid expansion is a potential new resource for these problem-solving courts.

Juror appraisals of forensic evidence: Effects of blind proficiency and cross-examination.

Forensic testimony plays a crucial role in many criminal cases, with requests to crime laboratories steadily increasing. As part of efforts to improve the reliability of forensic evidence, scientific and policy groups increasingly recommend routine and blind proficiency tests of practitioners. What is not known is how doing so affects how lay jurors assess testimony by forensic practitioners in court. In Study 1, we recruited 1398 lay participants, recruited online using Qualtrics to create a sample representative of the U.S. population with respect to age, gender, income, race/ethnicity, and geographic region. Each read a mock criminal trial transcript in which a forensic examiner presented the central evidence. The low-proficiency forensic examiner elicited a lower conviction rate and less favorable impressions than the control, an examiner for which no proficiency information was disclosed. However, the high-proficiency examiner did not correspondingly elicit a higher conviction rate or more favorable impressions than the control. In Study 2, 1420 participants, similarly recruited using Qualtrics, received the same testimony, but for some conditions the examiner was cross-examined by a defense attorney. We find cross-examination significantly reduced guilty votes and examiner ratings for low-proficiency examiners. These results suggest that disclosing results of blind proficiency testing can inform jury decision-making, and further, that defense lawyering can make proficiency information particularly salient at a criminal trial.

Error Rates, Likelihood Ratios, and Jury Evaluation of Forensic Evidence.

Forensic examiners regularly testify in criminal cases, informing the jurors whether crime scene evidence likely came from a source. In this study, we examine the impact of providing jurors with testimony further qualified by error rates and likelihood ratios, for expert testimony concerning two forensic disciplines: commonly used fingerprint comparison evidence and a novel technique involving voice comparison. Our method involved surveying mock jurors in Amazon Mechanical Turk (N = 897 laypeople) using written testimony and judicial instructions. Participants were more skeptical of voice analysis and generated fewer "guilty" decisions than for fingerprint analysis (B = 2.00, OR = 7.06, p = <0.000). We found that error rate information most strongly decreased "guilty" votes relative to no qualifying information for participants who heard fingerprint evidence (but not those that heard voice analysis evidence; B = -1.16, OR = 0.32, p = 0.007). We also found that error rates and conclusion types led to a greater decrease on "guilty" votes for fingerprint evidence than voice evidence (B = 1.44, OR = 4.23, p = 0.021). We conclude that these results suggest jurors adjust the weight placed on forensic evidence depending on their prior views about its reliability. Future research should develop testimony and judicial instructions that can better inform jurors of the strengths and limitations of forensic evidence.

Resolving latent conflict: What happens when latent print examiners enter the cage?

Latent print examination traditionally follows the ACE-V process, in which latent prints are first analyzed to determine whether they are suitable for comparison, and then compared to an exemplar and evaluated for similarities and differences. Despite standard operating procedures and quality controls designed, in part, to mitigate differences between examiners, latent print processing and review are inherently subjective. The ACE-V process addresses subjectivity, and the possibility of error, in the verification stage in which a second examiner repeats the analysis, comparison, and evaluation steps in a given case. Other procedures outside the ACE-V framework, such as consultation and conflict resolution, provide further opportunity to understand how differences between latent print examiners emerge. Despite the growing body of research on latent print examination, questions have emerged about how these procedures work in practice. This study reviews case processing data for two years of casework at the Houston Forensic Science Center (HFSC). We describe these data as cases proceed through each step of the ACE-V process, with a particular focus on verification, consultation, and conflict resolution. We discuss trends in these processes regarding modal types of disagreements, modal outcomes, and roles of the examiners involved. Results reveal implications for improving the practice of latent print examination.

Forensic bitemark identification: weak foundations, exaggerated claims.

Several forensic sciences, especially of the pattern-matching kind, are increasingly seen to lack the scientific foundation needed to justify continuing admission as trial evidence. Indeed, several have been abolished in the recent past. A likely next candidate for elimination is bitemark identification. A number of DNA exonerations have occurred in recent years for individuals convicted based on erroneous bitemark identifications. Intense scientific and legal scrutiny has resulted. An important National Academies review found little scientific support for the field. The Texas Forensic Science Commission recently recommended a moratorium on the admission of bitemark expert testimony. The California Supreme Court has a case before it that could start a national dismantling of forensic odontology. This article describes the (legal) basis for the rise of bitemark identification and the (scientific) basis for its impending fall. The article explains the general logic of forensic identification, the claims of bitemark identification, and reviews relevant empirical research on bitemark identification-highlighting both the lack of research and the lack of support provided by what research does exist. The rise and possible fall of bitemark identification evidence has broader implications-highlighting the weak scientific culture of forensic science and the law's difficulty in evaluating and responding to unreliable and unscientific evidence.