A noise audit of human-labeled benchmarks for machine commonsense reasoning.

With the advent of large language models, evaluating and benchmarking these systems on important AI problems has taken on newfound importance. Such benchmarking typically involves comparing the predictions of a system against human labels (or a single 'ground-truth'). However, much recent work in psychology has suggested that most tasks involving significant human judgment can have non-trivial degrees of noise. In his book, Kahneman suggests that noise may be a much more significant component of inaccuracy compared to bias, which has been studied more extensively in the AI community. This article proposes a detailed noise audit of human-labeled benchmarks in machine commonsense reasoning, an important current area of AI research. We conduct noise audits under two important experimental conditions: one in a smaller-scale but higher-quality labeling setting, and another in a larger-scale, more realistic online crowdsourced setting. Using Kahneman's framework of noise, our results consistently show non-trivial amounts of level, pattern, and system noise, even in the higher-quality setting, with comparable results in the crowdsourced setting. We find that noise can significantly influence the performance estimates that we obtain of commonsense reasoning systems, even if the 'system' is a human; in some cases, by almost 10 percent. Labeling noise also affects performance estimates of systems like ChatGPT by more than 4 percent. Our results suggest that the default practice in the AI community of assuming and using a 'single' ground-truth, even on problems requiring seemingly straightforward human judgment, may warrant empirical and methodological re-visiting.

GRADE guidance 37: rating imprecision in a body of evidence on test accuracy.

OBJECTIVES: To provide guidance on rating imprecision in a body of evidence assessing the accuracy of a single test. This guide will clarify when Grading of Recommendations Assessment, Development and Evaluation (GRADE) users should consider rating down the certainty of evidence by one or more levels for imprecision in test accuracy. STUDY DESIGN AND SETTING: A project group within the GRADE working group conducted iterative discussions and presentations at GRADE working group meetings to produce this guidance. RESULTS: Before rating the certainty of evidence, GRADE users should define the target of their certainty rating. GRADE recommends setting judgment thresholds defining what they consider a very accurate, accurate, inaccurate, and very inaccurate test. These thresholds should be set after considering consequences of testing and effects on people-important outcomes. GRADE's primary criterion for judging imprecision in test accuracy evidence is considering confidence intervals (i.e., CI approach) of absolute test accuracy results (true and false, positive, and negative results in a cohort of people). Based on the CI approach, when a CI appreciably crosses the predefined judgment threshold(s), one should consider rating down certainty of evidence by one or more levels, depending on the number of thresholds crossed. When the CI does not cross judgment threshold(s), GRADE suggests considering the sample size for an adequately powered test accuracy review (optimal or review information size [optimal information size (OIS)/review information size (RIS)]) in rating imprecision. If the combined sample size of the included studies in the review is smaller than the required OIS/RIS, one should consider rating down by one or more levels for imprecision. CONCLUSION: This paper extends previous GRADE guidance for rating imprecision in single test accuracy systematic reviews and guidelines, with a focus on the circumstances in which one should consider rating down one or more levels for imprecision.

Faculty Development on the Use of a Clinical Judgment Model in the Pre-licensure Nursing Curriculum.

Clinical judgment is an essential component of safe nursing practice that pre-licensure nursing students should develop by graduation from accredited nursing programs. For novice nurses, the consequences of underdeveloped clinical judgment skills that do not meet the demands of clinical practice are serious. This theory-practice gap correlates with increased numbers of errors occurring during care delivery, resulting in poorer patient outcomes. From a student perspective, this problem correlates with lower first-time pass rates on the NCLEX licensing exam. For nurse educators, there are uncertainties about how to resolve this complex and costly problem, but faculty development is one evidence-based solution to explore. The purpose of this article is to describe a three-pronged quality improvement project consisting of: 1) a faculty development session to teach faculty to implement the National Council State Boards of Nursing's Clinical Judgment Model (CJM) in their courses; 2) the use of the Clinical Judgment Tool; and 3) the implementation of a faculty champion to sustain and maintain ongoing faculty momentum to foster clinical judgment. This article focuses on how faculty can develop their own CJM faculty development session and use a CJM across the course curriculum.

The effect of task load, information reliability and interdependency on anticipation performance.

In sport, coaches often explicitly provide athletes with stable contextual information related to opponent action preferences to enhance anticipation performance. This information can be dependent on, or independent of, dynamic contextual information that only emerges during the sequence of play (e.g. opponent positioning). The interdependency between contextual information sources, and the associated cognitive demands of integrating information sources during anticipation, has not yet been systematically examined. We used a temporal occlusion paradigm to alter the reliability of contextual and kinematic information during the early, mid- and final phases of a two-versus-two soccer anticipation task. A dual-task paradigm was incorporated to investigate the impact of task load on skilled soccer players' ability to integrate information and update their judgements in each phase. Across conditions, participants received no contextual information (control) or stable contextual information (opponent preferences) that was dependent on, or independent of, dynamic contextual information (opponent positioning). As predicted, participants used reliable contextual and kinematic information to enhance anticipation. Further exploratory analysis suggested that increased task load detrimentally affected anticipation accuracy but only when both reliable contextual and kinematic information were available for integration in the final phase. This effect was observed irrespective of whether the stable contextual information was dependent on, or independent of, dynamic contextual information. Findings suggest that updating anticipatory judgements in the final phase of a sequence of play based on the integration of reliable contextual and kinematic information requires cognitive resources.

Do inconclusive forensic decisions disadvantage the innocent?

OBJECTIVE: Two experiments examined the potential for inconclusive forensic decisions to disadvantage the innocent. HYPOTHESES: Both experiments tested the hypothesis that inconclusive decisions produce more incriminating legal judgments than do clearly exculpatory forensic decisions. Experiment 2 also examined whether this hypothesized effect conformed to a confirmation bias, a communication error, or perceptual accuracy. METHOD: In Experiment 1 (N = 492), a forensic expert testified that physical evidence recovered from a crime scene either matched or did not match a suspect's evidence or produced an inconclusive result. In Experiment 2 (N = 1,002), a forensic expert testified that physical evidence recovered from a crime scene either matched or did not match a suspect's evidence, produced an inconclusive result, or was unsuitable for analysis. A fifth condition omitted the forensic evidence and expert testimony. RESULTS: The inconclusive decision produced less incriminating legal judgments than did the match forensic decision (|d|average = 0.96), more incriminating legal judgments than did the no-match forensic decision (|d|average = 0.62), and equivalent legal judgments to the unsuitable decision (|d|average = 0.12) and to legal judgments made in the absence of forensic evidence (|d|average = 0.07). These results suggest that participants interpreted the inconclusive decision to be forensically neutral, which is consistent with a communication error. CONCLUSION: The findings provide preliminary support for the idea that inconclusive decisions can put the innocent at risk of wrongful conviction by depriving them of a clearly exculpatory forensic decision. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The power of meaningful numbers: Attorney guidance and jury deliberation improve the reliability and gist validity of damage awards.

OBJECTIVE: A mock jury experiment tested the effects of attorney guidance and jury deliberation to mitigate the challenges that civil juries face in assessing damages. HYPOTHESES: We hypothesized that two types of attorney guidance (per diem, per diem + lump sum), theoretically based in the Hans-Reyna model of jury decision making, would improve jury decision making compared with no guidance against five key benchmarks: injury assessment, validity, reliability, verbatim-gist coherence, and metacognitive experience. We expected that deliberation would increase reliability of, confidence in, and polarization of awards compared with predeliberation. METHOD: Community members (N = 317; 61% women; 86.1% White; Mage = 48.68 years) deliberated in 54 mock juries. Participants watched a videotaped trial involving an automobile accident in which two plaintiffs sustained concussions (one mild and one severe). The plaintiffs' attorney's closing arguments varied attorney guidance (no guidance, per diem, per diem + lump sum). Mock jurors provided individual judgments before deliberating as a jury and reaching group verdicts and awards. RESULTS: Juries performed well against benchmarks. Providing gist-based guidance with a meaningful award recommendation increased the validity of jurors' individual damage awards (η²p jurors = .03) and the reliability of jury damage awards (η²p jurors = .04; η²p jurors = .20); gist-based guidance without an award recommendation did not improve performance against benchmarks and increased perceptions of decision-making difficulty (η²p = .13). Deliberation increased reliability of (η²p = .17), confidence in (η²p = .02), and polarization of (d = 2.14) awards compared with predeliberation. CONCLUSION: Juries performed well against objective benchmarks of performance (injury assessment, validity, reliability, and verbatim-gist coherence), and deliberation improved performance compared with predeliberation decisions. Jury decisions were further influenced by attorney closing arguments (the guidance manipulation), especially when the attorney requests a lump sum, which can serve as a powerful influence on jury awards, mainly by setting an upper limit. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Legal actors' and laypersons' utility judgments of eyewitness lineup procedures and outcomes.

OBJECTIVE: Recent attempts to model the relative performances of eyewitness lineup procedures necessarily include theoretical assumptions about the various costs/benefits, or utilities, of different identification outcomes. We collected data to incorporate empirically derived utilities into such modeling as well as data on various stakeholders' views of lineup procedures as tertiary objectives. HYPOTHESES: This research was exploratory; therefore, we did not have a priori hypotheses. METHOD: We surveyed judges' (n = 70), prosecutors' (n = 28), police officers' (n = 82), and laypersons' (n = 191) opinions about eyewitness identification procedures and the utilities of outcomes of eyewitness identification procedures. We incorporated the utility judgments into models comparing the desirability of various lineup reforms and compared policy preferences between our samples. RESULTS: All samples frequently mentioned estimator and system variables in open-ended evaluations of lineup procedures, but legal samples mentioned system variables more often than did laypersons. Reflector variables (e.g., confidence) were mentioned less often across the board, as was the scientific basis/standardization of identification policy (especially among laypersons). Utility judgments of various identification outcomes indicated that judges adopt values more closely aligned with normative legal ethics (i.e., the Blackstone ratio), whereas other stakeholders (especially laypersons) depart significantly from those standards. Utility models indicated general agreement among samples in lineup procedure preferences, which varied as a function of culprit-presence base rates. CONCLUSION: Although legal stakeholders vary in how they value eyewitness identification outcomes, their values imply relatively consistent policy preferences that sometimes depart from scientific recommendations. Nonetheless, all samples expressed support for using scientific research to inform legal policy regarding eyewitness evidence. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Probabilistic causal reasoning under time pressure.

While causal reasoning is a core facet of our cognitive abilities, its time-course has not received proper attention. As the duration of reasoning might prove crucial in understanding the underlying cognitive processes, we asked participants in two experiments to make probabilistic causal inferences while manipulating time pressure. We found that participants are less accurate under time pressure, a speed-accuracy-tradeoff, and that they respond more conservatively. Surprisingly, two other persistent reasoning errors-Markov violations and failures to explain away-appeared insensitive to time pressure. These observations seem related to confidence: Conservative inferences were associated with low confidence, whereas Markov violations and failures to explain were not. These findings challenge existing theories that predict an association between time pressure and all causal reasoning errors including conservatism. Our findings suggest that these errors should not be attributed to a single cognitive mechanism and emphasize that causal judgements are the result of multiple processes.

Neural correlates of metacognition: Disentangling the brain circuits underlying prospective and retrospective second-order judgments through noninvasive brain stimulation.

Metacognition encompasses the capability to monitor and control one's cognitive processes, with metamemory and metadecision configuring among the most studied higher order functions. Although imaging experiments evaluated the role of disparate brain regions, neural substrates of metacognitive judgments remain undetermined. The aim of this systematic review is to summarize and discuss the available evidence concerning the neural bases of metacognition which has been collected by assessing the effects of noninvasive brain stimulation (NIBS) on human subjects' metacognitive capacities. Based on such literature analysis, our goal is, at first, to verify whether prospective and retrospective second-order judgments are localized within separate brain circuits and, subsequently, to provide compelling clues useful for identifying new targets for future NIBS studies. The search was conducted following the preferred reporting items for systematic reviews and meta-analyses guidelines among PubMed, PsycINFO, PsycARTICLES, PSYNDEX, MEDLINE, and ERIC databases. Overall, 25 studies met the eligibility criteria, yielding a total of 36 experiments employing transcranial magnetic stimulation and 16 ones making use of transcranial electrical stimulation techniques, including transcranial direct current stimulation and transcranial alternating current stimulation. Importantly, we found that both perspective and retrospective judgments about both memory and perceptual decision-making performances depend on the activation of the anterior and lateral portions of the prefrontal cortex, as well as on the activity of more caudal regions such as the premotor cortex and the precuneus. Combining this evidence with results from previous imaging and lesion studies, we advance ventromedial prefrontal cortex as a promising target for future NIBS studies.

Looking at Mental Images: Eye-Tracking Mental Simulation During Retrospective Causal Judgment.

How do people evaluate causal relationships? Do they just consider what actually happened, or do they also consider what could have counterfactually happened? Using eye tracking and Gaussian process modeling, we investigated how people mentally simulated past events to judge what caused the outcomes to occur. Participants played a virtual ball-shooting game and then-while looking at a blank screen-mentally simulated (a) what actually happened, (b) what counterfactually could have happened, or (c) what caused the outcome to happen. Our findings showed that participants moved their eyes in patterns consistent with the actual or counterfactual events that they mentally simulated. When simulating what caused the outcome to occur, participants moved their eyes consistent with simulations of counterfactual possibilities. These results favor counterfactual theories of causal reasoning, demonstrate how eye movements can reflect simulation during this reasoning and provide a novel approach for investigating retrospective causal reasoning and counterfactual thinking.

The Lasater Clinical Judgment Rubric: 17 Years Later.

BACKGROUND: Nearly 17 years ago, the Lasater Clinical Judgment Rubric (LCJR) was published to provide a common language and trajectory of students' development to think like a nurse. METHOD: This article traces the uses of the LCJR from creation to the present and cites lessons learned from its use. RESULTS: During the intervening years, the LCJR has been used effectively as a debriefing guide in simulation and as a research instrument, as well as for formative assessment. The LCJR has been translated or is in process in 19 languages besides English. CONCLUSION: This article provides evidence of the efficacy of the LCJR and discusses important lessons learned. [J Nurs Educ. 2024;63(3):149-155.].

Evaluating the Importance of Clinical Judgment in Entry-Level Nursing.

BACKGROUND: This study investigated the extent to which clinical judgment is important in entry-level nursing practice. METHOD: A task analysis linkage study methodology based on a large-scale, comprehensive practice analysis was used for the study. The practice analysis validated more than 200 entry-level nursing tasks, and the linkage study sought to evaluate the importance of clinical judgment overall and for each specific task expectation. RESULTS: The results provide confirmatory evidence of the importance of clinical judgment. In addition, the work provides a comprehensive list of entry-level tasks and their associated clinical judgment importance. CONCLUSION: Clinical judgment is a vital skill at entry into the profession and is expected to grow over time. This study offers insights on the extent to which clinical judgment is a necessary skill and provides direct evidence of its importance for specific entry-level tasks. [J Nurs Educ. 2024;63(3):156-162.].

Reflective Clinical Judgment Questions to Educate the Next Generation of Nurses.

BACKGROUND: The complexity of health care requires entry-level nurses to have competent clinical judgment skills. In response, a nursing program created Reflective Clinical Judgment Questions (RCJQ) to guide students in the development of clinical judgment. METHOD: The RCJQ incorporates the Clinical Judgment Measurement Model, the National Council of State Boards of Nursing's action questions, and the American Association of Colleges of Nursing's core competencies for professional nursing education. The RCJQ includes cognitive process questions and self-reflection questions aligned to the prelicensure subcompetencies to direct student thinking and build a routine for clinical decision making. RESULTS: The RCJQ provides faculty with a framework to teach clinical judgment and incorporates self-reflective questions to guide decision making for safe and effective client care. CONCLUSION: The RCJQ streamlines the clinical judgment process and guides students to achieve essential outcomes in classroom, clinical, and simulation settings to prepare for clinical practice. [J Nurs Educ. 2024;63(3):182-185.].

Classroom Innovation Using the NCJMM: Preparing Nursing Students for the Next Generation NCLEX.

BACKGROUND: There is a growing demand for classroom creativity to increase engagement and build clinical judgment skills for nursing students. This article describes the design and implementation of an interactive classroom activity to enhance the development of clinical judgment while simultaneously orienting students to the new NCLEX Next Generation testing model. METHOD: Faculty developed an interactive unfolding case study incorporating the six dimensions of the NCSBN Clinical Judgment Measurement Model. The case study question types were adapted to use an interactive learning platform for first-year nursing students. Students' perceptions of learning, engagement, and clinical judgment were surveyed. RESULTS: Student responses regarding the case study implementation indicated this method was effective in maintaining engagement and persistence, as well as promoting nursing decision making. CONCLUSION: The time used for building innovative classroom activities is well spent to meet the objective of enhancing clinical judgment in the next generation of nursing students. [J Nurs Educ. 2024;63(3):188-191.].

Awe and time perception.

The aim of this study was to systematically examine the effect of awe-inducing stimuli on the judgment of time. Three experiments were conducted using temporal bisection tasks in which participants viewed awe-inducing and no awe-inducing images presented for different durations and were asked to judge whether their duration was similar to a short or long anchor duration. Images of panoramic landscapes and images of the faces of well-known and admired people were used in experiments 1 and 2, respectively. In experiment 3, they did not judge the duration of the images, but that of a neutral stimulus occurring during the presentation of images. In each experiment, participants rated the awe-inducing and no-awe-inducing images according to their components: admiration, beauty, awe, emotional valence, arousal, symbolic self-size, and full-body self-size. Results consistently showed significant time distortions when participants viewed the different awe-inducing images compared to the no-awe images, although the effect was weaker for the images of faces than for those of landscapes. Time distortion took the form of temporal lengthening in Experiments 1 and 2 and shortening in Experiment 3. These different temporal distortions are consistent with attention effects due to awe-inducing stimuli which capture attention to the detriment of time processing.

Children's and adults' evaluations of self-enhancement and self-deprecation depend on the usual performance of the self-presenter.

This study examined how evaluations of self-presentation vary with age depending on the self-presenter's usual performance. People's usual performance is a key factor because it generally influences the social evaluations and judgments that others make about them. Children aged 7 and 8 years (second graders) and 10 and 11 years (fifth graders), as well as adults, were presented with scenarios in which protagonists responded to praise after a good performance using either self-enhancement or self-deprecation. The other person in scenarios knew that the protagonist's usual performance on the task was either good or poor. After the protagonist responded to praise in a self-enhancing or self-deprecating way, the participants judged how the other person would evaluate the protagonist's ability (good/poor) and character (nice/mean). For ability evaluations specifically concerning protagonists who usually performed poorly, the results showed that by around 10 years of age children no longer tended to give more positive ability evaluations for self-enhancement than for self-deprecation. Adults gave less positive ability evaluations for self-deprecation than for self-enhancement, but only when the protagonists usually performed well. In relation to the character evaluations, by around 10 years of age self-enhancement led to less positive character evaluations than self-deprecation, but only when the protagonists usually performed poorly. Overall, second graders evaluated self-presenters as more competent and nicer. These results indicate that the expected evaluation of self-enhancement and self-deprecation is influenced by the usual level of performance but that there are developmental changes in this aspect of social cognition.

Don't be a rat: An investigation of the taboo against reporting other students for cheating.

This research examines barriers to reporting academic dishonesty in early adulthood (Study 1; N = 92) and adolescence (Study 2; N = 137). Participants were asked to describe a recent time they observed a peer cheating and to reflect on their decision about whether to report the cheating. They also responded to hypothetical scenarios about observing typical cheating actions, and the presence of social motives (e.g., whether people who report tend to gain reputations for being snitches) was manipulated in each scenario. Even though participants judged reporting to be the morally right thing to do, doing so was rare and approval for it was low, especially in adolescence. Participants also tended to say they would rather be friends with people who do not report cheaters than with those who do. Participants reasoned about a variety of social concerns to support their judgments about reporting (e.g., concern about their relationship with the cheater, concerns for others' welfare), and the manipulated social motives in the hypothetical scenarios significantly influenced judgments about reporting. These findings inform our understanding of the social dynamics that contribute to decisions about policing academic honesty.

Structural explanations lead young children and adults to rectify resource inequalities.

Decisions about how to divide resources have profound social and practical consequences. Do explanations regarding the source of existing inequalities influence how children and adults allocate new resources? When 3- to 6-year-old children (N = 201) learned that inequalities were caused by structural forces (stable external constraints affecting access to resources) as opposed to internal forces (effort), they rectified inequalities, overriding previously documented tendencies to perpetuate inequality or divide resources equally. Adults (N = 201) were more likely than children to rectify inequality spontaneously; this was further strengthened by a structural explanation but reversed by an effort-based explanation. Allocation behaviors were mirrored in judgments of which allocation choices by others were appropriate. These findings reveal how explanations powerfully guide social reasoning and action from childhood through adulthood.