Using Latent Semantic Analysis to Score Short Answer Constructed Responses: Automated Scoring of the Consequences Test.

Automated scoring based on Latent Semantic Analysis (LSA) has been successfully used to score essays and constrained short answer responses. Scoring tests that capture open-ended, short answer responses poses some challenges for machine learning approaches. We used LSA techniques to score short answer responses to the Consequences Test, a measure of creativity and divergent thinking that encourages a wide range of potential responses. Analyses demonstrated that the LSA scores were highly correlated with conventional Consequence Test scores, reaching a correlation of .94 with human raters and were moderately correlated with performance criteria. This approach to scoring short answer constructed responses solves many practical problems including the time for humans to rate open-ended responses and the difficulty in achieving reliable scoring.

Traffic crash involvement: experiential driving knowledge and stressful contextual antecedents.

Researchers have rarely examined stressful environments and psychological characteristics as predictors of driving behavior in the same study. The authors hypothesized that (a) safer drivers more accurately assess physical and emotional traffic hazards and (b) stress and emotional states elevate crash risk. The hypotheses were evaluated with procedural and declarative tacit driving knowledge tests requiring assessment of emotional and contextual hazards and with accident reports describing crash antecedents, including stressful events and environmental conditions. Analyses identified separate driving knowledge factors corresponding to emotional and contextual hazards that were significantly related to the crash criteria. Accident report analyses show that stress significantly elevates at-fault crash risk. The results demonstrate the importance of experiential knowledge acquired without instruction (procedural or tacit knowledge) and provide safety recommendations.

Toward an understanding of situational judgment item validity and group differences.

This paper evaluates 2 adjustments to common scoring approaches for situational judgment tests (SJTs). These adjustments can result in substantial improvements to item validity, reductions in mean racial differences, and resistance to coaching designed to improve scores. The first adjustment, applicable to SJTs that use Likert scales, controls for elevation and scatter (Cronbach & Gleser, 1953). This adjustment improves item validity. Also, because there is a White-Black mean difference in the preference for extreme responses on Likert scales (Bachman & O'Malley, 1984), these adjustments substantially reduce White-Black mean score differences. Furthermore, this adjustment often eliminates the score elevation associated with the coaching strategy of avoiding extreme responses (Cullen, Sackett, & Lievens, 2006). Item validity is shown to have a U-shaped relationship with item means. This holds both for SJTs with Likert score response formats and for SJTs where respondents identify the best and worst response option. Given the U-shaped relationship, the second adjustment is to drop items with midrange item means. This permits the SJT to be shortened, sometimes dramatically, without necessarily harming validity.