All prompts are created equal, but some prompts are more equal than others.

ABSTRACT

Scores assigned to college placement essays by a computer program (PEG) showed high agreement with the evaluations of human readers (r =.82). Further, both types of graders tended to assign higher or lower scores to essays written about particular topics. Content analyses by a second program (MCCA) indicated that themes in essays varied in terms of emphasis on "analytic," "emotional," or "practical" dimensions. Human and machine readers tended to give higher scores for analytic and practical themes, and lower scores for those involving emotion. The ranks of mean prompt-related grades were concordant with the ranks of mean analytic and practical content across topics. Such findings call for the refined standardization of prompts for future testing, and the need for care in the evaluation of existing essays.