Program Evaluation Standards for Utility Facilitate Stakeholder Internalization of Evaluative Thinking in the West Virginia Clinical Translational Science Institute.

Background: The program evaluation standards (PES) can be considered established criteria for high-quality evaluations. We emphasize PES Utility standards and evaluation capacity building as we strive for meaningful application of our work in the real world. Purpose: We focused our methodology on understanding how stakeholders discussed utility and how their perceptions related to our evaluation work aligned with the Utility domain of the program evaluation standards. Setting: The West Virginia Clinical Translational Science Institute (WVCTSI), a statewide multi-institutional entity for which we have conducted tracking and evaluation since 2012. Intervention: Sustained collaborative engagement of evaluation stakeholders with the goal of increasing their utilization of evaluation products and evaluative thinking. Research Design: Case study. Data Collection and Analysis: We interviewed five key stakeholders. We used themes developed from coding of interview data to inform document analyses. We used interview and document analyses to develop additional themes and illustrative examples, as well as to develop and describe a five-level evaluation uptake scale. Findings: We describe shifts in initiation, use, and internalization of evaluative thinking by non-evaluation personnel. These shifts prompted our development and application of an evaluation uptake scale to capture increased evaluation capacity among stakeholders over time. We discuss how focus on the PES Utility standards and evaluation capacity building facilitated these shifts, and their implications for maximizing utility of evaluation activity in large, complex programmatic evaluations.

General Chemistry Student Attitudes and Success with Use of Online Homework: Traditional-Responsive versus Adaptive-Responsive.

We investigated whether use of an adaptive-responsive online homework system (OHS) that tailors homework to students' prior knowledge and periodically reassesses students to promote learning through practice retrieval has inherent advantages over traditional-responsive online homework. A quasi-experimental cohort control post-test-only design with nonequivalent groups and propensity scores with nearest neighbor matching (n = 6,114 pairs) was used. The adaptive system was found to increase the odds of a higher final letter grade for average, below average, and failing students. However, despite the learning advantages, students self-reported less favorable attitudes toward adaptive-responsive (3.15 of 5) relative to traditional-responsive OHS (3.31). Specific to the adaptive OHS, the following were found: (i) student attitudes were moderately and positively correlated (r = 0.36, p < 0.01) to final letter grade, (ii) most students (95%) reported engaging in remediation of incorrect responses, (iii) a majority of students (69%) reported changes in study habits, and (iv) students recognized the benefit of using adaptive OHS by ranking its assignments and explanations or review materials as two of the top three most useful course aspects contributing to perceived learning. Instructors can use our findings to inform their choice of online homework system for formative assessment of chemistry learning by weighing the benefits, disadvantages, and learning pedagogies of traditional-responsive versus adaptive-responsive systems.