Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability, Validity, and Feasibility.

Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool-with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts.

A Comparison of Strategies for Recruiting Teachers Into Survey Panels.

We examine a range of options for recruiting teachers into a nationally representative survey panel. Recruitment strategies considered include a telephone-based approach and the use of promised incentives and pre-incentives of varying amounts and forms. Using a randomized experiment, we evaluate the effectiveness of five separate recruitment strategies and conduct a cost-benefit analysis. Our preferred strategy is one that uses a US$10 gift card as pre-incentive (it yielded a 27% rate of successful recruitment a cost of US$78 per recruited teacher). Statistical comparisons indicate that no other technique was superior to this strategy in terms of recruitment rate or cost-effectiveness. Efforts at refusal conversion after the initial approach were mostly ineffective. A comparison across demographic type characteristics of enrolled panelists and nonrespondents shows no substantial differences for any recruitment strategy considered. Hence, the potential for recruitment-level nonresponse to induce large bias into findings from surveys administered to the panel is minimal.

Toward a Culture of Consequences: Performance-Based Accountability Systems for Public Services.

Performance-based accountability systems (PBASs), which link incentives to measured performance as a means of improving services to the public, have gained popularity. While PBASs can vary widely across sectors, they share three main components: goals, incentives, and measures. Research suggests that PBASs influence provider behaviors, but little is known about PBAS effectiveness at achieving performance goals or about government and agency experiences. This study examines nine PBASs that are drawn from five sectors: child care, education, health care, public health emergency preparedness, and transportation. In the right circumstances, a PBAS can be an effective strategy for improving service delivery. Optimum circumstances include having a widely shared goal, unambiguous observable measures, meaningful incentives for those with control over the relevant inputs and processes, few competing interests, and adequate resources to design, implement, and operate the PBAS. However, these conditions are rarely fully realized, so it is difficult to design and implement PBASs that are uniformly effective. PBASs represent a promising policy option for improving the quality of service-delivery activities in many contexts. The evidence supports continued experimentation with and adoption of this approach in appropriate circumstances. Even so, PBAS design and its prospects for success depend on the context in which it will operate. Also, ongoing system evaluation and monitoring are integral components of a PBAS; they inform refinements that improve system functioning over time. Empirical evidence of the effects of performance-based public management is scarce. This article also describes a framework used to evaluate a PBAS. Such a system identifies individuals or organizations that must change their behavior for the performance of an activity to improve, chooses an implicit or explicit incentive structure to motivate these organizations or individuals to change, and then chooses performance measures tailored to inform the incentive structure appropriately. The study focused on systems in the child care, education, health care, public health emergency preparedness, and transportation sectors, mainly in the United States. Analysts could use this framework to seek empirical information in other sectors and other parts of the world. Additional empirical information could help refine existing PBASs and, more broadly, improve decisions on where to initiate new PBASs, how to implement them, and then how to design, manage, and refine them over time.