Designing and evaluating complex educational research to meet federal grant requirement: A demonstration of a funded large-scale protocol.

Federal funding agencies have invested significant resources supporting evidence-based, innovative approaches that address education problems and incorporate rigorous design and evaluation, particularly through randomized controlled trials (RCTs), the gold standard for yielding causal inference in scientific research. In this study, we introduced factors (i.e., evaluation design, attrition, outcome measures, analytic approach, and fidelity of implementation) that are often times required in the Federal Notice for application by the U.S. Department of Education, with an emphasis on What Works Clearinghouse (WWC) standards. We further presented a federally funded research protocol with a multi-year, clustered RCT design to determine the impact of an instructional intervention on students' academic performance in high-needs schools. In the protocol, we elaborated on how our research design, evaluation plan, power analysis, as well as confirmatory research questions and analytical approaches were aligned with the grant requirement and WWC standards. We intend to provide a road map to meeting WWC standards and to increase the likelihood of successful grant applications.

A Comparative Study of Problem-Based Learning and Traditional Approaches in College English Classrooms: Analyzing Pedagogical Behaviors Via Classroom Observation.

In this study, we described and compared an English as a foreign language (EFL) teacher's pedagogical behaviors in traditional and problem-based learning (PBL) classroom settings in a Chinese university. In spring 2019, we collected six 45-min videos, three in each condition, covering three modules: (a) warm-up and vocabulary, (b) essay structure, and (c) writing. The analyses of the teacher's pedagogical behaviors and her interaction with students indicated that the instructor spent most of the instructional time delivering higher-order thinking content in both traditional and PBL classes. The teacher's activity structure influenced students' communication mode. Although the instructor provided students with more group discussion activities in the PBL classroom, lecturing was observed to be the primary delivery method in both classes. These results suggest that the application of PBL strategies in the EFL classroom did not significantly restructure the teacher's pedagogical behaviors, and thus, failed to achieve the goal of providing students with more opportunities for improving their expressive English language proficiency. These findings underscore the need to develop an effective PBL-related curriculum and professional development opportunities for EFL teachers to effectively implement the PBL approach in the classroom.

Inter-rater reliability data of classroom observation: Fidelity in large-scale randomized research in education.

This dataset belongs to a large-scale randomized controlled trial (RCT) in educational research targeting English learning students and their teachers' instructional capacity. The dataset includes ratings conducted through classroom observations of 45-minute English as a Second language (ESL) blocks. Each coder rated 60 recorded video segments collected from each teacher. During the 20-second segment, ratings of six domains of teachers' instruction (i.e., ESL Strategies, Group, Activity Structure, Mode, Language Content, Language of Teacher, Language of Student) were collected. The dataset is organized by teacher, by coder, and by domain, for researchers to analyze inter-rater reliability among coders by domain and/or cross-domain. This data article is related to the research article Tong et al. [3] on "The determination of appropriate coefficient indices for inter-rater reliability: using classroom observation instruments as fidelity measures in large-scale randomized research".

Analyzing Complex Longitudinal Data in Educational Research: A Demonstration With Project English Language and Literacy Acquisition (ELLA) Data Using xxM.

When analyzing complex longitudinal data, especially data from different educational settings, researchers generally focus only on the mean part (i.e., the regression coefficients), ignoring the equally important random part (i.e., the random effect variances) of the model. By using Project English Language and Literacy Acquisition (ELLA) data, we demonstrated the importance of taking the complex data structure into account by carefully specifying the random part of the model, showing that not only can it affect the variance estimates, the standard errors, and the tests of significance of the regression coefficients, it also can offer different perspectives of the data, such as information related to the developmental process. We used xxM (Mehta, 2013), which can flexibly estimate different grade-level variances separately and the potential carryover effect from each grade factor to the later time measures. Implications of the findings and limitations of the study are discussed.