Automated feedback and writing: a multi-level meta-analysis of effects on students' performance.

Introduction: Adaptive learning opportunities and individualized, timely feedback are considered to be effective support measures for students' writing in educational contexts. However, the extensive time and expertise required to analyze numerous drafts of student writing pose a barrier to teaching. Automated writing evaluation (AWE) tools can be used for individual feedback based on advances in Artificial Intelligence (AI) technology. A number of primary (quasi-)experimental studies have investigated the effect of AWE feedback on students' writing performance. Methods: This paper provides a meta-analysis of the effectiveness of AWE feedback tools. The literature search yielded 4,462 entries, of which 20 studies (k = 84; N = 2, 828) met the pre-specified inclusion criteria. A moderator analysis investigated the impact of the characteristics of the learner, the intervention, and the outcome measures. Results: Overall, results based on a three-level model with random effects show a medium effect (g = 0.55) of automated feedback on students' writing performance. However, the significant heterogeneity in the data indicates that the use of automated feedback tools cannot be understood as a single consistent form of intervention. Even though for some of the moderators we found substantial differences in effect sizes, none of the subgroup comparisons were statistically significant. Discussion: We discuss these findings in light of automated feedback use in educational practice and give recommendations for future research.

A closer look at the domain-specific associations of openness with language achievement: Evidence on the role of intrinsic value from two large-scale longitudinal studies.

BACKGROUND: Many empirical investigations focus on how personality traits and academic motivation are related to academic achievement. Regarding the personality traits described in the five-factor model, prior research has shown associations between openness to experience and language achievement in particular. Following the principle of trait activation, associations with intrinsic value can explain these domain-specific relationships of openness with achievement. AIMS: This study investigated whether intrinsic value mediates the associations of openness to experience with language achievement to understand these associations more closely. Based on the principle of trait activation, we combined openness to experience with intrinsic value as a motivational construct more proximal to behaviour. SAMPLES: We capitalized on two large-scale longitudinal data sets (N = 1994; M = 16.8 years; 51% female; and N = 2722; M = 17.34 years; 58% female) of German-speaking students in upper secondary education with two points of measurement, respectively. METHODS: We assessed achievement using report card grades and standardized test scores, capturing three language subskills: reading and listening, measured by standardized tests, and writing competencies, measured by essay assignments. We conducted mediation analysis using bootstrapped confidence intervals and robust standard errors to account for dependencies in the data due to students clustered within classrooms. RESULTS AND CONCLUSIONS: Overall, our results show that intrinsic value mediated the relationship between openness and achievement in English as a foreign language. The results remained stable when controlling for prior achievement, predicting achievement gains. Our investigation provides empirical insights into the manifestations of personality in academic contexts. We discuss the limitations of our approach, with a focus on the use of a mediation analysis in this research design.

Read at home to do well at school: informal reading predicts achievement and motivation in English as a foreign language.

Introduction: Learning English as a foreign language is necessary for many students to actively participate in an increasingly globalized world. This study explores the role of informal English language engagement for students' reading and listening skills, as well as motivation to learn English. In an era of global interconnectedness, informal learning gains importance as a supplement to formal education. Methods: This study extends the evidence base by analyzing extramural reading and listening activities in a large-scale longitudinal investigation involving secondary school learners (N = 1,994) in Germany. Results: Our results show that frequent informal reading significantly relates to increases in students' English comprehension skills and their motivation for language learning, reaffirming previous cross-sectional findings. Discussion: The results highlight the relevance of informal language activities for effective language learning and students' English as a foreign language motivation. Additionally, discrepancies between reading and listening outcomes are discussed.

Is a Long Essay Always a Good Essay? The Effect of Text Length on Writing Assessment.

The assessment of text quality is a transdisciplinary issue concerning the research areas of educational assessment, language technology, and classroom instruction. Text length has been found to strongly influence human judgment of text quality. The question of whether text length is a construct-relevant aspect of writing competence or a source of judgment bias has been discussed controversially. This paper used both a correlational and an experimental approach to investigate this question. Secondary analyses were performed on a large-scale dataset with highly trained raters, showing an effect of text length beyond language proficiency. Furthermore, an experimental study found that pre-service teachers tended to undervalue text length when compared to professional ratings. The findings are discussed with respect to the role of training and context in writing assessment.