Using latent variable models to make gaming-the-system detection robust to context variations.

Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with reduced learning, challenging its validity in our study context. Our exploratory data analysis suggested that varying contextual factors across and within conditions contributed to this lack of association. We present a new approach, latent variable-based gaming detection (LV-GD), that controls for contextual factors and more robustly estimates student-level latent gaming tendencies. In LV-GD, a student is estimated as having a high gaming tendency if the student is detected to game more than the expected level of the population given the context. LV-GD applies a statistical model on top of an existing action-level gaming detector developed based on a typical human labeling process, without additional labeling effort. Across three datasets, we find that LV-GD consistently outperformed the original detector in validity measured by association between gaming and learning as well as reliability. LV-GD also afforded high practical utility: it more accurately revealed intervention effects on gaming, revealed a correlation between gaming and perceived competence in math and helped understand productive detected gaming behaviors. Our approach is not only useful for others wanting a cost-effective way to adapt a gaming detector to their context but is also generally applicable in creating robust behavioral measures.

An astonishing regularity in student learning rate.

Leveraging a scientific infrastructure for exploring how students learn, we have developed cognitive and statistical models of skill acquisition and used them to understand fundamental similarities and differences across learners. Our primary question was why do some students learn faster than others? Or, do they? We model data from student performance on groups of tasks that assess the same skill component and that provide follow-up instruction on student errors. Our models estimate, for both students and skills, initial correctness and learning rate, that is, the increase in correctness after each practice opportunity. We applied our models to 1.3 million observations across 27 datasets of student interactions with online practice systems in the context of elementary to college courses in math, science, and language. Despite the availability of up-front verbal instruction, like lectures and readings, students demonstrate modest initial prepractice performance, at about 65% accuracy. Despite being in the same course, students' initial performance varies substantially from about 55% correct for those in the lower half to 75% for those in the upper half. In contrast, and much to our surprise, we found students to be astonishingly similar in estimated learning rate, typically increasing by about 0.1 log odds or 2.5% in accuracy per opportunity. These findings pose a challenge for theories of learning to explain the odd combination of large variation in student initial performance and striking regularity in student learning rate.

Testing Theories of Transfer Using Error Rate Learning Curves.

We analyze naturally occurring datasets from student use of educational technologies to explore a long-standing question of the scope of transfer of learning. We contrast a faculty theory of broad transfer with a component theory of more constrained transfer. To test these theories, we develop statistical models of them. These models use latent variables to represent mental functions that are changed while learning to cause a reduction in error rates for new tasks. Strong versions of these models provide a common explanation for the variance in task difficulty and transfer. Weak versions decouple difficulty and transfer explanations by describing task difficulty with parameters for each unique task. We evaluate these models in terms of both their prediction accuracy on held-out data and their power in explaining task difficulty and learning transfer. In comparisons across eight datasets, we find that the component models provide both better predictions and better explanations than the faculty models. Weak model variations tend to improve generalization across students, but hurt generalization across items and make a sacrifice to explanatory power. More generally, the approach could be used to identify malleable components of cognitive functions, such as spatial reasoning or executive functions.

Data mining and education.

An emerging field of educational data mining (EDM) is building on and contributing to a wide variety of disciplines through analysis of data coming from various educational technologies. EDM researchers are addressing questions of cognition, metacognition, motivation, affect, language, social discourse, etc. using data from intelligent tutoring systems, massive open online courses, educational games and simulations, and discussion forums. The data include detailed action and timing logs of student interactions in user interfaces such as graded responses to questions or essays, steps in rich problem solving environments, games or simulations, discussion forum posts, or chat dialogs. They might also include external sensors such as eye tracking, facial expression, body movement, etc. We review how EDM has addressed the research questions that surround the psychology of learning with an emphasis on assessment, transfer of learning and model discovery, the role of affect, motivation and metacognition on learning, and analysis of language data and collaborative learning. For example, we discuss (1) how different statistical assessment methods were used in a data mining competition to improve prediction of student responses to intelligent tutor tasks, (2) how better cognitive models can be discovered from data and used to improve instruction, (3) how data-driven models of student affect can be used to focus discussion in a dialog-based tutoring system, and (4) how machine learning techniques applied to discussion data can be used to produce automated agents that support student learning as they collaborate in a chat room or a discussion board.

Are diagrams always helpful tools? developmental and individual differences in the effect of presentation format on student problem solving.

BACKGROUND: High school and college students demonstrate a verbal, or textual, advantage whereby beginning algebra problems in story format are easier to solve than matched equations (Koedinger & Nathan, 2004). Adding diagrams to the stories may further facilitate solution (Hembree, 1992; Koedinger & Terao, 2002). However, diagrams may not be universally beneficial (Ainsworth, 2006; Larkin & Simon, 1987). AIMS: To identify developmental and individual differences in the use of diagrams, story, and equation representations in problem solving. When do diagrams begin to aid problem-solving performance? Does the verbal advantage replicate for younger students? SAMPLE: Three hundred and seventy-three students (121 sixth, 117 seventh, 135 eighth grade) from an ethnically diverse middle school in the American Midwest participated in Experiment 1. In Experiment 2, 84 sixth graders who had participated in Experiment 1 were followed up in seventh and eighth grades. METHOD: In both experiments, students solved algebra problems in three matched presentation formats (equation, story, story + diagram). RESULTS: The textual advantage was replicated for all groups. While diagrams enhance performance of older and higher ability students, younger and lower-ability students do not benefit, and may even be hindered by a diagram's presence. CONCLUSIONS: The textual advantage is in place by sixth grade. Diagrams are not inherently helpful aids to student understanding and should be used cautiously in the middle school years, as students are developing competency for diagram comprehension during this time.

The knowledge-learning-instruction framework: bridging the science-practice chasm to enhance robust student learning.

Despite the accumulation of substantial cognitive science research relevant to education, there remains confusion and controversy in the application of research to educational practice. In support of a more systematic approach, we describe the Knowledge-Learning-Instruction (KLI) framework. KLI promotes the emergence of instructional principles of high potential for generality, while explicitly identifying constraints of and opportunities for detailed analysis of the knowledge students may acquire in courses. Drawing on research across domains of science, math, and language learning, we illustrate the analyses of knowledge, learning, and instructional events that the KLI framework affords. We present a set of three coordinated taxonomies of knowledge, learning, and instruction. For example, we identify three broad classes of learning events (LEs): (a) memory and fluency processes, (b) induction and refinement processes, and (c) understanding and sense-making processes, and we show how these can lead to different knowledge changes and constraints on optimal instructional choices.

Trade-offs between grounded and abstract representations: evidence from algebra problem solving.

This article explores the complementary strengths and weaknesses of grounded and abstract representations in the domain of early algebra. Abstract representations, such as algebraic symbols, are concise and easy to manipulate but are distanced from any physical referents. Grounded representations, such as verbal descriptions of situations, are more concrete and familiar, and they are more similar to physical objects and everyday experience. The complementary computational characteristics of grounded and abstract representations lead to trade-offs in problem-solving performance. In prior research with high school students solving relatively simple problems, Koedinger and Nathan (2004) demonstrated performance benefits of grounded representations over abstract representations-students were better at solving simple story problems than the analogous equations. This article extends this prior work to examine both simple and more complex problems in two samples of college students. On complex problems with two references to the unknown, a "symbolic advantage" emerged, such that students were better at solving equations than analogous story problems. Furthermore, the previously observed "verbal advantage" on simple problems was replicated. We thus provide empirical support for a trade-off between grounded, verbal representations, which show advantages on simpler problems, and abstract, symbolic representations, which show advantages on more complex problems.

Cognitive tutor: applied research in mathematics education.

For 25 years, we have been working to build cognitive models of mathematics, which have become a basis for middle- and high-school curricula. We discuss the theoretical background of this approach and evidence that the resulting curricula are more effective than other approaches to instruction. We also discuss how embedding a well specified theory in our instructional software allows us to dynamically evaluate the effectiveness of our instruction at a more detailed level than was previously possible. The current widespread use of the software is allowing us to test hypotheses across large numbers of students. We believe that this will lead to new approaches both to understanding mathematical cognition and to improving instruction.

Behavioral equivalence, but not neural equivalence--neural evidence of alternative strategies in mathematical thinking.

In a functional magnetic resonance imaging study, we investigated how people solve mathematically equivalent problems presented in two alternative formats: verbal, story format or symbolic, equation format. Although representation format had no effect on behavior, anterior prefrontal activation was greater in the story condition and posterior parietal activation was greater in the equation condition. These results show that there exist alternative neural pathways that implement different and yet equally efficient problem-solving strategies.