Multiple-choice pretesting potentiates learning of related information.

Although the testing effect has received a substantial amount of empirical attention, such research has largely focused on the effects of tests given after study. The present research examines the effect of using tests prior to study (i.e., as pretests), focusing particularly on how pretesting influences the subsequent learning of information that is not itself pretested but that is related to the pretested information. In Experiment 1, we found that multiple-choice pretesting was better for the learning of such related information than was cued-recall pretesting or a pre-fact-study control condition. In Experiment 2, we found that the increased learning of non-pretested related information following multiple-choice testing could not be attributed to increased time allocated to that information during subsequent study. Last, in Experiment 3, we showed that the benefits of multiple-choice pretesting over cued-recall pretesting for the learning of related information persist over 48 hours, thus demonstrating the promise of multiple-choice pretesting to potentiate learning in educational contexts. A possible explanation for the observed benefits of multiple-choice pretesting for enhancing the effectiveness with which related nontested information is learned during subsequent study is discussed.

Optimizing multiple-choice tests as tools for learning.

Answering multiple-choice questions with competitive alternatives can enhance performance on a later test, not only on questions about the information previously tested, but also on questions about related information not previously tested-in particular, on questions about information pertaining to the previously incorrect alternatives. In the present research, we assessed a possible explanation for this pattern: When multiple-choice questions contain competitive incorrect alternatives, test-takers are led to retrieve previously studied information pertaining to all of the alternatives in order to discriminate among them and select an answer, with such processing strengthening later access to information associated with both the correct and incorrect alternatives. Supporting this hypothesis, we found enhanced performance on a later cued-recall test for previously nontested questions when their answers had previously appeared as competitive incorrect alternatives in the initial multiple-choice test, but not when they had previously appeared as noncompetitive alternatives. Importantly, however, competitive alternatives were not more likely than noncompetitive alternatives to be intruded as incorrect responses, indicating that a general increased accessibility for previously presented incorrect alternatives could not be the explanation for these results. The present findings, replicated across two experiments (one in which corrective feedback was provided during the initial multiple-choice testing, and one in which it was not), thus strongly suggest that competitive multiple-choice questions can trigger beneficial retrieval processes for both tested and related information, and the results have implications for the effective use of multiple-choice tests as tools for learning.

Multiple-choice tests stabilize access to marginal knowledge.

Marginal knowledge refers to knowledge that is stored in memory, but is not accessible at a given moment. For example, one might struggle to remember who wrote The Call of the Wild, even if that knowledge is stored in memory. Knowing how best to stabilize access to marginal knowledge is important, given that new learning often requires accessing and building on prior knowledge. While even a single opportunity to restudy marginal knowledge boosts its later accessibility (Berger, Hall, & Bahrick, 1999), in many situations explicit relearning opportunities are not available. Our question is whether multiple-choice tests (which by definition expose the learner to the correct answers) can also serve this function and, if so, how testing compares to restudying given that tests can be particularly powerful learning devices (Roediger & Karpicke, 2006). In four experiments, we found that multiple-choice testing had the power to stabilize access to marginal knowledge, and to do so for at least up to a week. Importantly, such tests did not need to be paired with feedback, although testing was no more powerful than studying. Overall, the results support the idea that one's knowledge base is unstable, with individual pieces of information coming in and out of reach. The present findings have implications for a key educational challenge: ensuring that students have continuing access to information they have learned.

Can Multiple-Choice Testing Induce Desirable Difficulties? Evidence from the Laboratory and the Classroom.

The term desirable difficulties (Bjork, 1994) refers to conditions of learning that, though often appearing to cause difficulties for the learner and to slow down the process of acquisition, actually improve long-term retention and transfer. One known desirable difficulty is testing (as compared with restudy), although typically it is tests that clearly involve retrieval--such as free and cued recall tests--that are thought to induce these learning benefits and not multiple-choice tests. Nonetheless, multiple-choice testing is ubiquitous in educational settings and many other high-stakes situations. In this article, we discuss research, in both the laboratory and the classroom, exploring whether multiple-choice testing can also be fashioned to promote the type of retrieval processes known to improve learning, and we speculate about the necessary properties that multiple-choice questions must possess, as well as the metacognitive strategy students need to use in answering such questions, to achieve this goal.

Expecting to teach enhances learning and organization of knowledge in free recall of text passages.

The present research assessed the potential effects of expecting to teach on learning. In two experiments, participants studied passages either in preparation for a later test or in preparation for teaching the passage to another student who would then be tested. In reality, all participants were tested, and no one actually engaged in teaching. Participants expecting to teach produced more complete and better organized free recall of the passage (Experiment 1) and, in general, correctly answered more questions about the passage than did participants expecting a test (Experiment 1), particularly questions covering main points (Experiment 2), consistent with their having engaged in more effective learning strategies. Instilling an expectation to teach thus seems to be a simple, inexpensive intervention with the potential to increase learning efficiency at home and in the classroom.

Metacognitive Management of Attention in Online Learning.

Performance during training is a poor predictor of long-term retention. Worse yet, conditions of training that produce rapidly improving performance typically do not produce long-lasting, generalizable learning. As a result, learners and instructors alike can be misled into adopting training or educational experiences that are suboptimal for producing actual learning. Computer-based educational training platforms can counter this unfortunate tendency by providing only productive conditions of instruction-even if they are unintuitive (e.g., spacing instead of massing). The use of such platforms, however, introduces a different liability: being easy to interrupt. An assessment of this possible liability is needed given the enormous disruption to modern education brought about by COVID-19 and the subsequent widespread emergency adoption of computer-based remote instruction. The present study was therefore designed to (a) explore approaches for detecting interruptions that can be reasonably implemented by an instructor, (b) determine the frequency at which students are interrupted during a cognitive-science-based digital learning experience, and (c) establish the extent to which the pandemic and ensuing lockdowns affected students' metacognitive ability to maintain engagement with their digital learning experiences. Outliers in time data were analyzed with increasing complexity and decreasing subjectivity to identify when learners were interrupted. Results indicated that only between 1.565% and 3.206% of online interactions show evidence of learner interruption. And although classroom learning was inarguably disrupted by the pandemic, learning in the present, evidence-based platform appeared to be immune.

Why interleaving enhances inductive learning: the roles of discrimination and retrieval.

Kornell and Bjork (Psychological science 19:585-592, 2008) found that interleaving exemplars of different categories enhanced inductive learning of the concepts based on those exemplars. They hypothesized that the benefit of mixing exemplars from different categories is that doing so highlights differences between the categories. Kang and Pashler (Applied cognitive psychology 26:97-103, 2012) obtained results consistent with this discriminative-contrast hypothesis: Interleaving enhanced inductive learning, but temporal spacing, which does not highlight category differences, did not. We further tested the discriminative-contrast hypothesis by examining the effects of interleaving and spacing, as well as their combined effects. In three experiments, using photographs of butterflies and birds as the stimuli, temporal spacing was harmful when it interrupted the juxtaposition of interleaved categories, even when total spacing was held constant, supporting the discriminative-contrast hypothesis. Temporal spacing also had value, however, when it did not interrupt discrimination processing.

Multiple-choice tests exonerated, at least of some charges: fostering test-induced learning and avoiding test-induced forgetting.

Among the criticisms of multiple-choice tests is that-by exposing the correct answer as one of the alternatives-such tests engage recognition processes rather than the productive retrieval processes known to enhance later recall. We tested whether multiple-choice tests could trigger productive retrieval processes-provided the alternatives were made plausible enough to enable test takers to retrieve both why the correct alternatives were correct and why the incorrect alternatives were incorrect. In two experiments, we found not only that properly constructed multiple-choice tests can indeed trigger productive retrieval processes, but also that they had one potentially important advantage over cued-recall tests. Both testing formats fostered retention of previously tested information, but multiple-choice tests also facilitated recall of information pertaining to incorrect alternatives, whereas cued-recall tests did not. Thus, multiple-choice tests can be constructed so that they exercise the very retrieval processes they have been accused of bypassing.

The costs and benefits of testing text materials.

Tests have been shown to improve the later recall of tested information, a result known as the testing effect. Tests, however, can also impair the later recall of related information, an effect known as retrieval-induced forgetting. Although retrieval-induced forgetting has been demonstrated using a wide variety of materials, recent work suggests that learning information in the context of a coherent text passage may afford protection from retrieval-induced forgetting. In four experiments we explored the conditions under which retrieval-induced forgetting does and does not occur with such materials. We found that two factors-the coherence of the to-be-learned material and the competitiveness of retrieval practice-are important in determining whether retrieval-induced forgetting does or does not occur. Furthermore, even if retrieval-induced forgetting does occur, having the opportunity to restudy the forgotten information can prevent that forgetting from persisting. Taken together, these findings provide greater understanding of the costs and benefits of testing text materials, with possible implications for the optimisation of testing as a tool for learning in educational contexts.

Improving conceptual learning via pretests.

Although examples can be structured to emphasize diagnostic features of concepts, novice learners tend to focus on irrelevant surface features and struggle to encode deeper structures. Experiment 1 examined whether pretesting-answering questions about content before it is studied-could enhance learners' noticing of diagnostic features, making them easier to process during subsequent study. Participants studied statistical concepts with examples that emphasized surface details or deep structure, and then classified new examples of these concepts. Studying examples that emphasized deep structure increased classification performance compared to examples that emphasized surface details. Moreover, taking pretests prior to studying the examples increased classification performance and eliminated differential benefits of studying structure versus surface examples. Experiment 2 examined whether pretesting serves a role beyond directing attention. After studying different statistical concepts with only surface-emphasizing examples, classification performance was better when participants actually took pretests compared to being given the correct responses. It is the generative aspect of pretesting, beyond attention directing, that improves conceptual learning among novice learners. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Where and how to learn: The interactive benefits of contextual variation, restudying, and retrieval practice for learning.

Students are often advised to do all of their studying in one good place, but restudying to-be-learned material in a new context can enhance subsequent recall. We examined whether there are similar benefits for testing. In Experiment 1 (n = 106), participants studied a 36-word list and 48 hr later-when back in the same or a new context-either restudied or recalled the list without feedback. After another 48 hr, all participants free-recalled the list in a new context. Experiment 2 (n = 203) differed by having the testing-condition participants restudy the list before being tested. Across both experiments, testing in a new context reduced recall, which carried over to the final test, whereas restudying in a new context did not impair (and in Experiment 2, significantly enhanced) recall. These findings reveal critical interactions between contextual-variation and retrieval-practice effects, which we interpret as consistent with a distribution-of-memory-strengths framework.

Accelerated relearning after retrieval-induced forgetting: the benefit of being forgotten.

Research on retrieval-induced forgetting has demonstrated that retrieving some information from memory can cause the forgetting of other information in memory. Here, the authors report research on the relearning of items that have been subjected to retrieval-induced forgetting. Participants studied a list of category- exemplar pairs, underwent a series of retrieval-practice and relearning trials, and, finally, were tested on the initially studied pairs. The final recall of non-relearned items exhibited a cumulative effect of retrieval-induced forgetting such that the size of the effect increased with each block of retrieval practice. Of most interest, and very surprising from a common-sense standpoint, items that were relearned benefited more from that relearning if they had previously been forgotten. The results offer insights into the nature and durability of retrieval-induced forgetting and provide additional evidence that forgetting is an enabler--rather than a disabler--of future learning.

How should exemplars be sequenced in inductive learning? Empirical evidence versus learners' opinions.

The sequencing of exemplars during study can have a large effect on category or concept induction. Counter to learners' intuitions, interleaving exemplars from different categories is often more effective for learning the different underlying categories than is blocking all the exemplars by category (e.g., Kornell & Bjork, 2008). Prior research suggests that blocking and interleaving each support different aspects of induction: Interleaving appears to enhance between-category discrimination, whereas blocking appears to promote the learning of within-category commonalities. In Experiments 1 and 2, participants studied paintings by 12 artists and were asked to induce the different artists' painting styles. We explored whether hybrid schedules can leverage the benefits of both types of schedules, comparing blocked, interleaved, and 3 hybrid schedules-blocked-to-interleaved, interleaved-to-blocked, and miniblocks. The miniblocks and blocked-to-interleaved schedules were as effective, statistically, but not better than pure interleaving. The blocked schedule led to the worst performance. In Experiments 3 and 4, we explored participants' a priori beliefs by having them self-schedule hypothetical future category-learning tasks. Although participants demonstrated some metacognitive sophistication with respect to the relative benefits of blocked and interleaved study, pure interleaving was the least popular schedule, despite its being one of the most, effective schedules for learning. (PsycINFO Database Record

On the learning benefits of confidence-weighted testing.

Taking multiple-choice practice tests with competitive incorrect alternatives can enhance performance on related but different questions appearing on a later cued-recall test (Little et al., Psychol Sci 23:1337-1344, 2012). This benefit of multiple-choice testing, which does not occur when the practice test is a cued-recall test, appears attributable to participants attempting to retrieve not only why the correct alternative is correct but also why the other alternatives are incorrect. The present research was designed to examine whether a confidence-weighted multiple-choice format in which test-takers were allowed to indicate their relative confidence in the correctness of one alternative compared with the others (Bruno, J Econ Educ 20:5-22, 1989; Bruno, Item banking: Interactive testing and self-assessment: Volume 112 of NATO ASI Series, pp. 190-209, 1993) might increase the extent to which participants engaged in such productive retrievals. In two experiments, such confidence-weighted practice tests led to greater benefits in the ability of test-takers to answer new but related questions than did standard multiple-choice practice tests. These results point to ways to make multiple-choice testing a more powerful tool for learning.

On the difficulty of mending metacognitive illusions: A priori theories, fluency effects, and misattributions of the interleaving benefit.

Interleaving exemplars of to-be-learned categories-rather than blocking exemplars by category-typically enhances inductive learning. Learners, however, tend to believe the opposite, even after their own performance has benefited from interleaving. In Experiments 1 and 2, the authors examined the influence of 2 factors that they hypothesized contribute to the illusion that blocking enhances inductive learning: Namely, that (a) blocking creates a sense of fluent extraction during study of the features defining a given category, and (b) learners come to the experimental task with a pre-existing belief that blocking instruction by topic is superior to intermixing topics. In Experiments 3-5, the authors attempted to uproot learners' belief in the superiority of blocking through experience-based and theory-based debiasing techniques by (a) providing detailed theory-based information as to why blocking seems better, but is not, and (b) explicitly drawing attention to the link between study schedule and subsequent performance, both of which had only modest effects. Only when they disambiguated test performance on the 2 schedules by separating them (Experiment 6) did the combination of experience- and theory-based debiasing lead a majority of learners to appreciate interleaving. Overall, the results indicate that 3 influences combine to make altering learners' misconceptions difficult: the sense of fluency that can accompany nonoptimal modes of instruction; pre-existing beliefs learners bring to new tasks; and the willingness, even eagerness, to believe that 1 is unique as a learner-that what enhances others' learning differs from what enhances one's own learning. (PsycINFO Database Record

Retrieval-induced forgetting is associated with increased positivity when imagining the future.

People often think of themselves and their experiences in a more positive light than is objectively justified. Inhibitory control processes may promote this positivity bias by modulating the accessibility of negative thoughts and episodes from the past, which then limits their influence in the construction of imagined future events. We tested this hypothesis by investigating the correlation between retrieval-induced forgetting and the extent to which individuals imagine positive and negative episodic future events. First, we measured performance on a task requiring participants to imagine personal episodic events (either positive or negative), and then we correlated that measure with retrieval-induced forgetting. As predicted, individuals who exhibited higher levels of retrieval-induced forgetting imagined fewer negative episodic future events than did individuals who exhibited lower levels of retrieval-induced forgetting. This finding provides new insight into the possible role of retrieval-induced forgetting in autobiographical memory.

Judgments of learning as memory modifiers.

A frequent procedure used to study how individuals monitor their own learning is to collect judgments of learning (JOLs) during acquisition, considered to be important, in part, because such judgments are assumed to guide how individuals allocate their future learning resources. In such research, however, a tacit assumption is frequently made: Namely, that asking for such metacognitive judgments does not affect the learning process per se. In 3 experiments, the present research addressed the accuracy of this assumption and tested a possible account--based on aspects of Koriat's cue-utilization approach to JOLs (Koriat, 1997) and de Winstanley, Bjork, and Bjork's (1996) transfer-appropriate multifactor account of generation effects--for why the mere act of making JOLs might enhance later memory for the information so judged. Potential implications of the present findings for the future conduction of research using metacognitive measures as well as for students studying for exams is discussed.

Intentional forgetting can increase, not decrease, residual influences of to-be-forgotten information.

Intentionally forgotten information remains in memory at essentially full strength, as measured by recognition and priming, but access to that information is impaired, as measured by recall. Given that pattern, it seemed plausible that intentionally forgotten information might have a greater impact on certain subsequent judgments than would intentionally remembered information. In 2 experiments, participants cued to forget nonfamous names were subsequently more likely to make false attributions of fame to those names than were participants instructed to remember them. These findings implicate retrieval inhibition as a potent factor in the interplay of recollection and priming in memory and judgment. They also point to possible unintended consequences of instructions to forget, suppress, or disregard in legal or social settings.

Retrieval experience as a modifier of future encoding: another test effect.

Research on how individuals monitor their level of comprehension during study paints a picture of learners as being insensitive to many of the factors or conditions of learning that can enhance long-term retention and transfer. In previous research, however, deWinstanley and Bjork (2004) demonstrated that learners--if made sensitive to the memorial benefits of generation in the context of an informative test following study of a text passage in which they had encoded both to-be-read and to-be-generated critical items--then became more effective processors of future to-be-read information presented in a 2nd text passage. In Experiments 1 and 2 of the present research, we explored the potential applicability of this effect by testing whether it could survive certain types of activity-filled delays. In Experiments 3 and 4, we tested whether enhanced processing of contextual information, an encoding strategy that could possibly have been discovered by participants during the testing episode for the 1st text passage, was a potential underlying cause of this effect. Together, our results bring to light an additional benefit of test taking and point to what might be considered necessary and sufficient conditions for leading learners to become more effective processors of future to-be-learned information.

Thinking can cause forgetting: memory dynamics in creative problem solving.

Research on retrieval-induced forgetting has shown that retrieval can cause the forgetting of related or competing items in memory (Anderson, Bjork, & Bjork, 1994). In the present research, we examined whether an analogous phenomenon occurs in the context of creative problem solving. Using the Remote Associates Test (RAT; Mednick, 1962), we found that attempting to generate a novel common associate to 3 cue words caused the forgetting of other strong associates related to those cue words. This problem-solving-induced forgetting effect occurred even when participants failed to generate a viable solution, increased in magnitude when participants spent additional time problem solving, and was positively correlated with problem-solving success on a separate set of RAT problems. These results implicate a role for forgetting in overcoming fixation in creative problem solving.