How the wisdom of crowds, and of the crowd within, are affected by expertise.

We investigated the effect of expertise on the wisdom of crowds. Participants completed 60 trials of a numerical estimation task, during which they saw 50-100 asterisks and were asked to estimate how many stars they had just seen. Experiment 1 established that both inner- and outer-crowd wisdom extended to our novel task: Single responses alone were less accurate than responses aggregated across a single participant (showing inner-crowd wisdom) and responses aggregated across different participants were even more accurate (showing outer-crowd wisdom). In Experiment 2, prior to beginning the critical trials, participants did 12 practice trials with feedback, which greatly increased their accuracy. There was a benefit of outer-crowd wisdom relative to a single estimate. There was no inner-crowd wisdom effect, however; with high accuracy came highly restricted variance, and aggregating insufficiently varying responses is not beneficial. Our data suggest that experts give almost the same answer every time they are asked and so they should consult the outer crowd rather than solicit multiple estimates from themselves.

Answering a factual question today increases one's confidence in the same answer tomorrow - independent of fluency.

We investigated the cognitive processes that cause confidence to increase. Participants were asked 48 general-knowledge questions either once or three times, without feedback. After 2 min (Experiment 1) or 48 h (Experiment 2) they were asked the same questions again, and rated their confidence. Repeated questioning increased confidence but not accuracy. This increase, which replicated research on episodic memory in the eyewitness literature (e.g., Shaw, Journal of Experimental Psychology: Applied 2: 126-146, 1996), occurred even though accuracy was only around 25%. A mediation analysis identified response repetition, but not fluency, as a mechanism underlying growth in confidence. Thus, the basis for confidence judgments appears to be whether one's current response has been generated previously. In sum, answering a factual question increases confidence, but not accuracy, and this happens because learners use response repetition as a cue for confidence judgments.

How to activate students' natural desire to test themselves.

Testing oneself (i.e., doing retrieval practice) is an effective way to study. We attempted to make learners choose to test themselves more often. In Experiment 1, participants were asked how they wanted to study and were given four options: retrieval with no hint (e.g., idea: ______), a two-letter hint (e.g., idea: s____r), a four-letter hint (e.g., idea: se__er), or a presentation trial (e.g., idea: seeker). They tested themselves on the majority of trials. In Experiment 2, when the hint options were removed, they chose restudy rather than pure test on the majority of trials. These findings show that people prefer self-testing over restudy as long as they can get the answer right on the test. However, we would not recommend hints if they impaired learning compared to pure testing. Experiment 3 showed that this was not the case; the three retrieval conditions from Experiment 1 led to equivalent amounts of learning, and all three outperformed the pure presentation condition. We used different materials in Experiment 4 and found that the hints made retrieval slightly less beneficial when the hints made it possible to guess the answers without thinking back to the study phase (e.g., whip: pu__sh). In summary, hints catalyzed people's intuitive desire to self-test, without any downside for learning, thus making their self-regulated study more enjoyable and effective.

Audiovisual quality impacts assessments of job candidates in video interviews: Evidence for an AV quality bias.

Video job interviews have become a common hiring practice, allowing employers to save money and recruit from a wider applicant pool. But differences in job candidates' internet connections mean that some interviews will have higher audiovisual (AV) quality than others. We hypothesized that interviewers would be impacted by AV quality when they rated job candidates. In two experiments, participants viewed two-minute long simulated Skype interviews that were either unedited (fluent videos) or edited to mimic the effects of a poor internet connection (disfluent videos). Participants in both experiments rated job candidates from fluent videos as more hirable, even after being explicitly told to disregard AV quality (experiment 2). Our findings suggest that video interviews may favor job candidates with better internet connections and that being aware of this bias does not make it go away.

A metacognitive illusion in monkeys.

Like humans, monkeys can make accurate judgements about their own memory by reporting their confidence during cognitive tasks. Some have suggested that animals use associative learning to make accurate confidence judgements, while others have suggested animals directly access and estimate the strength of their memories. Here we test a third, non-exclusive possibility: perhaps monkeys, like humans, base metacognitive inferences on heuristic cues. Humans are known to use cues like perceptual fluency (e.g. how easy something is to see) when making metacognitive judgements. We tested monkeys using a match-to-sample task in which the perceptual fluency of the stimuli was manipulated. The monkeys made confidence wagers on their accuracy before or after each trial. We found that monkeys' wagers were affected by perceptual fluency even when their accuracy was not. This is novel evidence that animals are susceptible to metacognitive illusions similar to those experienced by humans.

Performance bias: Why judgments of learning are not affected by learning.

Past research has shown a performance bias: People expect their future performance level on a task to match their current performance level, even when there are good reasons to expect future performance to differ from current performance. One explanation of this bias is that judgments are controlled by what learners can observe, and while current performance is usually observable, changes in performance (i.e., learning or forgetting) are not. This explanation makes a prediction that we tested here: If learning becomes observable, it should begin to affect judgments. In three experiments, after practicing a skill, participants estimated how they performed in the past and how they expected to perform in the future. In Experiments 1 and 2, participants knew they had been improving, as shown by their responses, yet they did not predict that they would improve in the future. This finding was particularly striking because (a) they did improve in the future and (b) as Experiment 3 showed, they did hold the conscious belief that past improvement predicted future improvement. In short, when learning and performance are both observable, judgments of learning seem to be guided by performance and not learning.

Retrieval attempts enhance learning regardless of time spent trying to retrieve.

Attempting to retrieve information from memory is an engaging cognitive activity. We predicted that people would learn more when they had spent more time attempting to retrieve. In experiments 1a and 1b, participants were shown trivia questions for 0, 5, 10, or 30 seconds and then the answer was revealed. They took a final test immediately or after 48 hours. Retrieval enhanced learning, but the length of the retrieval attempt had no effect (i.e., final test performance was equivalent in the 5-, 10-, and 30-second conditions and worse in the 0-second condition). During the initial retrieval attempt, more time did increase recall, suggesting that participants continued to engage in productive retrieval activities when given more time. Showing the answer for longer (7 versus 2 seconds) increased learning in Experiments 2a and 2b. Experiment 3 examined the effect of retrieval success and Experiment 4 replicated the results using different materials. These results have direct implications for current theories of retrieval.

The influence of feedback on predictions of future memory performance.

Kornell and Rhodes (Journal of Experimental Psychology: Applied, 19, 1-13, 2013) reported that correct answer feedback impairs the accuracy of prospective memory judgments. The current experiments explored the boundaries of this effect. In Experiment 1, participants studied Lithuanian-English word pairs, took an initial test, and were either given correct answer feedback or no feedback at all. They then made a judgment of learning (JOL) regarding the likelihood of correctly recalling the English translation on a later test. Presenting the correct answer as feedback increased average JOLs but impaired relative accuracy on a final test. Therefore, Experiments 2-4 aimed to specifically ameliorate impairments in relative accuracy following feedback. Participants in Experiment 2 were exposed to right/wrong feedback, no feedback, and correct answer feedback while making JOLs. Using such a within-subjects design did not improve relative accuracy following correct answer feedback. Experiment 3 showed that previous exposure to a test-feedback-test cycle did not improve relative accuracy. In Experiment 4, feedback was scaffolded such that the correct answer was progressively revealed. Participants corrected more errors if they could generate the correct response with fewer letter cues. However, relative accuracy did not improve in comparison to the previous experiments. Accordingly, the current experiments suggest that participants may understand that feedback is beneficial, but receiving feedback diminishes prediction accuracy for specific items and participants do not appreciate the magnitude of the benefits of feedback.

Do the Best Teachers Get the Best Ratings?

We review recent studies that asked: do college students learn relatively more from teachers whom they rate highly on student evaluation forms? Recent studies measured learning at two-time points. When learning was measured with a test at the end of the course, the teachers who got the highest ratings were the ones who contributed the most to learning. But when learning was measured as performance in subsequent related courses, the teachers who had received relatively low ratings appeared to have been most effective. We speculate about why these effects occurred: making a course difficult in productive ways may decrease ratings but enhance learning. Despite their limitations, we do not suggest abandoning student ratings, but do recommend that student evaluation scores should not be the sole basis for evaluating college teaching and they should be recognized for what they are.

Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter.

Retrieving information from memory enhances learning. We propose a 2-stage framework to explain the benefits of retrieval. Stage 1 takes place as one attempts to retrieve an answer, which activates knowledge related to the retrieval cue. Stage 2 begins when the answer becomes available, at which point appropriate connections are strengthened and inappropriate connections may be weakened. This framework raises a basic question: Does it matter whether Stage 2 is initiated via successful retrieval or via an external presentation of the answer? To test this question, we asked participants to attempt retrieval and then randomly assigned items (which were equivalent otherwise) to be retrieved successfully or to be copied (i.e., not retrieved). Experiments 1, 2, 4, and 5 tested assumptions necessary for interpreting Experiments 3a, 3b, and 6. Experiments 3a, 3b, and 6 did not support the hypothesis that retrieval success produces more learning than does retrieval failure followed by feedback. It appears that retrieval attempts promote learning but retrieval success per se does not.

Young children bet on their numerical skills: metacognition in the numerical domain.

Metacognition, the ability to assess one's own knowledge, has been targeted as a critical learning mechanism in mathematics education. Yet the early childhood origins of metacognition have proven difficult to study. Using a novel nonverbal task and a comprehensive set of metacognitive measures, we provided the strongest evidence to date that young children are metacognitive. We showed that children as young as 5 years made metacognitive "bets" on their numerical discriminations in a wagering task. However, contrary to previous reports from adults, our results showed that children's metacognition is domain specific: Their metacognition in the numerical domain was unrelated to their metacognition in another domain (emotion discrimination). Moreover, children's metacognitive ability in only the numerical domain predicted their school-based mathematics knowledge. The data provide novel evidence that metacognition is a fundamental, domain-dependent cognitive ability in children. The findings have implications for theories of uncertainty and reveal new avenues for training metacognition in children.

Expecting to teach enhances learning and organization of knowledge in free recall of text passages.

The present research assessed the potential effects of expecting to teach on learning. In two experiments, participants studied passages either in preparation for a later test or in preparation for teaching the passage to another student who would then be tested. In reality, all participants were tested, and no one actually engaged in teaching. Participants expecting to teach produced more complete and better organized free recall of the passage (Experiment 1) and, in general, correctly answered more questions about the passage than did participants expecting a test (Experiment 1), particularly questions covering main points (Experiment 2), consistent with their having engaged in more effective learning strategies. Instilling an expectation to teach thus seems to be a simple, inexpensive intervention with the potential to increase learning efficiency at home and in the classroom.

Retrospective and prospective metacognitive judgments in rhesus macaques (Macaca mulatta).

A growing body of research suggests that some non-human animals are capable of making accurate metacognitive judgments. In previous studies, non-human animals have made either retrospective or prospective judgments (about how they did on a test or how they will do on a test, respectively). These two types of judgments are dissociable in humans. The current study tested the abilities of two rhesus macaque monkeys to make both retrospective and prospective judgments about their performance on the same memory task. Both monkeys had been trained previously to make retrospective confidence judgments. Both monkeys successfully demonstrated transfer of retrospective metacognitive judgments to the new memory task. Furthermore, both monkeys transferred their retrospective judgments to the prospective task (one, immediately, and one, following the elimination of a response bias). This study is the first to demonstrate both retrospective and prospective monitoring abilities in the same monkeys and on the same task, suggesting a greater level of flexibility in animals' metacognitive monitoring abilities than has been reported previously.

Where is the "meta" in animal metacognition?

Apes, dolphins, and some monkeys seem to have metacognitive abilities: They can accurately evaluate the likelihood that their response in cognitive task was (or will be) correct. These certainty judgments are seen as significant because they imply that animals can evaluate internal cognitive states, which may entail meaningful self-reflection. But little research has investigated what is being reflected upon: Researchers have assumed that when animals make metacognitive judgments they evaluate internal memory strength. Yet decades of research have demonstrated that humans cannot directly evaluate internal memory strength. Instead, they make certainty judgments by drawing inferences from cues they can evaluate, such as familiarity and ease of processing. It seems likely that animals do the same, but this hypothesis has not been tested. I suggest two strategies for investigating the internal cues that underlie animal metacognitive judgments. It is possible that animals, like humans, are capable of making certainty judgments based on internal cues without awareness or meaningful self-reflection.

Attempting to answer a meaningful question enhances subsequent learning even when feedback is delayed.

Attempting to retrieve information from memory enhances subsequent learning even if the retrieval attempt is unsuccessful. Recent evidence suggests that this benefit materializes only if subsequent study occurs immediately after the retrieval attempt. Previous studies have prompted retrieval using a cue (e.g., whale-???) that has no intrinsic answer. Experiment 1 replicated prior word pair studies, but in Experiment 2, when participants learned meaningful trivia questions, testing enhanced learning even when subsequent study was delayed. Even in Experiment 3, when subsequent study was delayed by up to 24 hr, tests enhanced learning on a final test another 24 hr later. These findings may give comfort to educators who worry that asking a question or giving a test, on which students inevitably make mistakes, impairs learning if feedback is not immediate. They also suggest that there is a consensus in the literature thus far: Questions with rich semantic content enhance subsequent learning even when feedback is delayed, but less meaningful questions without an intrinsic answer enhance learning only when feedback is immediate. (PsycINFO Database Record (c) 2013 APA, all rights reserved).

Appearances can be deceiving: instructor fluency increases perceptions of learning without increasing actual learning.

The present study explored the effects of lecture fluency on students' metacognitive awareness and regulation. Participants watched one of two short videos of an instructor explaining a scientific concept. In the fluent video, the instructor stood upright, maintained eye contact, and spoke fluidly without notes. In the disfluent video, the instructor slumped, looked away, and spoke haltingly with notes. After watching the video, participants in Experiment 1 were asked to predict how much of the content they would later be able to recall, and participants in Experiment 2 were given a text-based script of the video to study. Perceived learning was significantly higher for the fluent instructor than for the disfluent instructor (Experiment 1), although study time was not significantly affected by lecture fluency (Experiment 2). In both experiments, the fluent instructor was rated significantly higher than the disfluent instructor on traditional instructor evaluation questions, such as preparedness and effectiveness. However, in both experiments, lecture fluency did not significantly affect the amount of information learned. Thus, students' perceptions of their own learning and an instructor's effectiveness appear to be based on lecture fluency and not on actual learning.

Feedback reduces the metacognitive benefit of tests.

Testing long-term memory has dual benefits: It enhances learning and it helps learners discriminate what they know from what they do not know. The latter benefit, known as delayed judgment of learning (dJOL) effect, has been well documented, but in prior research participants have not been provided with test feedback. Yet when people study they almost universally (a) get feedback and (b) judge their learning subsequent to receiving the feedback. Thus, in the first three experiments, participants made JOLs following tests with feedback. Adding feedback significantly decreased the dJOL effect relative to conditions taking a test without receiving feedback. In Experiment 4, participants made decisions about which items to restudy (without actually restudying); adding feedback also decreased the accuracy of these decisions. These findings suggest that, in realistic situations, tests enhance self-monitoring, but not as much as previously thought. Judging memory based on prior test performance and ignoring the effects of feedback appears to produce an "illusion of not knowing."

When and why a failed test potentiates the effectiveness of subsequent study.

Teachers and trainers often try to prevent learners from making errors, but recent findings (e.g., Kornell, Hays, & Bjork, 2009) have demonstrated that tests can potentiate subsequent learning even when the correct answer is difficult or impossible to generate (e.g., "What is Nate Kornell's middle name?"). In 3 experiments, we explored when and why a failed test enhances learning. We found that failed tests followed by immediate feedback produced greater retention than did a presentation-only condition. Failed tests followed by delayed feedback, by contrast, did not produce such a benefit-except when the direction of the final test was reversed (i.e., the participants were provided with the target and had to produce the original cue). Our findings suggest that generating an incorrect response to a cue both activates the semantic network associated with the cue and suppresses the correct response. These processes appear to have 2 consequences: If feedback is presented immediately, the semantic activation enhances the mapping of the cue to the correct response; if feedback is presented at a delay, the prior suppression boosts the learning of the suppressed response.

Self-regulated learning: beliefs, techniques, and illusions.

Knowing how to manage one's own learning has become increasingly important in recent years, as both the need and the opportunities for individuals to learn on their own outside of formal classroom settings have grown. During that same period, however, research on learning, memory, and metacognitive processes has provided evidence that people often have a faulty mental model of how they learn and remember, making them prone to both misassessing and mismanaging their own learning. After a discussion of what learners need to understand in order to become effective stewards of their own learning, we first review research on what people believe about how they learn and then review research on how people's ongoing assessments of their own learning are influenced by current performance and the subjective sense of fluency. We conclude with a discussion of societal assumptions and attitudes that can be counterproductive in terms of individuals becoming maximally effective learners.

Why interleaving enhances inductive learning: the roles of discrimination and retrieval.

Kornell and Bjork (Psychological science 19:585-592, 2008) found that interleaving exemplars of different categories enhanced inductive learning of the concepts based on those exemplars. They hypothesized that the benefit of mixing exemplars from different categories is that doing so highlights differences between the categories. Kang and Pashler (Applied cognitive psychology 26:97-103, 2012) obtained results consistent with this discriminative-contrast hypothesis: Interleaving enhanced inductive learning, but temporal spacing, which does not highlight category differences, did not. We further tested the discriminative-contrast hypothesis by examining the effects of interleaving and spacing, as well as their combined effects. In three experiments, using photographs of butterflies and birds as the stimuli, temporal spacing was harmful when it interrupted the juxtaposition of interleaved categories, even when total spacing was held constant, supporting the discriminative-contrast hypothesis. Temporal spacing also had value, however, when it did not interrupt discrimination processing.