Timing's not everything: Immediate and delayed feedback are equally beneficial for performance in formative multiple-choice testing.

INTRODUCTION: Test-enhanced learning (TEL) is an impactful teaching and learning strategy that prioritises active learner engagement through the process of regular testing and reviewing. While it is clear that meaningful feedback optimises the effects of TEL, the ideal timing of this feedback (i.e. immediate or delayed) in a medical education setting is unclear. METHOD: Forty-one second-year medical students were recruited from the University of Melbourne. Participants were given a multiple-choice question test with a mix of immediate (i.e. post-item) and delayed (i.e. post-item-block) conceptual feedback. Students were then tested on near and far transfer items during an immediate post-test, and at a one-week follow-up. RESULTS: A logistic mixed effects model was used to predict the probability of successful near and far transfer. As expected, participants in our study tended to score lower on far transfer items than they did on near transfer items. In addition, correct initial response on a parent question predicted subsequent correct responding. Contrary to our hypotheses, the feedback timing effect was non-significant-there was no discernible difference between feedback delivered immediately versus delayed feedback. DISCUSSION: The findings of this study suggest that the timing of feedback delivery (post-item versus post-item-block) does not influence the efficacy of TEL in this medical education setting. We therefore suggest that educators may consider practical factors when determining appropriate TEL feedback timing in their setting.

SMARCB1/INI1-deficient pancreatic undifferentiated rhabdoid carcinoma mimicking solid pseudopapillary neoplasm: A case report and review of the literature.

BACKGROUND: SMARCB1/INI1-deficient pancreatic undifferentiated rhabdoid carcinoma is a very aggressive tumor that is rarely reported in the literature. The tumor has a predominant rhabdoid cell component and different patterns of growth have been reported. CASE SUMMARY: A 59-year-old woman presented with diffuse abdominal pain, increasing in severity and accompanied by weight loss, nausea, and vomiting. Imaging showed a pancreatic head mass. Fine needle aspiration demonstrated atypical epithelioid cells with a pseudopapillary growth pattern suggestive of solid pseudopapillary neoplasm. The excised neoplasm showed monotonous epithelioid and focally spindle cells with pseudopapillary structures, rhabdoid features, and loss of SMARCB1 protein expression with wild-type KRAS, consistent with a SMARCB1/INI1-deficient undifferentiated rhabdoid carcinoma. The patient's condition deteriorated rapidly following surgery and she expired 3 mo post operation. CONCLUSION: In this article, we report the first case of SMARCB1/INI1-deficient undifferentiated pancreatic rhabdoid carcinoma mimicking solid pseudopapillary neoplasm.

Beyond right or wrong: More effective feedback for formative multiple-choice tests.

INTRODUCTION: The role of feedback in test-enhanced learning is an understudied area that has the potential to improve student learning. This study investigates the influence of different forms of post-test feedback on retention and transfer of biomedical knowledge within a test-enhanced learning framework. METHODS: 64 participants from a Canadian and an Australian medical school sat two single-best-answer formative multiple choice tests one week apart. We compared the effects of conceptually focused, response-oriented, and simple right/wrong feedback on a learner's ability to correctly answer new (transfer) questions. On the first test occasion, participants received parent items with feedback, and then attempted items closely related (near transfer) to and more distant (far transfer) from parent items. In a repeat test at 1 week, participants were given different near and far transfer versions of parent items. Feedback type, and near and far transfer items were randomized within and across participants. RESULTS: Analysis demonstrated that response-oriented and conceptually focused feedback were superior to traditional right/wrong feedback for both types of transfer tasks and in both immediate and final retention test performance. However, there was no statistically significant difference between response-orientated and conceptually focused groups on near or far transfer problems, nor any differences in performance between our initial test occasion and the retention test 1 week later. As with most studies of transfer, participants' far transfer scores were lower than for near transfer. DISCUSSION: Right/wrong feedback appears to have limited potential to augment test-enhanced learning. Our work suggests that item-level feedback and feedback that identifies and elaborates on key conceptual knowledge are two important areas for future research on learning, retention and transfer.

Commentary: Paradigms, Axiology, and Praxeology in Medical Education Research.

This supplement includes 11 research papers accepted by the 2018 Research in Medical Education Committee. In this Commentary, the authors draw the attention of medical educators and researchers to questions about the researcher's philosophical paradigm. Such questions fundamentally guide research-the choice of conceptual frameworks, methods, and methodology. The authors provide a brief overview of research paradigms and the related concept of axiology and praxeology in medical education research. The authors map the 11 accepted research articles, describing the researchers' stated or implied worldviews and the impact on chosen methods. The authors close by encouraging researchers to state the research paradigm behind their research, comment on how they ensured that the methods used displayed consistency with that paradigm, and highlight the value the research adds to everyday education.

Planning Education for Long-Term Retention: The Cognitive Science and Implementation of Retrieval Practice.

Educational systems are rarely designed for long-term retention of information. Strong evidence has emerged from cognitive psychology and applied education studies that repeated retrieval of information significantly improves retention compared to repeated studying. This effect likely emerges from the processes of memory consolidation and reconsolidation. Consolidation and reconsolidation are the means by which memories are organized into associational networks or schemas that are created and recreated as memories are formed and recalled. As educators implement retrieval practice, they should consider how various test formats lead to different degrees of schema activation. Repeated acts of retrieval provide opportunities for schemas to be updated and strengthened. Spacing of retrieval allows more consolidated schemas to be reactivated. Feedback provides metacognitive monitoring to ensure retrieval accuracy and can lead to shifts from ineffective to effective retrieval strategies. By using the principles of retrieval practice, educators can improve the likelihood that learners will retain information for longer periods of time.

Functional Dysregulation of CDC42 Causes Diverse Developmental Phenotypes.

Exome sequencing has markedly enhanced the discovery of genes implicated in Mendelian disorders, particularly for individuals in whom a known clinical entity could not be assigned. This has led to the recognition that phenotypic heterogeneity resulting from allelic mutations occurs more commonly than previously appreciated. Here, we report that missense variants in CDC42, a gene encoding a small GTPase functioning as an intracellular signaling node, underlie a clinically heterogeneous group of phenotypes characterized by variable growth dysregulation, facial dysmorphism, and neurodevelopmental, immunological, and hematological anomalies, including a phenotype resembling Noonan syndrome, a developmental disorder caused by dysregulated RAS signaling. In silico, in vitro, and in vivo analyses demonstrate that mutations variably perturb CDC42 function by altering the switch between the active and inactive states of the GTPase and/or affecting CDC42 interaction with effectors, and differentially disturb cellular and developmental processes. These findings reveal the remarkably variable impact that dominantly acting CDC42 mutations have on cell function and development, creating challenges in syndrome definition, and exemplify the importance of functional profiling for syndrome recognition and delineation.

Tying knots: an activity theory analysis of student learning goals in clinical education.

CONTEXT: Learning goal programmes are often created to help students develop self-regulated learning skills; however, these programmes do not necessarily consider the social contexts surrounding learning goals or how they fit into daily educational practice. OBJECTIVES: We investigated a high-frequency learning goal programme in which students generated and shared weekly learning goals with their clinical teams in core Year 3 clerkships. Our study explores: (i) how learning goals were incorporated into the clinical work, and (ii) the factors that influenced the use of students' learning goals in work-based learning. METHODS: We conducted semi-structured interviews with 14 students and 14 supervisors (attending physicians and residents) sampled from all participating core clerkships. Interviews were coded for emerging themes. Using cultural historical activity theory and knotworking as theoretical lenses, we developed a model of the factors that influenced students' learning goal usage in a work-based learning context. RESULTS: Students and supervisors often faced the challenge of reconciling contradictions that arose when the desired outcomes of student skill development, grading and patient care were not aligned. Learning goals could function as tools for developing new ways of acting that overcame those contradictions by facilitating collaborative effort between students and their supervisors. However, for new collaborations to take place, both students and supervisors had to engage with the goals, and the necessary patients needed to be present. When any one part of the system did not converge around the learning goals, the impact of the learning goals programme was limited. CONCLUSIONS: Learning goals are potentially powerful tools to mediate interactions between students, supervisors and patients, and to reconcile contradictions in work-based learning environments. Learning goals provide a means to develop not only learners, but also learning systems.

High-Frequency Learning Goals: Using Self-Regulated Learning to Influence Day-to-Day Practice in Clinical Education.

PROBLEM: Although self-regulated learning (SRL) is considered a fundamental skill that must be developed in physician training, many programs of SRL utilize learning goals that are generated only at the beginning of learning experiences or are widely spaced apart in time. These goals are often not formally shared with those actually working with the learner in the clinical setting. INTERVENTION: We developed a program of written, student-generated weekly learning goals in which students focused on processes of becoming better doctors for their patients. These goals were shared at the beginning of each week with students' clinical teams for feedback and incorporation into the work. CONTEXT: The weekly learning goals program was developed and implemented as part of a required 3rd-year neurology clerkship. At the end of each 4-week clerkship, students were asked to evaluate the program through an anonymous electronic survey utilizing quantitative and open-ended qualitative questions. OUTCOME: Seventy-six of 82 students completed the evaluation survey (93% response rate). Eighty-six percent reported that the weekly learning goals increased their awareness of their thoughts and actions. Seventy-eight percent reported that the learning goals helped to improve their clinical performance to some degree, and 57% reported that the learning goals increased their focus on patient care. Students described a greater sense of focus on self-assessment and accountability from their goals. Students often commented that engagement from attendings and residents regarding their goals was a key element for successful learning from their goals. LESSONS LEARNED: Student-generated weekly learning goals on a neurology clerkship appear to be an effective method to operationalize SRL. For most students, the frequency of the goals allowed for close self-monitoring, and the act of sharing goals with the team opened a new avenue for dialogue between students and their supervisors.

Student views on the role of self-regulated learning in a surgery clerkship.

BACKGROUND: Self-regulated learning, including student-generated learning goals and flexibility in the learning structure are increasingly being used to enhance medical education. The role of these practices in surgical education of medical students has not been studied. MATERIALS AND METHODS: We administered an 18-question electronic survey to all third-year medical students at Washington University in St. Louis School of Medicine. Of the 126 students invited, 64 responded and 56 were included in the analysis. RESULTS: We found that third-year medical students develop learning goals at the beginning of the surgery clerkship. Although these learning goals theoretically can be a mechanism for enhanced student-faculty engagement, students are not aware of formal mechanisms for sharing these goals with faculty members. Furthermore, students report a lack of flexibility within the surgery clerkship and discomfort with requesting specific learning opportunities. Finally, students report that they believe increased flexibility could improve student engagement, learning, and the overall clerkship experience. CONCLUSIONS: We therefore propose that a mechanism for students to share their learning goals with faculty and an infrastructure in which student learning experiences can be tailored to fit with these individualized goals would enhance student surgical learning.

Using reflection to influence practice: student perceptions of daily reflection in clinical education.

PURPOSE: Reflection is a key element in learning from experience, but the impact of most programmes of reflection on daily practice remains unclear. We investigated students' perceptions of adding a daily written reflection assignment to a clinical rotation. METHODS: Third-year medical students on a single two-week rotation completed daily reflections analyzing their performance. Programme evaluation used a 33-question anonymized survey. Quantitative data were summarized and qualitative responses coded for recurring themes. RESULTS: Twenty-six students completed the survey (90 % response rate). Eighty-five percent of students felt that the daily reflections had a positive impact on their learning from clinical experience. Seventy-seven percent of students reported that the programme changed their awareness of their thoughts and actions, and 80 % felt that it improved their recall of experiences. A greater sense of mindfulness and focus on self-improvement were major themes that emerge from students' descriptions of the role of daily reflections in their learning. CONCLUSION: Overall, daily reflections demonstrated a positive learning influence. This exploratory study suggests students may benefit from more frequent, short reflections as opposed to more typically spaced reflective assignments.

Effects of Team-Based Learning on short-term and long-term retention of factual knowledge.

PURPOSE: We investigated the effect of Team-Based Learning (TBL) on long-term retention of knowledge in comparison to a traditional curriculum. METHODS: As TBL was incorporated into our curriculum in the 2008-2009 academic year, students were compared with those who received the traditional curriculum the year prior. Students in both the groups completed multiple-choice knowledge test at four time points spanning two years. Test performance was compared at each time point to assess changes in knowledge retention as a function of time. RESULTS: Baseline knowledge did not differ significantly between the TBL and control groups [51% versus 46%; t(84) = 0.91, p = 0.37, d = 0.20]. Performance improved after the course for both the groups, but was significantly higher in the TBL group [79% versus 59%; t(84) = 4.96, p = 0.000004, d = 0.95]. However, when assessed prior to the pediatrics clerkship, learning gains from TBL had largely disappeared and the small difference in performance was not significant [57% versus 51%; t(84) = 1.51, p = 0.14, d = 0.32]. CONCLUSION: Incorporating TBL into the pre-clinical pediatrics curriculum led to large gains in knowledge over the short-term, but these gains did not persist. Further research should focus on extending the impact of TBL on long-term knowledge retention.

Student-directed retrieval practice is a predictor of medical licensing examination performance.

INTRODUCTION: A large body of evidence indicates that retrieval practice (test-enhanced learning) and spaced repetition increase long-term information retention. Implementation of these strategies in medical curricula is unfortunately limited. However, students may choose to apply them autonomously when preparing for high-stakes, cumulative assessments, such as the United States Medical Licensing Examination Step 1. We examined the prevalence of specific self-directed methods of testing, with or without spaced repetition, among preclinical students and assessed the relationship between these methods and licensing examination performance. METHODS: Seventy-two medical students at one institution completed a survey concerning their use of user-generated (Anki) or commercially-available (Firecracker) flashcards intended for spaced repetition and of boards-style multiple-choice questions (MCQs). Other information collected included Step 1 score, past academic performance (Medical College Admission Test [MCAT] score, preclinical grades), and psychological factors that may have affected exam preparation or performance (feelings of depression, burnout, and test anxiety). RESULTS: All students reported using practice MCQs (mean 3870, SD 1472). Anki and Firecracker users comprised 31 and 49 % of respondents, respectively. In a multivariate regression model, significant independent predictors of Step 1 score included MCQs completed (unstandardized beta coefficient [B] = 2.2 × 10- 3, p < 0.001), unique Anki flashcards seen (B = 5.9 × 10- 4, p = 0.024), second-year honours (B = 1.198, p = 0.002), and MCAT score (B = 1.078, p = 0.003). Test anxiety was a significant negative predictor (B= - 1.986, p < 0.001). Unique Firecracker flashcards seen did not predict Step 1 score. Each additional 445 boards-style practice questions or 1700 unique Anki flashcards was associated with an additional point on Step 1 when controlling for other academic and psychological factors. CONCLUSIONS: Medical students engage extensively in self-initiated retrieval practice, often with spaced repetition. These practices are associated with superior performance on a medical licensing examination and should be considered for formal support by educators.

The effects of test-enhanced learning on long-term retention in AAN annual meeting courses.

OBJECTIVE: We measured the long-term retention of knowledge gained through selected American Academy of Neurology annual meeting courses and compared the effects of repeated quizzing (known as test-enhanced learning) and repeated studying on that retention. METHODS: Participants were recruited from 4 annual meeting courses. All participants took a pretest. This randomized, controlled trial utilized a within-subjects design in which each participant experienced 3 different postcourse activities with each activity performed on different material. Each key information point from the course was randomized in a counterbalanced fashion among participants to one of the 3 activities: repeated short-answer quizzing, repeated studying, and no further exposure to the materials. A final test covering all information points from the course was taken 5.5 months after the course. RESULTS: Thirty-five participants across the 4 courses completed the study. Average score on the pretest was 36%. Performance on the final test showed that repeated quizzing led to significantly greater long-term retention relative to both repeated studying (55% vs 46%; t[34] = 3.28, SEM = 0.03, p = 0.01, d = 0.49) and no further exposure (55% vs 44%; t[34] = 3.16, SEM = 0.03, p = 0.01, d = 0.58). Relative to the pretest baseline, repeated quizzing helped participants to retain almost twice as much of the knowledge acquired from the course compared to repeated studying or no further exposure. CONCLUSIONS: Whereas annual meeting continuing medical education (CME) courses lead to long-term gains in knowledge, when repeated quizzing is added, retention is significantly increased. CME planners may consider adding repeated quizzing to increase the impact of their courses.