A metacognitive confidence calibration (MCC) tool to help medical students scaffold diagnostic reasoning in decision-making during high-fidelity patient simulations.

The purpose of this study was to 1) help novice students scaffold problem-solving and engage safely in the deliberate practice of diagnostic reasoning and medical decision-making in real time; 2) assess how accurately students gather and apply data in medical reasoning and treatment during high-fidelity patient simulations (HFPSs); 3) identify students' scientific misconceptions related to the case; 4) promote student metacognitive processing, self-assessment, and self-efficacy; and 5) facilitate the explicit calibration of student confidence in deliberate reasoning with patient outcomes. In a mixed-method design, a metacognitive calibration self-assessing (MCC) survey tool was applied to HFPS (n = 80, 20 teams of 6 medical students) and semistructured interviews were conducted with faculty (n = 5). When scored by faculty with a rubric, the mean student accuracy ranged from 23% to 74%, whereas their self-assessment of confidence ranged from 71% to 86%. This result revealed overconfidence bias in novice students regarding the correctness of their wrong responses. The most common misconception identified was inverting cause and effect: metabolic acidosis was pointed to as the cause of the patient's problems rather than a consequence of untreated diabetes mellitus. The most common treatment error was overtreatment, with unnecessary added medication. Interviews with faculty suggested that the MCC tool improved the team process by slowing students down, requiring them to think through their answers, and that overall the tool improved their critical thinking. This study demonstrated the feasibility of using a metacognitive confidence calibration tool to assist novice students in learning safely to make deliberate diagnostic reasoning and decisions on patient care in real time during complex simulations while observing objectively their levels of psychological confidence against patient outcomes.NEW & NOTEWORTHY This study demonstrates the feasibility of a metacognitive confidence calibration tool (MCC) to assess and promote novices in the learning of diagnostic reasoning and treatment decisions on patient care in real time during high-fidelity patient simulations while comparing confidence and accuracy data and identifying students' scientific misconceptions. Results revealed the presence of overconfidence bias, overtreatment, and the misconception of metabolic acidosis as the cause of the patient's problems rather than a consequence of untreated diabetes mellitus.

Using the physiology of normal aging as a capstone integration exercise in a medical physiology course.

As the Baby Boomer generation reaches old age, there has been a significant increase in the number of older adults needing healthcare over the past decade. The physiology of aging is therefore a highly relevant topic for the preclinical medical curriculum. I describe a new capstone unit on the physiology of aging, placed at the end a medical physiology course, to provide a vehicle for integration of prior learning about physiology of each individual body system. Students were provided with online self-study modules as preparation for a mandatory small group case-based learning activity. A detailed case of an elderly female patient being assessed for fall risk was provided. Students were required to document a "Review of Systems" predicting decreased system functions due to senescence and to prepare a group concept map illustrating how physiologic deficits contributed to fall risk in the patient. Students successfully completed the activity and reported generally good satisfaction with the experience. The activity was judged an effective tool for students to consolidate prior learning and to apply physiology to an important medical topic. The lesson also provided several opportunities for curriculum integration with cell biology, biochemistry, anatomy, and clinical skills components.

Understanding reasons for electing gap years between undergraduate education and medical school and the impact of gap years on the student experience of medical education.

Gap years have grown in popularity among students attempting to go to medical school in recent years. According to the Association of American Medical Colleges (AAMC), more than half of matriculated students now have taken time off between their undergraduate studies and medical school. Although many claim benefits to taking gap years, research on the impacts of these experiences on medical students is largely nonexistent. This study aimed to qualitatively analyze the gap year phenomenon and its impacts on medical students through semistructured interviews conducted at the University of Central Florida College of Medicine. Through these interviews, the research team collected in-depth responses that were thematically analyzed. Analysis revealed the following major themes regarding the impacts of gap years: 1) adaptability to change and failure, 2) professional identity formation and understanding of team role, 3) understanding the "real world," 4) refocusing of goals, 5) resilience and stress management, and 6) reinforcement of motivation.

Patterns of medical student engagement in a second-year pathophysiology course: relationship to USMLE Step 1 performance.

Historically, attendance has been a marker of academic performance, but the current medical education literature has had mixed results. In addition, attendance is dropping in the preclinical curricula, whereas, at the same time, the focus on United States Medical Licensing Examination Step 1 performance is increasing. This present study is a mixed-method approach correlating student attendance and access to the formal curriculum in a second-year pathophysiology course to performance on Step 1. Additionally, survey and focus group data evaluated the usage and importance of both the formal curriculum and third-party resources. Out of 112 eligible students, 77 participated in the study. There was no correlation between attendance or access to the learning materials and Step 1 performance. There was a strong correlation between the performance on the final examination and that of Step 1 (r = 0.813; P < 0.001) and a moderate correlation between formative quiz (r = 0.321; P = 0.005) and individual readiness assessment test performance (r = 0.351; P = 0.002) and Step 1 performance. Survey and focus group data show that students place high importance on faculty-developed materials that they can use on their own, but not attendance. The third-party resources are highly used as an adjunct to the formal curriculum and to focus on Step 1 study. Attendance and access to the formal curriculum do not predict Step 1 performance, whereas performance on high- and low-stakes internal assessments do. Further study on how the lack of social interaction gained from attendance affects development of other competencies and the learning climate are warranted.

Relationship between classroom attendance and examination performance in a second-year medical pathophysiology class.

Studies completed with undergraduate populations have shown that attendance positively correlates with academic performance. A marked decline in classroom attendance within medical school has recently been noticed with the availability of video capture of lectures and other online material. This study compares these in the era of online material. It took place during the second-year Gastrointestinal and Renal Systems module. Attendance was mandatory at team-based learning and case-based learning sessions on new material and voluntary at lectures (29 sessions) and case-based learning on material previously covered (9 sessions). Attendance was recorded prospectively. All lectures were recorded, and all related files were available to students online. Performance was based on a 118 multiple-choice question final examination. Students voluntarily completed the Motivated Strategies for Learning Questionnaire (MSLQ). The study group consisted of 78 students (68% of 114 total) of whom 48 completed the MSLQ. Mean attendance was 24%, with 33% of students attending none of the nonmandatory sessions. The median score on the final exam for participants was 86.0 (range: 28.8). High levels of self-efficacy and the ability to self-regulate effort were predictive of low attendance. Attendance was positively predicted by an orientation toward peer learning and help seeking. There was no correlation between the percentage of classes attended and performance on the final exam. We conclude that different facets of self-regulated learning predict attendance, with highly confident students being the least likely to attend, and that attendance at in-class sessions is no longer a good marker for performance.

Best practices in summative assessment.

The goal of this review is to highlight key elements underpinning excellent high-stakes summative assessment. This guide is primarily aimed at faculty members with the responsibility of assigning student grades and is intended to be a practical tool to help throughout the process of planning, developing, and deploying tests as well as monitoring their effectiveness. After a brief overview of the criteria for high-quality assessment, the guide runs through best practices for aligning assessment with learning outcomes and compares common testing modalities. Next, the guide discusses the kind of validity evidence needed to support defensible grading of student performance. This review concentrates on how to measure the outcome of student learning; other reviews in this series will expand on the related concepts of formative testing and how to leverage testing for learning.

Understanding factors affecting participation in online formative quizzes: an interview study.

A positive correlation between performance and participation in formative quizzes and final summative examinations has been reported many times. The goal of the present interview study was to construct a model to explain why students may elect not to engage with formative assessment opportunities. Our medical school's preclinical curriculum has an established policy of offering weekly online quizzes in all courses during the first 2 yr. Quizzes do not count for credit. Semistructured interviews were recorded and transcribed verbatim, and a formal thematic analysis was applied. A total of 16 in-depth interviews were conducted, with 8 female and 8 male participants. Thematic analysis revealed four major interacting themes that we propose to converge to account for nonparticipation in quizzes: 1) inadequate feedback, 2) curriculum organization and student mistrust, 3) time constraints, and 4) fear of judgment. We propose seven practice points to improve the effectiveness of formative assessment quizzes of medical knowledge.

Developing a tool for observing group critical thinking skills in first-year medical students: a pilot study using physiology-based, high-fidelity patient simulations.

Most assessments of physiology in medical school use multiple choice tests that may not provide information about a student's critical thinking (CT) process. There are limited performance assessments, but high-fidelity patient simulations (HFPS) may be a feasible platform. The purpose of this pilot study was to determine whether a group's CT process could be observed over a series of HFPS. An instrument [Critical Thinking Skills Rating Instrument CTSRI)] was designed with the IDEAS framework. Fifteen groups of students participated in three HFPS that consisted of a basic knowledge quiz and introduction, HFPS session, and debriefing. HFPS were video recorded, and two raters reviewed and scored all HFPS encounters with the CTSRI independently. Interrater analysis suggested good reliability. There was a correlation between basic knowledge scores and three of the six observations on the CTSRI providing support for construct validity. The median CT ratings significantly increased for all observations between the groups' first and last simulation. However, there were still large percentages of video ratings that indicated students needed substantial prompting during the HFPS. The data from this pilot study suggest that it is feasible to observe CT skills in HFPS using the CTSRI. Based on the findings from this study, we strongly recommend that first-year medical students be competent in basic knowledge of the relevant physiology of the HFPS before participating, to minimize the risk of a poor learning experience.

Team-based learning in large enrollment classes.

The goal of this review is to highlight the key elements needed to successfully deploy team-based learning (TBL) in any class, but especially in large enrolment classes, where smooth logistics are essential. The text is based on a lecture and workshop given at the American Physiological Society's Institute on Teaching and Learning in Madison, WI, in June 2016. After a short overview of the TBL method, its underpinning in learning theory, and a summary of current evidence for its effectiveness, we present two case studies from our own teaching practices in a new medical school. The first case study explores critical elements of design and planning for a TBL module, and the second explores best practices in classroom management. As medical educators in the fields of physiology, pediatrics, nephrology, and family medicine, we present the objective views of subject matter experts who adopted TBL as one teaching method rather than TBL experts or advocates per se. The review is aimed primarily at faculty contemplating using TBL for the first time who are interested in exploring the significant benefits and challenges of TBL.

Application of a utility analysis to evaluate a novel assessment tool for clinically oriented physiology and pharmacology.

Multiple-choice questions are a gold-standard tool in medical school for assessment of knowledge and are the mainstay of licensing examinations. However, multiple-choice questions items can be criticized for lacking the ability to test higher-order learning or integrative thinking across multiple disciplines. Our objective was to develop a novel assessment that would address understanding of pathophysiology and pharmacology, evaluate learning at the levels of application, evaluation and synthesis, and allow students to demonstrate clinical reasoning. The rubric assesses student writeups of clinical case problems. The method is based on the physician's traditional postencounter Subjective, Objective, Assessment and Plan note. Students were required to correctly identify subjective and objective findings in authentic clinical case problems, to ascribe pathophysiological as well as pharmacological mechanisms to these findings, and to justify a list of differential diagnoses. A utility analysis was undertaken to evaluate the new assessment tool by appraising its reliability, validity, feasibility, cost effectiveness, acceptability, and educational impact using a mixed-method approach. The Subjective, Objective, Assessment and Plan assessment tool scored highly in terms of validity and educational impact and had acceptable levels of statistical reliability but was limited in terms of acceptance, feasibility, and cost effectiveness due to high time demands on expert graders and workload concerns from students. We conclude by making suggestions for improving the tool and recommend deployment of the instrument for low-stakes summative assessment or formative assessment.

Faculty reflections on the process of building an integrated preclerkship curriculum: a new school perspective.

This is a reflective essay based on the experience of developing a structure and function module within a new integrated medical curriculum. Our hope is that the insights we gained during a 4-yr journey in a new medical school will be transferable to others engaged with curriculum development. Here, we present an interpretive analysis of our personal experiences together with some original research data and a synthesis of the literature. We will argue that a focus on teaching faculty is the key to successful curriculum integration and suggest an agenda for faculty development. Our essay begins by exploring what curriculum integration really means and what its purpose might be. Our case study explores the challenges of building a shared understanding among stakeholders and of negotiating learning outcomes and methods of teaching as well as the process of developing content and assessment. We feel that many of our experiences in the new medical school are applicable in other settings, such as curriculum reform in established schools and for developers of competency-based premedical curricula. We conclude with recommendations to assist other curriculum planners and teachers by offering some benefits of hindsight.

Use of the NBME Comprehensive Basic Science Examination as a progress test in the preclerkship curriculum of a new medical school.

In the present study, we describe the innovative use of the National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination (CBSE) as a progress test during the preclerkship medical curriculum. The main aim of this study was to provide external validation of internally developed multiple-choice assessments in a new medical school. The CBSE is a practice exam for the United States Medical Licensing Examination (USMLE) Step 1 and is purchased directly from the NBME. We administered the CBSE five times during the first 2 yr of medical school. Student scores were compared with scores on newly created internal summative exams and to the USMLE Step 1. Significant correlations were observed between almost all our internal exams and CBSE scores over time as well as with USMLE Step 1 scores. The strength of correlations of internal exams to the CBSE and USMLE Step 1 broadly increased over time during the curriculum. Student scores on courses that have strong emphasis on physiology and pathophysiology correlated particularly well with USMLE Step 1 scores. Student progress, as measured by the CBSE, was found to be linear across time, and test performance fell behind the anticipated level by the end of the formal curriculum. These findings are discussed with respect to student learning behaviors. In conclusion, the CBSE was found to have good utility as a progress test and provided external validation of our new internally developed multiple-choice assessments. The data also provide performance benchmarks both for our future students to formatively assess their own progress and for other medical schools to compare learning progression patterns in different curricular models.

Insights gained from the analysis of performance and participation in online formative assessment.

BACKGROUND: Online quizzes are simple, cost-effective methods to provide formative assessment, but their effectiveness in enhancing learning and performance in medical education is unclear. PURPOSE: The purpose of this article is to determine the extent to which online quiz performance and participation enhances students' performance on summative examinations. METHODS: A retrospective case study investigating relationships between formative and summative assessment in terms of use and outcomes. RESULTS: Online quiz scores and the rates of quiz participation were significantly correlated with corresponding performance on summative examinations. However, correlations were not dependent on the specific quiz content, and changes in patterns of quiz use were not reflected in corresponding changes in summative examination performance. CONCLUSIONS: The voluntary use of online quizzes, as well as the score attained, provides a useful general indicator of student performance but is unlikely to be sensitive enough to direct an individual student's learning plan.

Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations?

The purpose of this study was to evaluate whether multiple-choice item difficulty could be predicted either by a subjective judgment by the question author or by applying a learning taxonomy to the items. Eight physiology faculty members teaching an upper-level undergraduate human physiology course consented to participate in the study. The faculty members annotated questions before exams with the descriptors "easy," "moderate," or "hard" and classified them according to whether they tested knowledge, comprehension, or application. Overall analysis showed a statistically significant, but relatively low, correlation between the intended item difficulty and actual student scores (ρ = -0.19, P < 0.01), indicating that, as intended item difficulty increased, the resulting student scores on items tended to decrease. Although this expected inverse relationship was detected, faculty members were correct only 48% of the time when estimating difficulty. There was also significant individual variation among faculty members in the ability to predict item difficulty (χ(2) = 16.84, P = 0.02). With regard to the cognitive level of items, no significant correlation was found between the item cognitive level and either actual student scores (ρ = -0.09, P = 0.14) or item discrimination (ρ = 0.05, P = 0.42). Despite the inability of faculty members to accurately predict item difficulty, the examinations were of high quality, as evidenced by reliability coefficients (Cronbach's α) of 0.70-0.92, the rejection of only 4 of 300 items in the postexamination review, and a mean item discrimination (point biserial) of 0.37. In conclusion, the effort of assigning annotations describing intended difficulty and cognitive levels to multiple-choice items is of doubtful value in terms of controlling examination difficulty. However, we also report that the process of annotating questions may enhance examination validity and can reveal aspects of the hidden curriculum.

Ethical approval for research in physiology education.

The goal of this article is to reflect on the contemporary ethical standards that should be applied to the publication of physiology education research. As teachers, we are all education researchers to some degree but our appreciation of when and how regulatory requirements apply to our work is variable. A significant number of articles in Advances in Physiology Education that might be classified as "research involving human participants" do not document ethical safeguards such as Institutional Review Board approval and informed consent, which are required according to journal policy. I elaborate my personal view that we should strive to maintain the present community standards for conducting and publishing education research. And, as always, I hope the road to hell is not paved with good intentions!

A peer-led supplemental tutorial project for medical physiology: implementation in a large class.

The purpose of this study was to evaluate the practicality of implementing a peer-teaching program in a large class (>350 students) of medical students and whether such a program is beneficial. Case-based problems were developed by faculty members to facilitate student problem solving and discussion. Voluntary student enrollment was available during the first week of a semester. Tutorials took place during out of class time and were facilitated by peers from the previous class. Tutors were selected for their outstanding performance in physiology; they were provided with training in facilitation skills and were given a package of model answers. Sixty-eight students enrolled in this pilot program and were organized into groups of approximately 8 students. On average, students attended four of six tutorials. Posttutorial quiz scores were significantly greater than paired pretest scores. Surveys showed that students had high expectations at the outset, which were matched with positive perceptions at the end of the tutorial program; the use of near-peer tutors was especially well received. Tutors also gave high approval ratings for their experiences. In conclusion, the peer tutoring program was logistically straightforward to implement in a large class and was endorsed by the participants.

Is formative assessment an effective way to improve learning? A Symposium at Experimental Biology 2008.

Formative assessment is designed to provide information about students' learning to help them and their teachers to identify deficiencies and misconceptions. It differs from summative assessment, which aims to rank students according to their achievements to determine which students pass or fail or to assign grades to students. This article reports on a symposium concerned with evidence for the effectiveness of formative assessment in improving learning. It was presented by the Teaching of Physiology Section of the American Physiological Society at the Experimental Biology Meeting of 2008.

Effective use of course management systems to enhance student learning: Experimental Biology 2007.

Course management systems are software packages that support teaching and learning by providing web-based tools, services, and resources. Electronic course management can facilitate course organization, teaching delivery, communication, collaboration, and assessment. This article reports on a symposium on the use of course management systems, presented by the Teaching of Physiology Section of the American Physiological Society, at the Experimental Biology Meeting of 2007.