A metacognitive confidence calibration (MCC) tool to help medical students scaffold diagnostic reasoning in decision-making during high-fidelity patient simulations.

The purpose of this study was to 1) help novice students scaffold problem-solving and engage safely in the deliberate practice of diagnostic reasoning and medical decision-making in real time; 2) assess how accurately students gather and apply data in medical reasoning and treatment during high-fidelity patient simulations (HFPSs); 3) identify students' scientific misconceptions related to the case; 4) promote student metacognitive processing, self-assessment, and self-efficacy; and 5) facilitate the explicit calibration of student confidence in deliberate reasoning with patient outcomes. In a mixed-method design, a metacognitive calibration self-assessing (MCC) survey tool was applied to HFPS (n = 80, 20 teams of 6 medical students) and semistructured interviews were conducted with faculty (n = 5). When scored by faculty with a rubric, the mean student accuracy ranged from 23% to 74%, whereas their self-assessment of confidence ranged from 71% to 86%. This result revealed overconfidence bias in novice students regarding the correctness of their wrong responses. The most common misconception identified was inverting cause and effect: metabolic acidosis was pointed to as the cause of the patient's problems rather than a consequence of untreated diabetes mellitus. The most common treatment error was overtreatment, with unnecessary added medication. Interviews with faculty suggested that the MCC tool improved the team process by slowing students down, requiring them to think through their answers, and that overall the tool improved their critical thinking. This study demonstrated the feasibility of using a metacognitive confidence calibration tool to assist novice students in learning safely to make deliberate diagnostic reasoning and decisions on patient care in real time during complex simulations while observing objectively their levels of psychological confidence against patient outcomes.NEW & NOTEWORTHY This study demonstrates the feasibility of a metacognitive confidence calibration tool (MCC) to assess and promote novices in the learning of diagnostic reasoning and treatment decisions on patient care in real time during high-fidelity patient simulations while comparing confidence and accuracy data and identifying students' scientific misconceptions. Results revealed the presence of overconfidence bias, overtreatment, and the misconception of metabolic acidosis as the cause of the patient's problems rather than a consequence of untreated diabetes mellitus.

Understanding reasons for electing gap years between undergraduate education and medical school and the impact of gap years on the student experience of medical education.

Gap years have grown in popularity among students attempting to go to medical school in recent years. According to the Association of American Medical Colleges (AAMC), more than half of matriculated students now have taken time off between their undergraduate studies and medical school. Although many claim benefits to taking gap years, research on the impacts of these experiences on medical students is largely nonexistent. This study aimed to qualitatively analyze the gap year phenomenon and its impacts on medical students through semistructured interviews conducted at the University of Central Florida College of Medicine. Through these interviews, the research team collected in-depth responses that were thematically analyzed. Analysis revealed the following major themes regarding the impacts of gap years: 1) adaptability to change and failure, 2) professional identity formation and understanding of team role, 3) understanding the "real world," 4) refocusing of goals, 5) resilience and stress management, and 6) reinforcement of motivation.

Using the physiology of normal aging as a capstone integration exercise in a medical physiology course.

As the Baby Boomer generation reaches old age, there has been a significant increase in the number of older adults needing healthcare over the past decade. The physiology of aging is therefore a highly relevant topic for the preclinical medical curriculum. I describe a new capstone unit on the physiology of aging, placed at the end a medical physiology course, to provide a vehicle for integration of prior learning about physiology of each individual body system. Students were provided with online self-study modules as preparation for a mandatory small group case-based learning activity. A detailed case of an elderly female patient being assessed for fall risk was provided. Students were required to document a "Review of Systems" predicting decreased system functions due to senescence and to prepare a group concept map illustrating how physiologic deficits contributed to fall risk in the patient. Students successfully completed the activity and reported generally good satisfaction with the experience. The activity was judged an effective tool for students to consolidate prior learning and to apply physiology to an important medical topic. The lesson also provided several opportunities for curriculum integration with cell biology, biochemistry, anatomy, and clinical skills components.

Patterns of medical student engagement in a second-year pathophysiology course: relationship to USMLE Step 1 performance.

Historically, attendance has been a marker of academic performance, but the current medical education literature has had mixed results. In addition, attendance is dropping in the preclinical curricula, whereas, at the same time, the focus on United States Medical Licensing Examination Step 1 performance is increasing. This present study is a mixed-method approach correlating student attendance and access to the formal curriculum in a second-year pathophysiology course to performance on Step 1. Additionally, survey and focus group data evaluated the usage and importance of both the formal curriculum and third-party resources. Out of 112 eligible students, 77 participated in the study. There was no correlation between attendance or access to the learning materials and Step 1 performance. There was a strong correlation between the performance on the final examination and that of Step 1 (r = 0.813; P < 0.001) and a moderate correlation between formative quiz (r = 0.321; P = 0.005) and individual readiness assessment test performance (r = 0.351; P = 0.002) and Step 1 performance. Survey and focus group data show that students place high importance on faculty-developed materials that they can use on their own, but not attendance. The third-party resources are highly used as an adjunct to the formal curriculum and to focus on Step 1 study. Attendance and access to the formal curriculum do not predict Step 1 performance, whereas performance on high- and low-stakes internal assessments do. Further study on how the lack of social interaction gained from attendance affects development of other competencies and the learning climate are warranted.

Relationship between classroom attendance and examination performance in a second-year medical pathophysiology class.

Studies completed with undergraduate populations have shown that attendance positively correlates with academic performance. A marked decline in classroom attendance within medical school has recently been noticed with the availability of video capture of lectures and other online material. This study compares these in the era of online material. It took place during the second-year Gastrointestinal and Renal Systems module. Attendance was mandatory at team-based learning and case-based learning sessions on new material and voluntary at lectures (29 sessions) and case-based learning on material previously covered (9 sessions). Attendance was recorded prospectively. All lectures were recorded, and all related files were available to students online. Performance was based on a 118 multiple-choice question final examination. Students voluntarily completed the Motivated Strategies for Learning Questionnaire (MSLQ). The study group consisted of 78 students (68% of 114 total) of whom 48 completed the MSLQ. Mean attendance was 24%, with 33% of students attending none of the nonmandatory sessions. The median score on the final exam for participants was 86.0 (range: 28.8). High levels of self-efficacy and the ability to self-regulate effort were predictive of low attendance. Attendance was positively predicted by an orientation toward peer learning and help seeking. There was no correlation between the percentage of classes attended and performance on the final exam. We conclude that different facets of self-regulated learning predict attendance, with highly confident students being the least likely to attend, and that attendance at in-class sessions is no longer a good marker for performance.

Developing a tool for observing group critical thinking skills in first-year medical students: a pilot study using physiology-based, high-fidelity patient simulations.

Most assessments of physiology in medical school use multiple choice tests that may not provide information about a student's critical thinking (CT) process. There are limited performance assessments, but high-fidelity patient simulations (HFPS) may be a feasible platform. The purpose of this pilot study was to determine whether a group's CT process could be observed over a series of HFPS. An instrument [Critical Thinking Skills Rating Instrument CTSRI)] was designed with the IDEAS framework. Fifteen groups of students participated in three HFPS that consisted of a basic knowledge quiz and introduction, HFPS session, and debriefing. HFPS were video recorded, and two raters reviewed and scored all HFPS encounters with the CTSRI independently. Interrater analysis suggested good reliability. There was a correlation between basic knowledge scores and three of the six observations on the CTSRI providing support for construct validity. The median CT ratings significantly increased for all observations between the groups' first and last simulation. However, there were still large percentages of video ratings that indicated students needed substantial prompting during the HFPS. The data from this pilot study suggest that it is feasible to observe CT skills in HFPS using the CTSRI. Based on the findings from this study, we strongly recommend that first-year medical students be competent in basic knowledge of the relevant physiology of the HFPS before participating, to minimize the risk of a poor learning experience.

Understanding factors affecting participation in online formative quizzes: an interview study.

A positive correlation between performance and participation in formative quizzes and final summative examinations has been reported many times. The goal of the present interview study was to construct a model to explain why students may elect not to engage with formative assessment opportunities. Our medical school's preclinical curriculum has an established policy of offering weekly online quizzes in all courses during the first 2 yr. Quizzes do not count for credit. Semistructured interviews were recorded and transcribed verbatim, and a formal thematic analysis was applied. A total of 16 in-depth interviews were conducted, with 8 female and 8 male participants. Thematic analysis revealed four major interacting themes that we propose to converge to account for nonparticipation in quizzes: 1) inadequate feedback, 2) curriculum organization and student mistrust, 3) time constraints, and 4) fear of judgment. We propose seven practice points to improve the effectiveness of formative assessment quizzes of medical knowledge.

Best practices in summative assessment.

The goal of this review is to highlight key elements underpinning excellent high-stakes summative assessment. This guide is primarily aimed at faculty members with the responsibility of assigning student grades and is intended to be a practical tool to help throughout the process of planning, developing, and deploying tests as well as monitoring their effectiveness. After a brief overview of the criteria for high-quality assessment, the guide runs through best practices for aligning assessment with learning outcomes and compares common testing modalities. Next, the guide discusses the kind of validity evidence needed to support defensible grading of student performance. This review concentrates on how to measure the outcome of student learning; other reviews in this series will expand on the related concepts of formative testing and how to leverage testing for learning.

Team-based learning in large enrollment classes.

The goal of this review is to highlight the key elements needed to successfully deploy team-based learning (TBL) in any class, but especially in large enrolment classes, where smooth logistics are essential. The text is based on a lecture and workshop given at the American Physiological Society's Institute on Teaching and Learning in Madison, WI, in June 2016. After a short overview of the TBL method, its underpinning in learning theory, and a summary of current evidence for its effectiveness, we present two case studies from our own teaching practices in a new medical school. The first case study explores critical elements of design and planning for a TBL module, and the second explores best practices in classroom management. As medical educators in the fields of physiology, pediatrics, nephrology, and family medicine, we present the objective views of subject matter experts who adopted TBL as one teaching method rather than TBL experts or advocates per se. The review is aimed primarily at faculty contemplating using TBL for the first time who are interested in exploring the significant benefits and challenges of TBL.

Application of a utility analysis to evaluate a novel assessment tool for clinically oriented physiology and pharmacology.

Multiple-choice questions are a gold-standard tool in medical school for assessment of knowledge and are the mainstay of licensing examinations. However, multiple-choice questions items can be criticized for lacking the ability to test higher-order learning or integrative thinking across multiple disciplines. Our objective was to develop a novel assessment that would address understanding of pathophysiology and pharmacology, evaluate learning at the levels of application, evaluation and synthesis, and allow students to demonstrate clinical reasoning. The rubric assesses student writeups of clinical case problems. The method is based on the physician's traditional postencounter Subjective, Objective, Assessment and Plan note. Students were required to correctly identify subjective and objective findings in authentic clinical case problems, to ascribe pathophysiological as well as pharmacological mechanisms to these findings, and to justify a list of differential diagnoses. A utility analysis was undertaken to evaluate the new assessment tool by appraising its reliability, validity, feasibility, cost effectiveness, acceptability, and educational impact using a mixed-method approach. The Subjective, Objective, Assessment and Plan assessment tool scored highly in terms of validity and educational impact and had acceptable levels of statistical reliability but was limited in terms of acceptance, feasibility, and cost effectiveness due to high time demands on expert graders and workload concerns from students. We conclude by making suggestions for improving the tool and recommend deployment of the instrument for low-stakes summative assessment or formative assessment.

Use of the NBME Comprehensive Basic Science Examination as a progress test in the preclerkship curriculum of a new medical school.

In the present study, we describe the innovative use of the National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination (CBSE) as a progress test during the preclerkship medical curriculum. The main aim of this study was to provide external validation of internally developed multiple-choice assessments in a new medical school. The CBSE is a practice exam for the United States Medical Licensing Examination (USMLE) Step 1 and is purchased directly from the NBME. We administered the CBSE five times during the first 2 yr of medical school. Student scores were compared with scores on newly created internal summative exams and to the USMLE Step 1. Significant correlations were observed between almost all our internal exams and CBSE scores over time as well as with USMLE Step 1 scores. The strength of correlations of internal exams to the CBSE and USMLE Step 1 broadly increased over time during the curriculum. Student scores on courses that have strong emphasis on physiology and pathophysiology correlated particularly well with USMLE Step 1 scores. Student progress, as measured by the CBSE, was found to be linear across time, and test performance fell behind the anticipated level by the end of the formal curriculum. These findings are discussed with respect to student learning behaviors. In conclusion, the CBSE was found to have good utility as a progress test and provided external validation of our new internally developed multiple-choice assessments. The data also provide performance benchmarks both for our future students to formatively assess their own progress and for other medical schools to compare learning progression patterns in different curricular models.

Faculty reflections on the process of building an integrated preclerkship curriculum: a new school perspective.

This is a reflective essay based on the experience of developing a structure and function module within a new integrated medical curriculum. Our hope is that the insights we gained during a 4-yr journey in a new medical school will be transferable to others engaged with curriculum development. Here, we present an interpretive analysis of our personal experiences together with some original research data and a synthesis of the literature. We will argue that a focus on teaching faculty is the key to successful curriculum integration and suggest an agenda for faculty development. Our essay begins by exploring what curriculum integration really means and what its purpose might be. Our case study explores the challenges of building a shared understanding among stakeholders and of negotiating learning outcomes and methods of teaching as well as the process of developing content and assessment. We feel that many of our experiences in the new medical school are applicable in other settings, such as curriculum reform in established schools and for developers of competency-based premedical curricula. We conclude with recommendations to assist other curriculum planners and teachers by offering some benefits of hindsight.

Development and evaluation of an interactive electronic laboratory manual for cooperative learning of medical histology.

This article describes the development of an interactive computer-based laboratory manual, created to facilitate the teaching and learning of medical histology. The overarching goal of developing the manual is to facilitate self-directed group interactivities that actively engage students during laboratory sessions. The design of the manual includes guided instruction for students to navigate virtual slides, exercises for students to monitor learning, and cases to provide clinical relevance. At the end of the laboratory activities, student groups can generate a laboratory report that may be used to provide formative feedback. The instructional value of the manual was evaluated by a questionnaire containing both closed-ended and open-ended items. Closed-ended items using a five-point Likert-scale assessed the format and navigation, instructional contents, group process, and learning process. Open-ended items assessed student's perception on the effectiveness of the manual in facilitating their learning. After implementation for two consecutive years, student evaluation of the manual was highly positive and indicated that it facilitated their learning by reinforcing and clarifying classroom sessions, improved their understanding, facilitated active and cooperative learning, and supported self-monitoring of their learning.

Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations?

The purpose of this study was to evaluate whether multiple-choice item difficulty could be predicted either by a subjective judgment by the question author or by applying a learning taxonomy to the items. Eight physiology faculty members teaching an upper-level undergraduate human physiology course consented to participate in the study. The faculty members annotated questions before exams with the descriptors "easy," "moderate," or "hard" and classified them according to whether they tested knowledge, comprehension, or application. Overall analysis showed a statistically significant, but relatively low, correlation between the intended item difficulty and actual student scores (ρ = -0.19, P < 0.01), indicating that, as intended item difficulty increased, the resulting student scores on items tended to decrease. Although this expected inverse relationship was detected, faculty members were correct only 48% of the time when estimating difficulty. There was also significant individual variation among faculty members in the ability to predict item difficulty (χ(2) = 16.84, P = 0.02). With regard to the cognitive level of items, no significant correlation was found between the item cognitive level and either actual student scores (ρ = -0.09, P = 0.14) or item discrimination (ρ = 0.05, P = 0.42). Despite the inability of faculty members to accurately predict item difficulty, the examinations were of high quality, as evidenced by reliability coefficients (Cronbach's α) of 0.70-0.92, the rejection of only 4 of 300 items in the postexamination review, and a mean item discrimination (point biserial) of 0.37. In conclusion, the effort of assigning annotations describing intended difficulty and cognitive levels to multiple-choice items is of doubtful value in terms of controlling examination difficulty. However, we also report that the process of annotating questions may enhance examination validity and can reveal aspects of the hidden curriculum.

Insights gained from the analysis of performance and participation in online formative assessment.

BACKGROUND: Online quizzes are simple, cost-effective methods to provide formative assessment, but their effectiveness in enhancing learning and performance in medical education is unclear. PURPOSE: The purpose of this article is to determine the extent to which online quiz performance and participation enhances students' performance on summative examinations. METHODS: A retrospective case study investigating relationships between formative and summative assessment in terms of use and outcomes. RESULTS: Online quiz scores and the rates of quiz participation were significantly correlated with corresponding performance on summative examinations. However, correlations were not dependent on the specific quiz content, and changes in patterns of quiz use were not reflected in corresponding changes in summative examination performance. CONCLUSIONS: The voluntary use of online quizzes, as well as the score attained, provides a useful general indicator of student performance but is unlikely to be sensitive enough to direct an individual student's learning plan.

The use of self-learning modules to facilitate learning of basic science concepts in an integrated medical curriculum.

This study used qualitative and quantitative approaches to evaluate the effectiveness of self-learning modules (SLMs) developed to facilitate and individualize students' learning of basic medical sciences. Twenty physiology and nineteen microanatomy SLMs were designed with interactive images, animations, narrations, and self-assessments. Of 41 medical students, 40 students voluntarily completed a questionnaire with open-ended and closed-ended items to evaluate students' attitudes and perspectives on the learning value of SLMs. Closed-ended items were assessed on a five-point Likert scale (5 = high score) and the data were expressed as mean ± standard deviation. Open-ended questions further evaluated students' perspectives on the effectiveness of SLMs; student responses to open-ended questions were analyzed to identify shared patterns or themes in their experience using SLMs. The results of the midterm examination were also analyzed to compare student performance on items related to SLMs and traditional sessions. Students positively evaluated their experience using the SLMs with an overall mean score of 4.25 (SD ± 0.84). Most students (97%) indicated that the SLMs improved understanding and facilitated learning basic science concepts. SLMs were reported to allow learner control, to help in preparation for subsequent in-class discussion, and to improve understanding and retention. A significant difference in students' performance was observed when comparing SLM-related items with non-SLM items in the midterm examination (P < 0.05). In conclusion, the use of SLMs in an integrated basic science curriculum has the potential to individualize the teaching and improve the learning of basic sciences.

Ethical approval for research in physiology education.

The goal of this article is to reflect on the contemporary ethical standards that should be applied to the publication of physiology education research. As teachers, we are all education researchers to some degree but our appreciation of when and how regulatory requirements apply to our work is variable. A significant number of articles in Advances in Physiology Education that might be classified as "research involving human participants" do not document ethical safeguards such as Institutional Review Board approval and informed consent, which are required according to journal policy. I elaborate my personal view that we should strive to maintain the present community standards for conducting and publishing education research. And, as always, I hope the road to hell is not paved with good intentions!