Validation of a performance assessment instrument in problem-based learning tutorials using two cohorts of medical students.

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60 %), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep(2) = .68, ϕ = .60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < .01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.

Student Perceived Value of Anatomy Pedagogy, Part I: Prosection or Dissection?

A series of three annual surveys of David Geffen School of Medicine (DGSOM) at UCLA students and UCR/UCLA Thomas Haider Program in Biomedical Sciences students were administered from 2010 to 2012 to ascertain student perceptions of which anatomy pedagogy-prosection or dissection-was most valuable to them during the first year of preclinical medical education and for the entire medical school experience in general. Students were asked, "What value does gross anatomy education have in preclinical medical education?" We further asked the students who participated in both prosection and dissection pedagogies, "Would you have preferred an anatomy curriculum like the Summer Anatomy Dissection during your first year in medical school instead of prosection?" All students who responded to the survey viewed anatomy as a highly valued part of the medical curriculum, specifically referring to four major themes: Anatomy is (1) the basis for medical understanding, (2) part of the overall medical school experience, (3) a bridge to understanding pathology and physiology, and (4) the foundation for clinical skills. Students who participated in both prosection and dissection pedagogies surprisingly and overwhelmingly advocated for a prosection curriculum for the first year of medical school, not a dissection curriculum. Time efficiency was the dominant theme in survey responses from students who learned anatomy through prosection and then dissection. Students, regardless of whether interested in surgery/radiology or not, appreciated both pedagogies but commented that prosection was sufficient for learning basic anatomy, while dissection was a necessary experience in preparation for the anatomical medical specialties. This suggests that anatomy instruction should be integrated into the clinical years of medical education.

Student Perceived Value of Anatomy Pedagogy, Part II: Clinical Practice and Assessment.

We describe student beliefs of how anatomy education influenced their preparation for standardized clinical assessments and clinical skills. We conducted three annual surveys of students of the David Geffen School of Medicine (DGSOM) at the University of California Los Angeles (UCLA) and students of the University of California, Riverside (UCR)/UCLA Thomas Haider Program in Biomedical Sciences from 2010 to 2012. Students were asked, "What specific knowledge or skills did you learn from your gross anatomy experience that helped you prepare for USMLE board exams, third-year clerkships, and physical examination skills?" All students who responded to the survey viewed anatomy as a highly valued part of the medical curriculum. Almost all students felt that anatomy knowledge in general was useful for their success with United States Medical Licensing Examination (USMLE) exams, how they perceived their physical exam skills, and how they perceived their preparation for third- or fourth-year clerkships. On the other hand, when asked about how the anatomy curriculum helped prepare students for fourth-year clerkships, there was a downward trend over a three-year period with each subsequent class. Although anatomy is a highly valued part of the medical school experience, students value integration of the anatomical and clinical sciences, as evidenced by a perceived diminishing value of anatomy pedagogy taught outside of clinical context with subsequent classes over the course of three years.

Identifying longitudinal growth trajectories of learning domains in problem-based learning: a latent growth curve modeling approach using SEM.

To determine the direction and extent to which medical student scores (as observed by small-group tutors) on four problem-based-learning-related domains change over nine consecutive blocks during a two-year period (Domains: Problem Solving/Use of Information/Group Process/Professionalism). Latent growth curve modeling is used to analyze performance trajectories in each domain of two cohorts of 1st and 2nd year students (n = 296). Slopes of the growth trajectories show similar linear increments in the first three domains. Further analysis revealed relative strong individual variability in initial scores but not in their later increments. Professionalism, on the other hand, shows low variability and has very small, insignificant slope increments. In this study, we showed that the learning domains (Problem Solving, Use of Information, and Group Process) observed during PBL tutorials are not only related to each other but also develop cumulatively over time. Professionalism, in contrast to the other domains studied, is less affected by the curriculum suggesting that this represents a stable characteristic. The observation that the PBL tutorial has an equal benefit to all students is noteworthy and needs further investigation.

Clinical clerkship timing revisited: Support for non-uniform sequencing.

BACKGROUND: The logistical necessity of students taking required clinical clerkship rotations in non-uniform sequences may have significant consequences on performance. AIMS: To investigate (1) the impact of previous clinical clerkship experience and the national licensing examination (USMLE Step 1) rankings on end-of-clerkship overall skills assessments and (2) the effect of clinical clerkship order on end-of-year objective structured clinical examination (OSCE) performance. METHOD: The core clinical phase of UCLA consists at six clinical clerkships during a 48-week continuum. The clerkships are structured in two different tracks with different rotation order. Clinical performance scores were analyzed for 124 year-3 students in the two different tracks. RESULTS: Performance, as reviewed on a standardized clinical clerkship evaluation form by faculty and residents, improved significantly across the clinical year, regardless of track suggesting an accumulated advantage of previous experiences. There were no significant differences between tracks on the OSCE. CONCLUSIONS: Overall clinical skills are more directly impacted by the time-of-year and not order whereas knowledge of specific specialties may be impacted by the order rotations that are taken. While this may not be obvious to students, perhaps we should share these results for their use in tracking their personal growth in clinical skills.

Relationship between medical student perceptions of mistreatment and mistreatment sensitivity.

BACKGROUND: National statistics reveal that efforts to reduce medical student mistreatment have been largely ineffective. Some hypothesize that as supervisors gain skills in professionalism, medical students become more sensitive. AIMS: The purpose of this study was to determine if medical student perceptions of mistreatment are correlated with mistreatment sensitivity. METHOD: At the end of their third year, 175 medical students completed an Abuse Sensitivity Questionnaire, focused on student assessment of hypothetical scenarios which might be perceived as abusive, and the annual Well-Being Survey, which includes measurement of incident rates of mistreatment. It was hypothesized that those students who identified the scenarios as abusive would also be more likely to perceive that they had been mistreated. RESULTS: Student perceptions of mistreatment were not statistically correlated with individual's responses to the scenarios or to a statistically derived abuse sensitivity variable. There were no differences in abuse sensitivity by student age or ethnicity. Women were more likely than men to consider it "harsh" to be called incompetent during rounds (p < 0.0005). CONCLUSION: This study provides preliminary evidence that challenges the hypothesis that medical students who perceive mistreatment by their superiors are simply more sensitive.

Educational Experiences Residents Perceive As Most Helpful for the Acquisition of the ACGME Competencies.

BACKGROUND: The Accreditation Council for Graduate Medical Education (ACGME) requires physicians in training to be educated in 6 competencies considered important for independent medical practice. There is little information about the experiences that residents feel contribute most to the acquisition of the competencies. OBJECTIVE: To understand how residents perceive their learning of the ACGME competencies and to determine which educational activities were most helpful in acquiring these competencies. METHOD: A web-based survey created by the graduate medical education office for institutional program monitoring and evaluation was sent to all residents in ACGME-accredited programs at the David Geffen School of Medicine, University of California-Los Angeles, from 2007 to 2010. Residents responded to questions about the adequacy of their learning for each of the 6 competencies and which learning activities were most helpful in competency acquisition. RESULTS: We analyzed 1378 responses collected from postgraduate year-1 (PGY-1) to PGY-3 residents in 12 different residency programs, surveyed between 2007 and 2010. The overall response rate varied by year (66%-82%). Most residents (80%-97%) stated that their learning of the 6 ACGME competencies was "adequate." Patient care activities and observation of attending physicians and peers were listed as the 2 most helpful learning activities for acquiring the 6 competencies. CONCLUSION: Our findings reinforce the importance of learning from role models during patient care activities and the heterogeneity of learning activities needed for acquiring all 6 competencies.

A new approach to learning how to teach: medical students as instructional designers.

As students at the David Geffen School of Medicine at UCLA, the student authors were given the opportunity to develop their own creative projects which would be used to teach future medical students. They chose their own topics, planned and researched their projects, and then implemented the projects in interactive digital Adobe Flash files. In the first project they created interactive case-based radiology teaching files. In the second project they integrated photographic images into the existing illustrative anatomy files. Students in subsequent years have learned from these files on computers both at home and in the school's anatomy lab. The experience of creating the files served as an opportunity for hands-on learning for the student authors, both of the material and of the practice of teaching. In this paper they describe why they undertook these projects, what exactly they did, and the impact their creation had on them. The projects demonstrate that student-driven educational materials are both possible and beneficial. Furthermore, their experience has allowed them to conclude that faculty at other medical schools should consider providing students with opportunities to develop their own creative projects that contribute to the curriculum.

Clinical competence understood through the construct validity of three clerkship assessments.

OBJECTIVES: This study examined the construct validity of three commonly used clerkship performance assessments, including preceptors' evaluations, objective structured clinical examination (OSCE)-type clinical performance measures, and the National Board of Medical Examiners (NBME) medicine subject examination, in order to better understand their conceptual structures and utility in the explanation of clinical competence. METHODS: A total of 686 students who took an in-patient medicine clerkship during the period 2003 to 2007 participated in the study. Exploratory and confirmatory factor analyses using structural equation modelling were adopted to examine the latent domains underlying various indicators assessed by these three measures and the pattern of indicator-domain relationships. RESULTS: Factor analyses found three latent constructs, labelled Clinical Performance, Interpersonal Skills and Clinical Knowledge, underlying the observed measures. The three domains were modestly correlated with one another (inter-factor correlation coefficients ranged from 0.39 to 0.54). They also tapped a common higher-order construct, Clinical Competence, in varying degrees of magnitude (0.73, 0.74, 0.53, respectively). CONCLUSIONS: The study demonstrated that although the three commonly used tools for assessing clerkship performance contributed uniquely to the understanding of clinical performance, they also attested to a shared domain of clinical competence in their assessment. The study confirmed the need for a multiple-methods approach to clinical performance assessment. Findings also revealed that clerkship preceptors need to differentiate their evaluation of students' performances, and that the OSCE did not assess a single domain of clinical competence.

Utility of the AAMC's Graduation Questionnaire to study behavioral and social sciences domains in undergraduate medical education.

PURPOSE: The Institute of Medicine (IOM) report on social and behavioral sciences (SBS) indicated that 50% of morbidity and mortality in the United States is associated with SBS factors, which the report also found were inadequately taught in medical school. A multischool collaborative explored whether the Association of American Medical Colleges Graduation Questionnaire (GQ) could be used to study changes in the six SBS domains identified in the IOM report. METHOD: A content analysis conducted with the GQ identified 30 SBS variables, which were narrowed to 24 using a modified Delphi approach. Summary data were pooled from nine medical schools for 2006 and 2007, representing 1,126 students. Data were generated on students' perceptions of curricular experiences, attitudes related to SBS curricula, and confidence with relevant clinical knowledge and skills. The authors determined the sample sizes required for various effect sizes to assess the utility of the GQ. RESULTS: The 24 variables were classified into five of six IOM domains representing a total of nine analytic categories with cumulative scale means ranging from 60.8 to 93.4. Taking into account the correlations among measures over time, and assuming a two-sided test, 80% power, alpha at .05, and standard deviation of 4.1, the authors found that 34 medical schools would be required for inclusion to attain an estimated effect size of 0.50 (50%). With a sample size of nine schools, the ability to detect changes would require a very high effect size of 107%. CONCLUSIONS: Detecting SBS changes associated with curricular innovations would require a large collaborative of medical schools. Using a national measure (the GQ) to assess curricular innovations in most areas of SBS is possible if enough medical schools were involved in such an effort.

Student body racial and ethnic composition and diversity-related outcomes in US medical schools.

CONTEXT: Many medical schools assert that a racially and ethnically diverse student body is an important element in educating physicians to meet the needs of a diverse society. However, there is limited evidence addressing the educational effects of student body racial diversity. OBJECTIVE: To determine whether student body racial and ethnic diversity is associated with diversity-related outcomes among US medical students. DESIGN, SETTING, AND PARTICIPANTS: A Web-based survey (Graduation Questionnaire) administered by the Association of American Medical Colleges of 20,112 graduating medical students (64% of all graduating students in 2003 and 2004) from 118 allopathic medical schools in the United States. Historically black and Puerto Rican medical schools were excluded. MAIN OUTCOME MEASURES: Students' self-rated preparedness to care for patients from other racial and ethnic backgrounds, attitudes about equity and access to care, and intent to practice in an underserved area. RESULTS: White students within the highest quintile for student body racial and ethnic diversity, measured by the proportion of underrepresented minority (URM) students, were more likely to rate themselves as highly prepared to care for minority populations than those in the lowest diversity quintile (61.1% vs 53.9%, respectively; P < .001; adjusted odds ratio [OR], 1.33; 95% confidence interval [CI], 1.13-1.57). This association was strongest in schools in which students perceived a positive climate for interracial interaction. White students in the highest URM quintile were also more likely to have strong attitudes endorsing equitable access to care (54.8% vs 44.2%, respectively; P < .001; adjusted OR, 1.42; 95% CI, 1.15-1.74). For nonwhite students, after adjustment there were no significant associations between student body URM proportions and diversity-related outcomes. Student body URM proportions were not associated with white or nonwhite students' plans to practice in underserved communities, although URM students were substantially more likely than white or nonwhite/non-URM students to plan to serve the underserved (48.7% vs 18.8% vs 16.2%, respectively; P < .001). CONCLUSION: Student body racial and ethnic diversity within US medical schools is associated with outcomes consistent with the goal of preparing students to meet the needs of a diverse population.

The impact of case specificity and generalisable skills on clinical performance: a correlated traits-correlated methods approach.

CONTEXT: The finding of case or content specificity in medical problem solving moved the focus of research away from generalisable skills towards the importance of content knowledge. However, controversy about the content dependency of clinical performance and the generalisability of skills remains. OBJECTIVES: This study aimed to explore the relative impact of both perspectives (case specificity and generalisable skills) on different components (history taking, physical examination, communication) of clinical performance within and across cases. METHODS: Data from a clinical performance examination (CPX) taken by 350 Year 3 students were used in a correlated traits-correlated methods (CTCM) approach using confirmatory factor analysis, whereby 'traits' refers to generalisable skills and 'methods' to individual cases. The baseline CTCM model was analysed and compared with four nested models using structural equation modelling techniques. The CPX consisted of three skills components and five cases. RESULTS: Comparison of the four different models with the least-restricted baseline CTCM model revealed that a model with uncorrelated generalisable skills factors and correlated case-specific knowledge factors represented the data best. The generalisable processes found in history taking, physical examination and communication were responsible for half the explained variance, in comparison with the variance related to case specificity. Conclusions Pure knowledge-based and pure skill-based perspectives on clinical performance both seem too one-dimensional and new evidence supports the idea that a substantial amount of variance contributes to both aspects of performance. It could be concluded that generalisable skills and specialised knowledge go hand in hand: both are essential aspects of clinical performance.

Is clinical competence perceived differently for student daily performance on the wards versus clerkship grading?

Clinical rotations play an important role in the medical curriculum and are considered crucial for student learning. However, competencies that should be learned can differ from those that are assessed. In order to explore which competencies are considered important for daily performance of student on the wards and to what extent clinical teachers consider the same competencies important for clerkship grading, a survey that consisted of 21 different student characteristics was administered to clinical teachers. Two independent factor analyses using structural equation modeling were conducted to abstract underlying latent relationships among the different student characteristics and to define a clinical competence profile for daily performance of students on the wards and clerkship grading. Differences between the degree of importance for student daily ward performance and clerkship grading are considered and discussed. The results of the survey indicate that the degree of importance of competencies are rated different for daily performance of students on the wards and clerkship grades. Competencies related to the diagnostic process are more important for clerkship grading, whereas interpersonal skills, professional qualities, and motivation are more important for daily ward performance. It is concluded that the components of clinical competence considered important for adequate performance are not necessarily in alignment with what is required for grading. Future research should focus on an explanation why clinical educators think differently about the importance of competencies for student examination in contrast to what is required for adequate daily performance on the wards.

Two perspectives on the effects of a curriculum change: student experience and the United States medical licensing examination, step 1.

BACKGROUND: Students' perceptions of curricular experience and study effort were compared for a traditional and a new integrative, interdisciplinary curriculum at a single institution. United States Medical Licensing Exam (USMLE) Step 1 scores were examined for subgroup interactions. METHOD: Medical students from four cohorts completed an educational goals survey and USMLE Step 1. Analysis included subgroup performance based on admissions data. RESULTS: Students rated the new curriculum as more helpful in achieving educational goals. USMLE Step 1 was significantly higher for students in the lowest quartile of MCAT scores in the new compared with the previous curriculum. CONCLUSIONS: To understand the outcomes of a large-scale curricular intervention, interactions of curriculum and aptitude should be examined.

In-depth learning: one school's initiatives to foster integration of ethics, values, and the human dimensions of medicine.

Today's medical student curriculum is a lock-step experience that provides a broad survey of medicine with little opportunity to pursue fully integrated, in-depth learning. To teach students about the human dimensions of health care, many schools simply have added courses that survey general areas such as ethics, values, and patient-doctor relationships. However, a superficial, broad-brush approach does not offer students sufficient opportunity to engage with these topics in substantive and meaningful ways. The authors propose that a theme-based, individualized, in-depth learning experience (in which students pursue a focused project comprehensively and in detail)--one that is an integral part of the curriculum--helps students learn to blend values and ethics with medicine in a way that cannot occur during rapid-paced topical survey courses. Furthermore, it is in the depths of a learning experience that one comes face to face with the realities of uncertainty: the realization that unanswerable questions outnumber answerable ones; the awareness of the difficulty in accumulating sufficient evidence to answer a question that is, in fact, answerable; the recognition that many patients' problems transcend available evidence and must be addressed by the art of medicine; the realization that a patient can have a condition that one cannot diagnose and that may even get better for reasons that one cannot understand. The authors describe three initiatives at the University of Pittsburgh School of Medicine, two of which have been offered for more than 10 years, that illustrate the value of in-depth learning experiences. These in-depth experiences blend situated learning, reflective exercises, faculty mentoring, critical reading of literature, and constructive feedback in a prescribed but individualized curriculum.

Clinical competence: general ability or case-specific?

BACKGROUND: Before the 1970s, research into the development of clinical competence was mainly focused on general problem-solving abilities. The scope of research changed when Elstein and colleagues discovered that individual ability to solve clinical problems varies considerably across cases. It was concluded that problem solving abilities are highly dependent on domain-specific knowledge rather than on general problem solving skills. Elstein called this phenomenon "case specificity." PURPOSE: The finding of content specificity will be contrasted with the existence of a general clinical problem solving ability, and the relationship between preclinical knowledge and a problem solving ability will be investigated. METHODS: A correlation matrix was calculated with clerkship final scores from 10 disciplines to examine the magnitude of the interrelations. A confirmatory factor analysis was applied to the corresponding covariances using structural equation modeling to investigate whether scores on finals shared any common variance across clerkships. Finally, two additional models were tested to examine the nature of the relationship between preclinical knowledge and problem solving. RESULTS: Low to moderate correlations across clerkship disciplines were found, supporting the original findings of content specificity. Further investigation showed that in addition to specific knowledge, a general, content-independent ability is needed to perform on these examinations. CONCLUSIONS: Clinical competence, as measured in this study, is based on a combination of specific preclinical knowledge and a problem-solving ability. Case specificity fits perfectly well in this interactional perspective on clinical problem solving but does not explain it. The phenomenon "case specificity" is therefore not solely a result of content knowledge, but of level of experience and level of case difficulty.

Differential effects of two types of formative assessment in predicting performance of first-year medical students.

Formative assessments are systematically designed instructional interventions to assess and provide feedback on students' strengths and weaknesses in the course of teaching and learning. Despite their known benefits to student attitudes and learning, medical school curricula have been slow to integrate such assessments into the curriculum. This study investigates how performance on two different modes of formative assessment relate to each other and to performance on summative assessments in an integrated, medical-school environment. Two types of formative assessment were administered to 146 first-year medical students each week over 8 weeks: a timed, closed-book component to assess factual recall and image recognition, and an un-timed, open-book component to assess higher order reasoning including the ability to identify and access appropriate resources and to integrate and apply knowledge. Analogous summative assessments were administered in the ninth week. Models relating formative and summative assessment performance were tested using Structural Equation Modeling. Two latent variables underlying achievement on formative and summative assessments could be identified; a "formative-assessment factor" and a "summative-assessment factor," with the former predicting the latter. A latent variable underlying achievement on open-book formative assessments was highly predictive of achievement on both open- and closed-book summative assessments, whereas a latent variable underlying closed-book assessments only predicted performance on the closed-book summative assessment. Formative assessments can be used as effective predictive tools of summative performance in medical school. Open-book, un-timed assessments of higher order processes appeared to be better predictors of overall summative performance than closed-book, timed assessments of factual recall and image recognition.

Influence of clerkship experiences on clinical competence.

BACKGROUND: Clerkship experiences are considered crucial for the development of clinical competence. Yet whether there is a direct relationship between the nature and volume of patient encounters and learning outcomes is far from clear. Some evidence in the literature points towards the importance of clinical supervision on student learning, but the relationship between clinical supervision, patient encounters and student competence remains unclear. OBJECTIVES: This study aimed firstly to determine the variation in students' clinical experiences within and across sites; secondly, to identify the causes of this variation, and thirdly, to investigate the consequences of this variation on students' competence. METHODS: Clerkship students at 12 hospital sites recorded their patient encounters in logbooks. Site characteristics that might influence the variation in patient encounters were collected. Student competence was determined by 3 independent indicators: a practical end-of-clerkship examination; a theoretical end-of-clerkship examination, and an evaluation of professional performance. A model was developed to test the available clerkship data using structural equation modelling (SEM) software. RESULTS: Analysis of the logbooks revealed a large variation in the number of patients encountered by students. The average length of patient stay, number of patients admitted, and quality of supervision accounted partly for this variation. An increased number of patient encounters did not directly lead to improved competence. Quality of supervision turned out to be crucially important because it directly impacted student learning and also positively influenced the number of patient encounters. CONCLUSION: Monitoring the effectiveness of clerkship by merely asking students to keep a tally of the problems and diseases they encounter, without paying attention to the quality of supervision, does not contribute towards improving student learning.

Inducing expertise effects in clinical case recall.

BACKGROUND: This study was directed at illuminating a well known phenomenon in the medical expertise literature, the 'intermediate effect' in clinical case recall. This robust phenomenon consists of the finding that medical students of intermediate levels of expertise outperform both experts and novices in clinical case recall after diagnosing cases. It deals in particular with the findings of OME researchers who have reported a monotonically increasing recall with level of expertise. PURPOSE: To address possible causes for this anomaly in medical expertise and to experimentally demonstrate how data elaboration can cause expertise effects in clinical case recall. METHOD: Expert nephrologists, intermediate level students and novices were presented with 6 medical cases under 3 different conditions: laboratory data cases without special instructions, laboratory data cases with instructions to elaborate, and cases with laboratory data and a relevant clinical context. RESULTS: Only when participants were required to elaborate on each of the information units presented to them did case recall show an expertise effect. If laboratory data are framed within the context of a patient's history and physical examination data, the 'intermediate effect' appears. CONCLUSIONS: The instructions used in the elaboration condition seem to have induced a deeper, more detailed, analysis of the patient case. It is therefore interesting to note that these instructions only affected the recall of the experts and had no effect on the novices' or intermediates' recall. We might conclude from this that expertise effects in clinical case recall are only produced when the normal processing of patient information is disrupted.