Medical Student Experiences With Electronic Health Records Nationally: A Longitudinal Analysis Including School-Level Effects.

PURPOSE: Increasingly, medical school graduates have been expected to be competent in the knowledge and skills associated with effective electronic health record (EHR) use. Yet little is known about how student experiences with EHRs have changed over time or how these trends vary by medical school. This study examined shifts in U.S. medical student interactions with EHRs during their clinical education, 2012-2016, and how these interactions varied by clerkship within and across medical schools. METHOD: Data came from an online survey about EHR use administered to medical students after completing the Step 2 Clinical Knowledge portion of the United States Medical Licensing Examination. For a sample of 17,202 medical students from 140 U.S.-based Liaison Committee on Medical Education accredited medical school campuses, multilevel modeling techniques were used to estimate overall and school-specific trends in student access to and entry of information into EHRs for 7 core inpatient clerkships. RESULTS: Results showed upward trajectories in likelihood of student EHR experiences, with smaller increases found for information entry compared with access. These trends varied by inpatient clerkship rotation, with some disciplines exhibiting more rapid increases than others. For both access and information entry, estimated clerkship-specific trajectories differed by medical school in terms of size and direction. For all clerkships, greater school-to-school variation in the likelihood of student entry, compared with student access, remained at the end of the study period. CONCLUSIONS: Increases in medical student interactions with EHRs suggest a growing commitment to educating students on safe and effective EHR use. Nonetheless, at some schools and in some clerkships, students may receive inadequate educational opportunities to practice using EHRs. In turn, medical students may be differentially prepared to effectively engage with EHRs upon entering residency, particularly with the knowledge and skills needed to effectively document and transmit information in EHRs.

Surgical Trainee Performance and Alignment With Surgical Program Director Expectations.

OBJECTIVE: To examine the alignment between graduating surgical trainee operative performance and a prior survey of surgical program director expectations. BACKGROUND: Surgical trainee operative training is expected to prepare residents to independently perform clinically important surgical procedures. METHODS: We conducted a cross-sectional observational study of US general surgery residents' rated operative performance for Core general surgery procedures. Residents' expected performance on those procedures at the time of graduation was compared to the current list of Core general surgery procedures ranked by their importance for clinical practice, as assessed via a previous national survey of general surgery program directors. We also examined the frequency of individual procedures logged by residents over the course of their training. RESULTS: Operative performance ratings for 29,885 procedures performed by 1861 surgical residents in 54 general surgery programs were analyzed. For each Core general surgery procedure, adjusted mean probability of a graduating resident being deemed practice-ready ranged from 0.59 to 0.99 (mean 0.90, standard deviation 0.08). There was weak correlation between the readiness of trainees to independently perform a procedure at the time of graduation and that procedure's historical importance to clinical practice ( p = 0.22, 95% confidence interval 0.01-0.41, P = 0.06). Residents also continue to have limited opportunities to learn many procedures that are important for clinical practice. CONCLUSION: The operative performance of graduating general surgery residents may not be well aligned with surgical program director expectations.

A Proposed Blueprint for Operative Performance Training, Assessment, and Certification.

OBJECTIVE: The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees. SUMMARY BACKGROUND DATA: Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved. METHODS: The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties. RESULTS: Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items. CONCLUSIONS: A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust.

Medical student experiences with accessing and entering patient information in electronic health records during the obstetrics-gynecology clerkship.

BACKGROUND: Medical school graduates should be able to enter information from patient encounters and to write orders and prescriptions in the electronic health record. Studies have shown that, although students often can access electronic health records, some students may receive inadequate preparation for these skills. Greater understanding of student exposure to electronic health records during their obstetrics and gynecology clerkships can help to determine the extent to which students receive the educational experiences that may best prepare them for their future training and practice. OBJECTIVE: The purpose of this study was to examine medical student reporting of electronic health record use during the obstetrics and gynecology clerkship. STUDY DESIGN: A Step 2 Clinical Knowledge End-of-Examination Survey about electronic health record use was administered to medical students after they completed the Step 2 Clinical Knowledge component of the United States Medical Licensing Examination. For inpatient and outpatient rotations, students were asked if they accessed a record and if they entered notes or orders into it. Descriptive statistics for a sample of 16,366 medical students who graduated from Liaison Committee on Medical Education-accredited schools from 2012-2016 summarize student interactions with electronic health records by rotation type and graduation year. Chi-square techniques were used to examine mean differences in access and entry. RESULTS: The survey had an overall response rate of 70%. In 2016, most survey respondents (94%) accessed electronic health records during their obstetrics and gynecology clerkship, but 26% of them reported "read-only" access. On the inpatient service, <10% of students reported any order entry; 58% of them reported entering progress notes, and 47% of them reported entering an admitting history and physical. CONCLUSION: Medical school graduates who are entering obstetrics and gynecology residencies are expected to be competent in documenting clinical encounters and entering orders, including those that are unique to obstetrics and gynecology. This study shows that some students may receive less experience with entering information into electronic health records during their obstetrics and gynecology clerkships than others, which could result in unequal levels of preparedness for graduate medical education.

Number of Operative Performance Ratings Needed to Reliably Assess the Difficulty of Surgical Procedures.

OBJECTIVE: The profession of surgery is entering a new era of "big data," where analyses of longitudinal trainee assessment data will be used to inform ongoing efforts to improve surgical education. Given the high-stakes implications of these types of analyses, researchers must define the conditions under which estimates derived from these large datasets remain valid. With this study, we determine the number of assessments of residents' performances needed to reliably assess the difficulty of "Core" surgical procedures. DESIGN: Using the SIMPL smartphone application from the Procedural Learning and Safety Collaborative, 402 attending surgeons directly observed and provided workplace-based assessments for 488 categorical residents after 5259 performances of 87 Core surgical procedures performed at 14 institutions. We used these faculty ratings to construct a linear mixed model with resident performance as the outcome variable and multiple predictors including, most significantly, the operative procedure as a random effect. We interpreted the variance in performance ratings attributable to the procedure, after controlling for other variables, as the "difficulty" of performing the procedure. We conducted a generalizability analysis and decision study to estimate the number of SIMPL performance ratings needed to reliably estimate the difficulty of a typical Core procedure. RESULTS: Twenty-four faculty ratings of resident operative performance were necessary to reliably estimate the difficulty of a typical Core surgical procedure (mean dependability coefficient 0.80, 95% confidence interval 0.73-0.87). CONCLUSIONS: At least 24 operative performance ratings are required to reliably estimate the difficulty of a typical Core surgical procedure. Future research using performance ratings to establish procedure difficulty should include adequate numbers of ratings given the high-stakes implications of those results for curriculum design and policy.

Electronic Health Record Use in Internal Medicine Clerkships and Sub-internships for Medical Students Graduating from 2012 to 2016.

BACKGROUND: As electronic health records (EHRs) became broadly available in medical practice, effective use of EHRs by medical students emerged as an essential aspect of medical education. While new federal clinical documentation guidelines have the potential to encourage greater medical student EHR use and enhance student learning experiences with respect to EHRs, little is known nationally about how students have engaged with EHRs in the past. OBJECTIVE: This study examines medical student accounts of EHR use during their internal medicine (IM) clerkships and sub-internships during a 5-year time period prior to the new clinical documentation guidelines. DESIGN: An online survey about EHR use was administered to medical students immediately after they completed USMLE Step 2 CK. PARTICIPANTS: The sample included 16,602 medical students planning to graduate from US medical schools from 2012 to 2016. MAIN MEASURES: Descriptive statistics were computed to determine the average percentage of students engaged in various health record activities during their IM educational experiences by graduation year. KEY RESULTS: The vast majority (99%) of medical students used EHRs during IM clerkships or sub-internships. Most students reported that they entered information into EHRs during the inpatient component of the IM clerkship (84%), outpatient component of the IM clerkship (70%), and the IM sub-internship (92%). Yet, 43% of the students who graduated in 2016 never entered admission orders and 35% of them never entered post-admission orders. CONCLUSIONS: Medical school graduates ought to be able to effectively document clinical encounters and enter orders into EHR systems. Although most students used and entered information into EHRs during their IM clinical training, many students appear to have received inadequate opportunities to enter notes or orders, in particular. Implications for graduate medical education preparedness are considered. Future research should address similar questions using comparable national data collected after the recent guideline changes.

How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

Medical Student Use of Electronic and Paper Health Records During Inpatient Clinical Clerkships: Results of a National Longitudinal Study.

PURPOSE: An important goal of medical education is to teach students to use an electronic health record (EHR) safely and effectively. The purpose of this study is to examine medical student accounts of EHR use during their core inpatient clinical clerkships using a national sample. Paper health records (PHRs) are similarly examined. METHOD: An online survey about health record use within the inpatient component of six core clerkships was administered to medical students after they completed Step 2 Clinical Knowledge of the United States Medical Licensing Examination. The sample included 17,202 U.S. medical students graduating between 2012 and 2016. Mean percentages of clerkships in which students engaged in various health record activities were computed, and analysis of variance was used to examine differences. RESULTS: The mean percentages of clerkships in which a student accessed or entered information into an EHR increased from 78% to 93% and 59% to 72%, respectively. For students who used an EHR, the mean percentage of clerkships in which they entered information remained constant at 76%. Students entered notes during the majority of their clerkships, with increases over time. However, students entered orders in less than a quarter of their clerkships, with decreases over time. The percentage of clerkships in which students used PHRs was lower and declining. CONCLUSIONS: Although students used an EHR in the majority of their inpatient core clerkships, they received limited educational experiences related to order and note writing, which could translate into a lack of preparedness for future training and practice.

Longitudinal assessments in continuing specialty certification and lifelong learning.

Longitudinal assessment (LA) involves the regular, spaced delivery of a limited number of questions on practice relevant content on a computer or mobile internet platform. Depending on the platform, participants may indicate relevance of the content to their practice and confidence in their answer prior to receiving immediate feedback (including critiques) on each question. Individual dashboards may be included to assist participants in tracking progress and identifying areas of strength and weaknesss across a content blueprint. This paper provides an overview of the theoretical underpinnings underlying LA programs, briefly describes current uses of LA in medicine and suggests areas for evaluating the role of LA in continuing medical specialty certification and continuing professional development.

Exploring the Relationships Between USMLE Performance and Disciplinary Action in Practice: A Validity Study of Score Inferences From a Licensure Examination.

PURPOSE: Physicians must pass the United States Medical Licensing Examination (USMLE) to obtain an unrestricted license to practice allopathic medicine in the United States. Little is known, however, about how well USMLE performance relates to physician behavior in practice, particularly conduct inconsistent with safe, effective patient care. The authors examined the extent to which USMLE scores relate to the odds of receiving a disciplinary action from a U.S. state medical board. METHOD: Controlling for multiple factors, the authors used non-nested multilevel logistic regression analyses to estimate the relationships between scores and receiving an action. The sample included 164,725 physicians who graduated from U.S. MD-granting medical schools between 1994 and 2006. RESULTS: Physicians had a mean Step 1 score of 214 (standard deviation [SD] = 21) and a mean Step 2 Clinical Knowledge (CK) score of 213 (SD = 23). Of the physicians, 2,205 (1.3%) received at least one action. Physicians with higher Step 2 CK scores had lower odds of receiving an action. A 1-SD increase in Step 2 CK scores corresponded to a decrease in the chance of disciplinary action by roughly 25% (odds ratio = 0.75; 95% CI = 0.70-0.80). After accounting for Step 2 CK scores, Step 1 scores were unrelated to the odds of receiving an action. CONCLUSIONS: USMLE Step 2 CK scores provide useful information about the odds a physician will receive an official sanction for problematic practice behavior. These results provide validity evidence supporting current interpretation and use of Step 2 CK scores.

Trends in national licensing examinations in medicine.

CONTEXT: As a contribution to this special issue commemorating the journal's 50th volume, this paper seeks to explore directions for national licensing examinations (NLEs) in medicine. Increases in the numbers of new medical schools and the mobility of doctors across national borders mean that NLEs are becoming even more important to ensuring physician competence. OBJECTIVES: The purpose of this paper is to explore the use of NLEs in the future in the context of global changes in medical education and health care delivery. METHODS: Because the literature related to NLEs is so large, we have not attempted a comprehensive review, but have focused instead on a small number of topics on which we think we have something useful to say. The paper is organised around five predicted trends for NLEs. DISCUSSION: The first section discusses reasons why we think the use of NLEs will increase in the coming years. The second section discusses the ongoing problem of content specificity and its implications for the design of NLEs. The third section examines the evolution of large-scale, standardised cognitive assessments in NLEs and suggests some future directions. Reflecting the fact that NLEs are, increasingly, attempting to assess more than just knowledge, the fourth section addresses the future of large-scale clinical skills assessments in NLEs, predicting both increases in their use and some shifts in the nature of the stations used. The fifth section discusses workplace-based assessments, predicting increases in their use for formative assessment and identifying some limitations in their direct application in NLEs. The concluding section discusses the cost of NLEs and indulges in some further speculations about their evolution.

US Medical Student Performance on the NBME Subject Examination in Internal Medicine: Do Clerkship Sequence and Clerkship Length Matter?

BACKGROUND: Prior to graduation, US medical students are required to complete clinical clerkship rotations, most commonly in the specialty areas of family medicine, internal medicine, obstetrics and gynecology (ob/gyn), pediatrics, psychiatry, and surgery. Within a school, the sequence in which students complete these clerkships varies. In addition, the length of these rotations varies, both within a school for different clerkships and between schools for the same clerkship. OBJECTIVE: The present study investigated the effects of clerkship sequence and length on performance on the National Board of Medical Examiner's subject examination in internal medicine. PARTICIPANTS: The study sample included 16,091 students from 67 US Liaison Committee on Medical Education (LCME)-accredited medical schools who graduated in 2012 or 2013. MAIN MEASURES: Student-level measures included first-attempt internal medicine subject examination scores, first-attempt USMLE Step 1 scores, and five dichotomous variables capturing whether or not students completed rotations in family medicine, ob/gyn, pediatrics, psychiatry, and surgery prior to taking the internal medicine rotation. School-level measures included clerkship length and average Step 1 score. DESIGN: Multilevel models with students nested in schools were estimated with internal medicine subject examination scores as the dependent measure. Step 1 scores and the five dichotomous variables were treated as student-level predictors. Internal medicine clerkship length and average Step 1 score were used to predict school-to-school variation in average internal medicine subject examination scores. KEY RESULTS: Completion of rotations in surgery, pediatrics and family medicine prior to taking the internal medicine examination significantly improved scores, with the largest benefit observed for surgery (coefficient = 1.58 points; p value < 0.01); completion of rotations in ob/gyn and psychiatry were unrelated to internal medicine subject examination performance. At the school level, longer internal medicine clerkships were associated with higher scores on the internal medicine examination (coefficient = 0.23 points/week; p value < 0.01). CONCLUSIONS: The order in which students complete clinical clerkships and the length of the internal medicine clerkship are associated with their internal medicine subject examination scores. Findings may have implications for curriculum re-design.

Team cohesiveness, team size and team performance in team-based learning teams.

OBJECTIVES: The purpose of this study was to explore the relationships among variables associated with teams in team-based learning (TBL) settings and team outcomes. METHODS: We administered the National Board of Medical Examiners (NBME) Psychiatry Subject Test first to individuals and then to teams of Year three students at four medical schools that used TBL in their psychiatry core clerkships. Team cohesion was analysed using the Team Performance Scale (TPS). Bivariate correlation and linear regression analysis were used to analyse the relationships among team-level variables (mean individual TPS scores for each team, mean individual NBME scores of teams, team size, rotation and gender make-up) and team NBME test scores. A hierarchical linear model was used to test the effects of individual TPS and individual NBME test scores within each team, as well as the effects of the team-level variables of team size, team rotation and gender on team NBME test scores. Individual NBME test and TPS scores were nested within teams and treated as subsampling units. RESULTS: Individual NBME test scores and individual TPS scores were positively and statistically significantly (p < 0.01) associated with team NBME test scores, when team rotation, team size and gender make-up were controlled for. Higher team NBME test scores were associated with teams rotating later in the year and larger teams (p < 0.01). Gender make-up was not significantly associated. CONCLUSIONS: The results of an NBME Psychiatry Subject Test administered to TBL teams at four medical schools suggest that larger teams on later rotations score higher on a team NBME test. Individual NBME test scores and team cohesion were positively and significantly associated with team NBME test scores. These results suggest the need for additional studies focusing on team outcomes, team cohesion, team size, rotation and other factors as they relate to the effective and efficient performance of TBL teams in health science education.

International variation in performance by clinical discipline and task on the United States medical licensing examination step 2 clinical knowledge component.

PURPOSE: To investigate country-to-country variation in performance across clinical science disciplines and tasks for examinees taking the Step 2 Clinical Knowledge (CK) component of the United States Medical Licensing Examination. METHOD: In 2012 the authors analyzed demographic characteristics, total scores, and percent-correct clinical science discipline and task scores for more than 88,500 examinees taking Step 2 CK for the first time during the 2008-2010 academic years. For each examinee and score, differences between the score and the mean performance of examinees at U.S. MD-granting medical schools were calculated, and mean differences by country of medical school were tabulated for analysis of country-to-country variation in performance by clinical discipline and task. RESULTS: Controlling for overall performance relative to U.S. examinees, results showed that international medical graduates (IMGs) performed best in Surgery and worst in Psychiatry for clinical discipline scores; for clinical tasks, IMGs performed best in Understanding Mechanisms of Disease and worst in Promoting Preventive Medicine and Health Maintenance. The pattern of results was strongest for IMGs attending schools in the Middle East and Australasia, present to a lesser degree for IMGs attending schools in Europe, and absent for IMGs attending Caribbean medical schools. CONCLUSIONS: Country-to-country differences in relative performance were present for both clinical discipline and task scores. Possible explanations include differences in learning outcomes, curriculum emphasis and clinical experience, standards of care, and culture, as well as the effects of English as a second language and relative emphasis on preparing students to take the Step 2 CK exam.

Changes in anatomy instruction and USMLE performance: empirical evidence on the absence of a relationship.

Anatomy instruction has evolved over the past two decades as many medical schools have undergone various types of curricular reform. To provide empirical evidence about whether or not curricular changes impact the acquisition and retention of anatomy knowledge, this study investigated the effect of variation in gross anatomy course hours, curricular approach (stand-alone versus integrated), and laboratory experience (dissection versus dissection and prosection) on USMLE Steps 1 and 2 Clinical Knowledge (CK) scores. Gross anatomy course directors at 54 United States schools provided information about their gross anatomy courses via an online survey (response rate of 42%). Survey responses were matched with USMLE scores for 6,411 examinees entering LCME-accredited schools in 2007 and taking Step 1 for the first time in 2009. Regression analyses were conducted to examine relationships between gross anatomy instructional characteristics and USMLE performance. Step 1 total scores, Step 1 gross anatomy sub-scores, and Step 2 CK scores were unrelated to instructional hours, controlling for MCAT scores. Examinees from schools with integrated curricula scored slightly lower on Steps 1 and 2 CK than those from stand-alone courses (effect sizes of 2.1 and 1.9 on score scales with SDs of 22 and 20, respectively). Examinees with dissection and prosection experience performed slightly better on Step 2 CK than examinees in courses with dissection only laboratories (effect size of 1.2). Results suggest variation in course hours is unrelated to performance on Steps 1 and 2 CK. Although differences were observed in relation to curricular approach and laboratory experience, effect sizes were small.

The relationship between the american board of anesthesiology part 1 certification examination and the United States medical licensing examination.

BACKGROUND: The graduate medical education community uses results from the United States Medical Licensing Examination (USMLE) to inform decisions about individuals' readiness for postgraduate training. OBJECTIVE: We sought to determine the relationship between performance on the USMLE and the American Board of Anesthesiology (ABA) Part 1 Certification Examination using a national sample of examinees, and we considered the relationship in the context of undergraduate medical education location and examination content. METHODS: Approximately 7800 individuals met inclusion criteria. The relationships between USMLE scores and ABA Part 1 pass rates were examined, and predictions for the strength of the relationship between USMLE content areas and ABA performance were compared with observed relationships. RESULTS: Pearson correlations between ABA Part 1 scores and USMLE Steps 1, 2 (clinical knowledge), and 3 scores for first-taker US/Canadian graduates were .59, .56, and .53, respectively. A clear relationship was demonstrated between USMLE scores and pass rates on ABA Part 1, and content experts were able to successfully predict the USMLE content categories that would least or most likely relate to ABA Part 1 scores. CONCLUSIONS: The analysis provided evidence on a national scale that results from the USMLE and the ABA Part 1 were correlated and that success on the latter examination was associated with level of USMLE performance. Both testing programs have been successful in conceptualizing many of the knowledge areas of interest and in developing test content to reflect those areas.

A multilevel analysis of examinee gender, standardized patient gender, and United States medical licensing examination step 2 clinical skills communication and interpersonal skills scores.

BACKGROUND: Women typically demonstrate stronger communication skills on performance-based assessments using human raters in medical education settings. This study examines the effects of examinee and rater gender on communication and interpersonal skills (CIS) scores from the performance-based component of the United States Medical Licensing Examination, the Step 2 Clinical Skills (CS) examination. METHOD: Data included demographic and performance information for examinees that took Step 2 CS for the first time in 2009. The sample contained 27,910 examinees, 625 standardized patient/case combinations, and 278,776 scored patient encounters. Hierarchical linear modeling techniques were employed with CIS scores as the outcome measure. RESULTS: Females tend to slightly outperform males on CIS, when other variables related to performance are taken into account. No evidence of an examinee and rater gender interaction effect was found. CONCLUSIONS: Results provide validity evidence supporting the interpretation and use of Step 2 CS CIS scores.

The relationship between USMLE Step 2 CS patient note ratings and time spent on the note: do examinees who spend more time write better notes?

BACKGROUND: During the United States Medical Licensing Examination Step 2 Clinical Skills examination, examinees rotate through 12 standardized patient (SP) encounters. Examinees have 25 minutes per encounter to interact with SPs and complete postencounter patient notes (PNs), and they may end the SP interaction early to spend extra time on the PN. The current work assesses the time examinees are spending on PNs and whether this is related to performance on the PN. METHOD: Encounters from 2,479 examinees' videos were time-stamped to indicate total encounter time and PN time. Hierarchical linear modeling was employed to assess how well total and PN time, along with other examinee and case-rater variables, predicted PN scores. RESULTS: Examinee variables explained a significant portion of within-case-rater variability, but while PN time was significantly related to PN ratings, the effect was small. CONCLUSIONS: The results suggest that spending additional time on the PN does not translate to a meaningful score increase.

Progress testing in clinical science education: results of a pilot project between the National Board of Medical Examiners and a US Medical School.

BACKGROUND: Though progress tests have been used for several decades in various medical education settings, a few studies have offered analytic frameworks that could be used by practitioners to model growth of knowledge as a function of curricular and other variables of interest. AIM: To explore the use of one form of progress testing in clinical education by modeling growth of knowledge in various disciplines as well as by assessing the impact of recent training (core rotation order) on performance using hierarchical linear modeling (HLM) and analysis of variance (ANOVA) frameworks. METHODS: This study included performances across four test administrations occurring between July 2006 and July 2007 for 130 students from a US medical school who graduated in 2008. Measures-nested-in-examinees HLM growth curve analyses were run to estimate clinical science knowledge growth over time and repeated measures ANOVAs were run to assess the effect of recent training on performance. RESULTS: Core rotation order was related to growth rates for total and pediatrics scores only. Additionally, scores were higher in a given discipline if training had occurred immediately prior to the test administration. CONCLUSIONS: This study provides a useful progress testing framework for assessing medical students' growth of knowledge across their clinical science education and the related impact of training.