Assessing the growth in clinical skills using a progress clinical skills examination.

BACKGROUND: This study evaluates the generalizability of an eight-station progress clinical skills examination and assesses the growth in performance for six clinical skills domains among first- and second-year medical students over four time points during the academic year. METHODS: We conducted a generalizability study for longitudinal and cross-sectional comparisons and assessed growth in six clinical skill domains via repeated measures ANOVA over the first and second year of medical school. RESULTS: The generalizability of the examination domain scores was low but consistent with previous studies of data gathering and communication skills. Variations in case difficulty across administrations of the examination made it difficult to assess longitudinal growth. It was possible to compare students at different training levels and the interaction of training level and growth. Second-year students outperformed first-year students, but first-year students' clinical skills performance grew faster than second-year students narrowing the gap in clinical skills over the students' first year of medical school. CONCLUSIONS: Case specificity limits the ability to assess longitudinal growth in clinical skills through progress testing. Providing students with early clinical skills training and authentic clinical experiences appears to result in the rapid growth of clinical skills during the first year of medical school.

Attendance, engagement and performance in a medical school curriculum: early findings from competency-based progress testing in a new medical school curriculum.

INTRODUCTION: Medical students often do not value attending in-person large group sessions. It is also not clear from prior research whether attendance at large group sessions impact on performance in medical school. The goal of this study was to assess the relationship between voluntary attendance in large group sessions organized as a "flipped classroom" in a new innovative curriculum and students' mastery of clinical applications of basic science knowledge. METHODOLOGY: Our students' ability to apply basic science knowledge to clinical problems is assessed via progress testing using three methodologies: a locally developed multiple-choice examination, written examination developed through the National Board of Medical Examiners (NBME) Customized Assessment Services Program and post encounter questions included in a clinical skills examination. We analyzed the relationship between voluntary attendance at weekly large group "flipped classroom" sessions and the students' performance on examinations given at four intervals over the initial 24-week module of the medical school curriculum. RESULTS: Complete data were available for 167 students. A total of 82 students (49.1%) attended all large group sessions, 65 students (38.9%) missed one or two sessions and 20 students (12.0%) missed three or more sessions. There were no difference between the students in the groups on their medical admission (MCAT) examination scores. The growth in performance from each time point until the next was statistically significant. There was no statistically significant difference in growth between the students who had no absences and those who had one or two absences. Students who missed three or more sessions performed significantly lower than their peers over the 24 week module and were more likely to score one or more standard deviations below the class mean on the assessments. CONCLUSIONS: We found no relationship between attendance and MCAT scores suggesting the differences in performance on the progress tests was not due to initial differences in knowledge or reasoning skills. While the study was not experimental, it suggests large group sessions using a "flipped classroom" approach to provide reinforcement, feedback and practice may be effective for increasing learning and retention in the application of basic science knowledge among first year medical students.

A survey of authors publishing in four megajournals.

Aim. To determine the characteristics of megajournal authors, the nature of the manuscripts they are submitting to these journals, factors influencing their decision to publish in a megajournal, sources of funding for article processing charges (APCs) or other fees and their likelihood of submitting to a megajournal in the future. Methods. Web-based survey of 2,128 authors who recently published in BMJ Open, PeerJ, PLOS ONE or SAGE Open. Results. The response rate ranged from 26% for BMJ Open to 47% for SAGE Open. The authors were international, largely academics who had recently published in both subscription and Open Access (OA) journals. Across journals about 25% of the articles were preliminary findings and just under half were resubmissions of manuscripts rejected by other journals. Editors from other BMJ journals and perhaps to a lesser extent SAGE and PLOS journals appear to be encouraging authors to submit manuscripts that were rejected by the editor's journals to a megajournal published by the same publisher. Quality of the journal and speed of the review process were important factors across all four journals. Impact factor was important for PLOS ONE authors but less so for BMJ Open authors, which also has an impact factor. The review criteria and the fact the journal was OA were other significant factors particularly important for PeerJ authors. The reputation of the publisher was an important factor for SAGE Open and BMJ Open. About half of PLOS ONE and around a third of BMJ Open and PeerJ authors used grant funding for publishing charges while only about 10% of SAGE Open used grant funding for publication charges. Around 60% of SAGE Open and 32% of PeerJ authors self-funded their publication fees however the fees are modest for these journals. The majority of authors from all 4 journals were pleased with their experience and indicated they were likely to submit to the same or similar journal in the future. Conclusions. Megajournals are drawing an international group of authors who tend to be experienced academics. They are choosing to publish in megajournals for a variety of reasons but most seem to value the quality of the journal and the speed of the review/publication process. Having a broad scope was not a key factor for most authors though being OA was important for PeerJ and SAGE Open authors. Most authors appeared pleased with the experience and indicated they are likely to submit future manuscripts to the same or similar megajournal which seems to suggest these journals will continue to grow in popularity.

Glycohemoglobin A1c: A promising screening tool in gestational diabetes mellitus.

CONTEXT: Current screening tests for gestational diabetes mellitus (GDM) are inconvenient. Therefore, alternative screening tests for GDM are desirable. The use of glycohemoglobin A1c (HbA1c) in screening for GDM remains controversial. AIM: We undertook this study to evaluate the utility of HbA1c in screening for GDM. SETTINGS AND DESIGN: Retrospective study in a tertiary teaching hospital. MATERIALS AND METHODS: Laboratory records were reviewed to identify pregnant women who underwent both oral glucose tolerance test (OGTT) and HbA1c measurements over a 16-months period. The association of OGTT with HbA1c was evaluated. STATISTICAL ANALYSIS USED: Data were collected using SPSS software. Comparisons of the means and calculations of sensitivities were performed. RESULTS: Of 145 eligible patients, 124 had GDM and 21 patients did not, per OGTT. The percentages of patients with HbA1c values (reference range of 4.8%-6.0%) equal to or above sequential cut-point values of 5.0%, 5.5%, 6.0%, 6.5% and 7.0% (i.e., sensitivity values) were 100%, 98.4%, 87.1%, 62.9% and 39.5%, respectively. The mean HbA1c of the patients with GDM was 6.9 + 0.8% compared to 6.4 + 0.6% for those without GDM (P< 0.006). At an arbitrary cut-off value of 6.0% (the upper limit of normal), HbA1c would have picked up 87.1% of patients with GDM. CONCLUSIONS: This study suggests that HbA1c is a reasonably sensitive screening measure of GDM in this high-risk population. Acknowledging limitations resulting from the study design, further prospective studies are warranted to verify this conclusion, and to evaluate the specificity of HbA1c as a screening test for GDM.

Sources of measurement error in an ECG examination: implications for performance-based assessments.

OBJECTIVE: To assess the sources of measurement error in an electrocardiogram (ECG) interpretation examination given in a third-year internal medicine clerkship. DESIGN: Three successive generalizability studies were conducted. (1) Multiple faculty rated student responses to a previously administered exam. (2) The rating criteria were revised and study 1 was repeated. (3) The examination was converted into an extended matching format including multiple cases with the same underlying cardiac problem. RESULTS: The discrepancies among raters (main effects and interactions) were dwarfed by the error associated with case specificity. The largest source of the differences among raters was in rating student errors of commission rather than student errors of omission. Revisions in the rating criteria may have helped increase inter-rater reliability slightly however, due to case specificity, it had little impact on the overall reliability of the exam. The third study indicated the majority of the variability in student performance across cases was in performance across cases within the same type of cardiac problem rather than between different types of cardiac problems. CONCLUSIONS: Case specificity was the overwhelming source of measurement error. The variation among cases came mainly from discrepancies in performance between examples of the same cardiac problem rather than from differences in performance across different types of cardiac problems. This suggests it is necessary to include a large number of cases even if the goal is to assess performance on only a few types of cardiac problems.

A randomized trial comparing digital and live lecture formats [ISRCTN40455708.

BACKGROUND: Medical education is increasingly being conducted in community-based teaching sites at diverse locations, making it difficult to provide a consistent curriculum. We conducted a randomized trial to assess whether students who viewed digital lectures would perform as well on a measure of cognitive knowledge as students who viewed live lectures. Students' perceptions of the digital lecture format and their opinion as whether a digital lecture format could serve as an adequate replacement for live lectures was also assessed. METHODS: Students were randomized to either attend a lecture series at our main campus or view digital versions of the same lectures at community-based teaching sites. Both groups completed the same examination based on the lectures, and the group viewing the digital lectures completed a feedback form on the digital format. RESULTS: There were no differences in performance as measured by means or average rank. Despite technical problems, the students who viewed the digital lectures overwhelmingly felt the digital lectures could replace live lectures. CONCLUSIONS: This study provides preliminary evidence digital lectures can be a viable alternative to live lectures as a means of delivering didactic presentations in a community-based setting.

Using a formative simulated patient exercise for curriculum evaluation.

BACKGROUND: It is not clear that teaching specific history taking, physical examination and patient teaching techniques to medical students results in durable behavioural changes. We used a quasi-experimental design that approximated a randomized double blinded trial to examine whether a Participatory Decision-Making (PDM) educational module taught in a clerkship improves performance on a Simulated Patient Exercise (SPE) in another clerkship, and how this is influenced by the time between training and assessment. METHODS: Third year medical students in an internal medicine clerkship were assessed on their use of PDM skills in an SPE conducted in the second week of the clerkship. The rotational structure of the third year clerkships formed a pseudo-randomized design where students had 1) completed the family practice clerkship containing a training module on PDM skills approximately four weeks prior to the SPE, 2) completed the family medicine clerkship and the training module approximately 12 weeks prior to the SPE or 3) had not completed the family medicine clerkship and the PDM training module at the time they were assessed via the SPE. RESULTS: Based on limited pilot data there were statistically significant differences between students who received PDM training approximately four weeks prior to the SPE and students who received training approximately 12 weeks prior to the SPE. Students who received training 12 weeks prior to the SPE performed better than those who received training four weeks prior to the SPE. In a second comparison students who received training four weeks prior to the SPE performed better than those who did not receive training but the differences narrowly missed statistical significance (P < 0.05). CONCLUSION: This pilot study demonstrated the feasibility of a methodology for conducting rigorous curricular evaluations using natural experiments based on the structure of clinical rotations. In addition, it provided preliminary data suggesting targeted educational interventions can result in marked improvements in the clinical skills spontaneously exhibited by physician trainees in a setting different from which the skills were taught.

The rating reliability calculator.

BACKGROUND: Rating scales form an important means of gathering evaluation data. Since important decisions are often based on these evaluations, determining the reliability of rating data can be critical. Most commonly used methods of estimating reliability require a complete set of ratings i.e. every subject being rated must be rated by each judge. Over fifty years ago Ebel described an algorithm for estimating the reliability of ratings based on incomplete data. While his article has been widely cited over the years, software based on the algorithm is not readily available. This paper describes an easy-to-use Web-based utility for estimating the reliability of ratings based on incomplete data using Ebel's algorithm. METHODS: The program is available public use on our server and the source code is freely available under GNU General Public License. The utility is written in PHP, a common open source imbedded scripting language. The rating data can be entered in a convenient format on the user's personal computer that the program will upload to the server for calculating the reliability and other statistics describing the ratings. RESULTS: When the program is run it displays the reliability, number of subject rated, harmonic mean number of judges rating each subject, the mean and standard deviation of the averaged ratings per subject. The program also displays the mean, standard deviation and number of ratings for each subject rated. Additionally the program will estimate the reliability of an average of a number of ratings for each subject via the Spearman-Brown prophecy formula. CONCLUSION: This simple web-based program provides a convenient means of estimating the reliability of rating data without the need to conduct special studies in order to provide complete rating data. I would welcome other researchers revising and enhancing the program.

Global Ratings of Student Performance in a Standardized Patient Examination: Is the Whole More than the Sum of the Parts?

Standardized patient examinations (SPE) are widely used in medical education to assess skills that cannot be measured with written examinations. Trained actors termed standardized patients (SPs) are used to simulate patients with specific medical problems. SPs typically use behaviorally specific checklists and rating scales to evaluate examinees. This study explored the use of faculty and SP global ratings of students' clinical and interpersonal skills in an SPE. The reliability of global ratings was found to on par with more specific behaviorally anchored ratings. Global ratings were also found to be predictive of written tests of clinical knowledge and ratings of actual clinical performance after controlling for behaviorally anchored ratings. Faculty global ratings were more reliable and more predictive of other performance than SP global ratings. These results suggest global ratings by faculty observers and possibly SPs can provide unique and useful information in these performance-based examinations.