Using Self-Regulated Learning Microanalysis to Examine Regulatory Processes in Clerkship Students Engaged in Practice Questions.

Introduction: Self-regulated learning is a cyclical process of forethought, performance, and self-reflection that has been used as an assessment tool in medical education. No prior studies have evaluated SRL processes for answering multiple-choice questions (MCQs) and most evaluated one or two iterations of a non-MCQ task. SRL assessment during MCQs may elucidate reasons why learners are successful or not on these questions that are encountered repeatedly during medical education. Methods: Internal medicine clerkship students at three institutions participated in a SRL microanalytic protocol that targeted strategic planning, metacognitive monitoring, causal attributions, and adaptive inferences across seven MCQs. Responses were transcribed and coded according to previously published methods for microanalytic protocols. Results: Forty-four students participated. In the forethought phase, students commonly endorsed prioritizing relevant features as their diagnostic strategy (n = 20, 45%) but few mentioned higher-order diagnostic reasoning processes such as integrating clinical information (n = 5, 11%) or comparing/contrasting diagnoses (n = 0, 0%). However, in the performance phase, students' metacognitive processes included high frequencies of integration (n = 38, 86%) and comparing/contrasting (n = 24, 55%). In the self-reflection phase, 93% (n = 41) of students faulted their management reasoning and 84% (n = 37) made negative references to their abilities. Less than 10% (n = 4) of students indicated that they would adapt their diagnostic reasoning process for these questions. Discussion: This study describes in detail student self-regulatory processes during MCQs. We found that students engaged in higher-order diagnostic reasoning processes but were not explicit about it and seldom reflected critically on these processes after selecting an incorrect answer. Self-reflections focused almost exclusively on management reasoning and negative references to abilities which may decrease self-efficacy. Encouraging students to identify and evaluate diagnostic reasoning processes and make attributions to controllable factors may improve performance.

Why Does This Learner Perform Poorly on Tests? Using Self-Regulated Learning Theory to Diagnose the Problem and Implement Solutions.

PROBLEM: Learners who underperform on standardized tests are common throughout all levels of medical education and require considerable faculty time and effort to remediate. Current methods for remediating test-taking difficulties are typically not grounded in educational theory or supported by high-quality evidence. APPROACH: A test-taking assessment was developed based on self-regulated learning (SRL) microanalytic assessment and training and used during academic year 2012-2013. This method assesses the SRL subprocesses of strategic planning, self-monitoring, causal attributions, and adaptive inferences during the educational task of answering test questions. The method was designed for one-on-one use by teacher and learner, and for learner self-assessment and practice. OUTCOMES: At one academic institution, this method was used to categorize learners into struggling test-taker subtypes which correspond to deficiencies in the SRL subprocesses outlined above. A learning plan based on the SRL deficiency was developed for each struggling test-taker subtype. In a small number of internal medicine residents with low in-training examination scores, use of this method yielded improvements in 2013 in-training examination score that doubled the expected improvement based on historical averages. NEXT STEPS: This method is a novel application of SRL theory to a commonly encountered problem in medical education: the learner who performs poorly on tests. Large-scale, multicenter studies of medical learners at a variety of training levels and program types are needed to determine the effectiveness and generalizability of this intervention.

Perspective of the Graduating Medical Student: The Ideal Curriculum for the Fourth Year of Undergraduate Medical Education.

OBJECTIVE: To explore medical students' perspective regarding the fourth year of medical school and common educational activities thereof. METHODS: The authors surveyed students graduating in 2012 with a military service obligation about the importance of common fourth-year activities, the proportion of the fourth year devoted to these activities, and important considerations for the fourth-year curriculum. The authors calculated mean importance scores for educational activities and mean proportions of the fourth year that should be devoted to certain activities. Two reviewers independently coded free-text answers to identify and calculate frequencies for common themes. RESULTS: The response rate was 40% (376/942). Participants rated activities related to improving clinical skills and securing the residency of their choice as more than activities such as learning business skills, conducting research, and studying basic sciences. Participants indicated that electives and direct patient care should comprise the majority of the fourth year and frequently mentioned improving specialty-specific clinical skills, pursuing personal medical interests, and taking time to relax as important fourth-year themes. CONCLUSIONS: Students value activities related to securing and succeeding in their chosen residency and the opportunity to pursue electives and take vacation. Faculty should consider the student perspective when reforming curricula.

Characteristics of Highly Rated Internal Medicine Attendings Before and After the 2004 Work-Hour Restrictions.

OBJECTIVES: To describe the characteristics of top-rated Internal Medicine attendings and whether they changed after implementation of the 2004 work-hour restrictions. METHODS: Mixed methods study of resident ratings of medicine attendings (Period 1: 1994-1996, n = 250 and Period 2: 2007-2009, n = 152). Residents evaluated 17 attending characteristics. The top 25% of "overall" ratings were classified as "highly rated." Two free-text questions included "What was your attending's best characteristic?" and "How could your attending best improve?" and were coded in duplicate, using grounded qualitative methods. RESULTS: There were no differences in the characteristics of highly rated attendings in the two time periods. Characteristics associated with being a top-rated attending included enthusiasm (odds ratio [OR]: 5.69, 2.78-11.67), balanced teaching style (OR: 3.63, 1.64-8.02), promoting independent thinking (OR: 2.90, 0.96-8.74), fund of knowledge (OR: 2.73, 1.13-6.58), and time management (OR: 1.78, 1.14-2.80). Among the 1,410 utterances, valued attending attributes included helpfulness, promoting independent thinking, and having strong medical knowledge. CONCLUSIONS: The characteristics valued by residents in attendings did not change over time despite a major structural change in work hours and patterns of teaching. These valued characteristics continue to be a strong general fund of knowledge, enthusiasm for teaching, and balance between didactic and bedside approaches.

A Case of Paraspinal Muscle Rhabdomyolysis in a 22-Year-Old Male After Ingesting a Supplement Containing Higenamine.

Rhabdomyolysis is a painful and potentially life-threatening injury in which muscle breaks down in response to an insult. In this case report, we describe the clinical course of a 22-year-old man who developed paraspinal muscle rhabdomyolysis and possible compartment syndrome in association with strenuous activity and the ingestion of an exercise supplement containing the stimulant higenamine. Although he did not experience any renal damage, he did experience a high level of pain acutely and for the next 4 months. A history of exercise supplement intake should increase the index of suspicion for rhabdomyolysis when a patient presents with moderate to severe muscle pain, even if the muscle group is not in a classic location for rhabdomyolysis.

Predicting medical school and internship success: does the quality of the research and clinical experience matter?

OBJECTIVES: This article explores specific aspects of self-reported clinical and research experience and their relationship to performance in medical training. METHODS: This is a retrospective cohort study conducted at the Uniformed Services University. The American Medical College Application Service application was used to discern students' self-reported clinical and research experience. Two authors applied a classification scheme for clinical and research experience to the self-reported experiences. Study outcomes included medical school grade point average (GPA), U.S. Medical Licensing Examination (USMLE) scores, and intern expertise and professionalism scores. A linear regression analysis was conducted for each outcome while controlling for prematriculation GPA. RESULTS: Data were retrieved on 1,020 matriculants. There were several statistically significant but small differences across outcomes when comparing the various categories of clinical experience with no clinical experience. The technician-level experience group had a decrease of 0.1 in cumulative GPA in comparison to students without self-reported clinical experience (p = 0.004). This group also performed 5 points lower on the USMLE Step 2 than students who did not report clinical experience (p = 0.013). The various levels of self-reported research experience were unrelated to success in medical school and graduate medical education. DISCUSSION: These findings indicate that self-reported technician-level clinical experience is related to a small reduction in typically reported outcomes in medical school.

Assessing curriculum effectiveness: a survey of Uniformed Services University medical school graduates.

PURPOSE: This study assessed alumni perceptions of their preparedness for clinical practice using the Accreditation Council for Graduate Medical Education (ACGME) competencies. We hypothesized that our alumni's perception of preparedness would be highest for military-unique practice and professionalism and lowest for system-based practice and practice-based learning and improvement. METHOD: 1,189 alumni who graduated from the Uniformed Services University (USU) between 1980 and 2001 completed a survey modeled to assess the ACGME competencies on a 5-point, Likert-type scale. Specifically, self-reports of competencies related to patient care, communication and interpersonal skills, medical knowledge, professionalism, systems-based practice, practice-based learning and improvement, and military-unique practice were evaluated. RESULTS: Consistent with our expectations as the nation's military medical school, our graduates were most confident in their preparedness for military-unique practice, which included items assessing military leadership (M = 4.30, SD = 0.65). USU graduates also indicated being well prepared for the challenges of residency education in the domain of professionalism (M = 4.02, SD = 0.72). Self-reports were also high for competencies related to patient care (M = 3.86, SD = 0.68), communication and interpersonal skills (M = 3.88, SD = 0.66), and medical knowledge (M = 3.78, SD = 0.73). Consistent with expectations, systems-based practice (M = 3.50, SD = 0.70) and practice-based learning and improvement (M = 3.57, SD = 0.62) were the lowest rated competencies, although self-reported preparedness was still quite high. DISCUSSION: Our findings suggest that, from the perspective of our graduates, USU is providing both an effective military-unique curriculum and is preparing trainees for residency training. Further, these results support the notion that graduates are prepared to lead and to practice medicine in austere environments. Compared to other competencies that were assessed, self-ratings for systems-based practice and practice-based learning and improvement were the lowest, which suggests the need to continue to improve USU education in these areas.

The long-term career outcome study: lessons learned and implications for educational practice.

The work of the Long-Term Career Outcome Study has been a program of scholarship spanning 10 years. Borrowing from established quality assurance literature, the Long-Term Career Outcome Study team has organized its scholarship into three phases; before medical school, during medical school, and after medical school. The purpose of this commentary is to address two fundamental questions: (1) what has been learned? and (2) how does this knowledge translate to educational practice and policy now and into the future? We believe that answers to these questions are relevant not only to our institution but also to other educational institutions seeking to provide high-quality health professions education.

Special Forces Medical Sergeants' perceptions and beliefs regarding their current medical sustainment program: implications for the field.

BACKGROUND: Special Forces Medical Sergeants (SFMS) are trained to provide trauma and medical care in support of military operations and diplomatic missions throughout the world with indirect physician oversight. This study assessed their perceptions of the current program designed to sustain their medical skills. METHODS: An Internet-based survey was developed using the constructs of the Theory of Reasoned Action/Planned Behavior and validated through survey best practices. RESULTS: Of the 334 respondents, 92.8% had deployed at least once as an SFMS. Respondents reported spending 4 hours per week sustaining their medical skills and were highly confident that they could perform their duties on a no-notice deployment. On a 5-point, Likert-type response scale, SFMS felt that only slight change is needed to the Special Operations Medical Skills Sustainment Course (mean: 2.17; standard deviation [SD]: 1.05), while moderate change is needed to the Medical Proficiency Training (mean: 2.82; SD: 1.21) and nontrauma modules (mean: 3.02; SD: 1.22). Respondents desire a medical sustainment program that is provided by subject matter experts, involves actual patient care, incorporates new technology, uses hands-on simulation, and is always available. CONCLUSIONS: SFMS are challenged to sustain their medical skills in the current operational environment, and barriers to medical training should be minimized to facilitate sustainment training. Changes to the current medical sustainment program should incorporate operator-level perspectives to ensure acceptability and utility but must be balanced with organizational realities. Improving the medical sustainment program will prepare SFMS for the challenges of future missions.

Validity evidence for medical school OSCEs: associations with USMLE® step assessments.

BACKGROUND: Recently, there has been a surge in the use of objective structured clinical examinations (OSCEs) at medical schools around the world, and with this growth has come the concomitant need to validate such assessments. PURPOSES: The current study examined the associations between student performance on several school-level clinical skills and knowledge assessments, including two OSCEs, the National Board of Medical Examiners® (NBME) Subject Examinations, and the United States Medical Licensing Examination® (USMLE) Step 2 Clinical Skills (CS) and Step 3 assessments. METHODS: The sample consisted of 806 medical students from the Uniformed Services University of the Health Sciences. We conducted Pearson correlation analysis as well as stepwise multiple linear regression modeling to examine the strength of associations between students' performance on 2nd- and 3rd-year OSCEs and their two Step 2 CS component scores and Step 3 scores. RESULTS: Positive associations were found between the OSCE variables and the USMLE scores; in particular, student performance on both the 2nd- and 3rd-year OSCEs was more strongly associated with the two Step 2 CS component scores than with Step 3 scores. CONCLUSIONS: These findings, although preliminary, provide some predictive validity evidence for the use of OSCEs in determining readiness of medical students for clinical practice and licensure.

Lifestyle factors and primary care specialty selection: comparing 2012-2013 graduating and matriculating medical students' thoughts on specialty lifestyle.

PURPOSE: To compare how first-year (MS1) and fourth-year students (MS4) ascribe importance to lifestyle domains and specialty characteristics in specialty selection, and compare students' ratings with their primary care (PC) interest. METHOD: In March 2013, MS4s from 11 U.S. MD-granting medical schools were surveyed. Using a five-point Likert-type scale (1 = not important at all; 5 = extremely important), respondents rated the importance of 5 lifestyle domains and 21 specialty selection characteristics. One-way analysis of variance was used to assess differences by PC interest among MS4s. Using logistic regression, ratings from MS4s were compared with prior analyses of ratings by MS1s who matriculated to the same 11 schools in 2012. RESULTS: The response rate was 57% (965/1,701). MS4s, as compared with MS1s, rated as more important to good lifestyle: time off (4.3 versus 4.0), schedule control (4.2 versus 3.9), and financial compensation (3.4 versus 3.2). More MS4s than MS1s selected "time-off" (262/906 [30%] versus 136/969 [14%]) and "control of work schedule" (169/906 [19%] versus 146/969 [15%]) as the most important lifestyle domains. In both classes, PC interest was associated with higher ratings of working with the underserved and lower ratings of prestige and salary. CONCLUSIONS: In the 2012-2013 academic year, matriculating students and graduating students had similar perceptions of lifestyle and specialty characteristics associated with PC interest. Graduating students placed more importance on schedule control and time off than matriculating students. Specialties should consider addressing a perceived lack of schedule control or inadequate time off to attract students.

What aspects of letters of recommendation predict performance in medical school? Findings from one institution.

PURPOSE: To study medical students' letters of recommendation (LORs) from their applications to medical school to determine whether these predicted medical school performance, because many researchers have questioned LORs' predictive validity. METHOD: A retrospective cohort study of three consecutive graduating classes (2007-2009) at the Uniformed Services University of the Health Sciences was performed. In each class, the 27 students who had been elected into the Alpha Omega Alpha (AOA) Honor Medical Society were defined as top graduates, and the 27 students with the lowest cumulative grade point average (GPA) were designated as "bottom of the class" graduates. For each student, the first three LORs (if available) in the application packet were independently coded by two blinded investigators using a comprehensive list of 76 characteristics. Each characteristic was compared with graduation status (top or bottom of the class), and those with statistical significance related to graduation status were inserted into a logistic regression model, with undergraduate GPA and Medical College Admission Test score included as control variables. RESULTS: Four hundred thirty-seven LORs were included. Of 76 LOR characteristics, 7 were associated with graduation status (P ≤ .05), and 3 remained significant in the regression model. Being rated as "the best" among peers and having an employer or supervisor as the LOR author were associated with induction into AOA, whereas having nonpositive comments was associated with bottom of the class students. CONCLUSIONS: LORs have limited value to admission committees, as very few LOR characteristics predict how students perform during medical school.

Is poor performance on NBME clinical subject examinations associated with a failing score on the USMLE step 3 examination?

PURPOSE: To investigate the association between poor performance on National Board of Medical Examiners clinical subject examinations across six core clerkships and performance on the United States Medical Licensing Examination Step 3 examination. METHOD: In 2012, the authors studied matriculants from the Uniformed Services University of the Health Sciences with available Step 3 scores and subject exam scores on all six clerkships (Classes of 2007-2011, N = 654). Poor performance on subject exams was defined as scoring one standard deviation (SD) or more below the mean using the national norms of the corresponding test year. The association between poor performance on the subject exams and the probability of passing or failing Step 3 was tested using contingency table analyses and logistic regression modeling. RESULTS: Students performing poorly on one subject exam were significantly more likely to fail Step 3 (OR 14.23 [95% CI 1.7-119.3]) compared with students with no subject exam scores that were 1 SD below the mean. Poor performance on more than one subject exam further increased the chances of failing (OR 33.41 [95% CI 4.4-254.2]). This latter group represented 27% of the entire cohort, yet contained 70% of the students who failed Step 3. CONCLUSIONS: These findings suggest that individual schools could benefit from a review of subject exam performance to develop and validate their own criteria for identifying students at risk for failing Step 3.

Developing questionnaires for educational research: AMEE Guide No. 87.

In this AMEE Guide, we consider the design and development of self-administered surveys, commonly called questionnaires. Questionnaires are widely employed in medical education research. Unfortunately, the processes used to develop such questionnaires vary in quality and lack consistent, rigorous standards. Consequently, the quality of the questionnaires used in medical education research is highly variable. To address this problem, this AMEE Guide presents a systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales. These seven steps do not address all aspects of survey design, nor do they represent the only way to develop a high-quality questionnaire. Instead, these steps synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels. Addressing each of these steps systematically will improve the probabilities that survey designers will accurately measure what they intend to measure.

Reliability and validity of clinician ECG interpretation for athletes.

BACKGROUND: Electrocardiogram (ECG) with preparticipation evaluation (PPE) for athletes remains controversial in the United States and diagnostic accuracy of clinician ECG interpretation is unclear. This study aimed to assess reliability and validity of clinician ECG interpretation using expert-validated ECGs according to the 2010 European Society of Cardiology (ESC) interpretation criteria. METHODS: This is a blinded, prospective study of diagnostic accuracy of clinician ECG interpretation. Anonymized ECGs were validated for normal and abnormal patterns by blinded expert interpreters according to the ESC interpretation criteria from October 2011 through March 2012. Six pairs of clinician interpreters were recruited from relevant clinical specialties in an academic medical center in March 2012. Each clinician interpreted 85 ECGs according to the ESC interpretation guidelines. Cohen and Fleiss' kappa, sensitivity, and specificity were calculated within specialties and across primary care and cardiology specialty groups. RESULTS: Experts interpreted 189 ECGs yielding a kappa of 0.63, demonstrating "substantial" inter-rater agreement. A total of 85 validated ECGs, including 26 abnormals, were selected for clinician interpretation. The kappa across cardiology specialists was "substantial" and "moderate" across primary care (0.69 vs 0.52, respectively, P < 0.001). Sensitivity and specificity to detect abnormal patterns were similar between cardiology and primary care groups (sensitivity 93.3% vs 81.3%, respectively, P = 0.31; specificity 88.8% vs 89.8%, respectively, P = 0.91). CONCLUSIONS: Clinician ECG interpretation according to the ESC interpretation criteria appears to demonstrate limited reliability and validity. Before widespread adoption of ECG for PPE of U.S. athletes, further research of training focused on improved reliability and validity of clinician ECG interpretation is warranted.

Primary care, the ROAD less traveled: what first-year medical students want in a specialty.

PURPOSE: Medical students are increasingly choosing non-primary-care specialties. Students consider lifestyle in selecting their specialty, but how entering medical students perceive lifestyle is unknown. This study investigates how first-year students value or rate lifestyle domains and specialty-selection characteristics and whether their ratings vary by interest in primary care (PC). METHOD: During the 2012-2013 academic year, the authors conducted a cross-sectional survey of first-year medical students from 11 MD-granting medical schools. Using a five-point Likert-type scale (1 = not important at all; 5 = extremely important), respondents rated the importance of 5 domains of good lifestyle and 21 characteristics related to specialty selection. The authors classified students into five groups by PC interest and assessed differences by PC interest using one-way analysis of variance. RESULTS: Of 1,704 participants, 1,020 responded (60%). The option "type of work I am doing" was the highest-rated lifestyle domain (mean 4.8, standard deviation [SD] 0.6). "Being satisfied with the job" was the highest-rated specialty-selection characteristic (mean 4.7, SD 0.5). "Availability of practice locations in rural areas" was rated lowest (mean 2.0, SD 1.1). As PC interest decreased, the importance of "opportunities to work with underserved populations" also decreased, but importance of "average salary earned" increased (effect sizes of 0.98 and 0.94, respectively). CONCLUSIONS: First-year students valued enjoying work. The importance of financial compensation was inversely associated with interest in PC. Examining the determinants of enjoyable work may inform interventions to help students attain professional fulfillment in PC.

Internet versus in-person counseling for patients taking varenicline for smoking cessation.

INTRODUCTION: Quit rates for smoking cessation attempts are maximized by using counseling with medication. Internet-based counseling might be a suitable replacement for in-person counseling. METHODS: Patients in a military medical system in the active phase of quitting presented for study intake. They were randomized to in-person counseling (n = 44) or Internet counseling (n = 173). In-person counseling consisted of four 1.5 hour classes based on the American Cancer Society's Freshstart program. Internet counseling consisted of daily e-mails with recommended activities through Pfizer's GetQuit program. Both groups were concomitantly treated with standard dose varenicline. The primary outcome was the quit rate at 12 weeks, defined as abstinence and an exhaled carbon monoxide level <10 ppm at the 12-week visit. All those lost to follow-up were considered persistent smokers. RESULTS: 217 smokers were randomized, of which 43% returned for the 12-week follow-up visit. Quit rates between the two groups were similar (Internet group: 21%, n = 36/173; in-person group: 18%, n = 8/44, p = 0.7). CONCLUSIONS: Internet-based counseling might be equivalent to in-person counseling for smoking cessation in patients taking varenicline. Additional studies with more complete and longer-term (≥1 year) follow-up are needed to confirm these findings.

The R.O.A.D. confirmed: ratings of specialties' lifestyles by fourth-year US medical students with a military service obligation.

BACKGROUND AND OBJECTIVES: Lifestyle factors influence medical specialty choice, but which specialties are perceived to have the best lifestyles is poorly described in scientific literature. The objective of the study was to determine the rating of specialties by lifestyle. METHODS: All fourth-year US medical students with a Department of Defense service obligation who participated in the 2008/2009 military Match were invited to participate in a survey following the Match. The survey listed 18 specialties and asked students to rate the lifestyle of each one on a 9-point scale, and the mean score was used as the rating. Students also listed their specialty choice in the Match. RESULTS: The response rate was 52%, as 418 of the 797 eligible students responded and provided a rating for at least nine of the 18 specialties. The four specialties rated highest for lifestyle (1--9, with 9 being highest) were dermatology (8.4), radiology (8.1), ophthalmology (8.0), and anesthesia (7.5). The four specialties rated lowest were orthopedics (4.0), neurosurgery (3.1), general surgery (2.6), and obstetrics-gynecology (2.5). Family medicine (5.7) was the top-rated primary care specialty, followed by pediatrics (5.3) and internal medicine (4.7). Students rated the lifestyle of their own specialty only slightly higher (range 0.02 to 1.8) than all other students. CONCLUSIONS: The R.O.A.D. specialties (radiology, ophthalmology, anesthesia, and dermatology) are the top specialties with respect to lifestyle as viewed by current students. Students perceive their own specialty's lifestyle realistically. Research determining why a specialty perceived as having a lower-rated lifestyle is acceptable to some students and not others is needed.