Crowdsourcing for assessment items to support adaptive learning.

PURPOSE: Adaptive learning requires frequent and valid assessments for learners to track progress against their goals. This study determined if multiple-choice questions (MCQs) "crowdsourced" from medical learners could meet the standards of many large-scale testing programs. METHODS: Users of a medical education app (Osmosis.org, Baltimore, MD) volunteered to submit case-based MCQs. Eleven volunteers were selected to submit MCQs targeted to second year medical students. Two hundred MCQs were subjected to duplicate review by a panel of internal medicine faculty who rated each item for relevance, content accuracy, and quality of response option explanations. A sample of 121 items was pretested on clinical subject exams completed by a national sample of U.S. medical students. RESULTS: Seventy-eight percent of the 200 MCQs met faculty reviewer standards based on relevance, accuracy, and quality of explanations. Of the 121 pretested MCQs, 50% met acceptable statistical criteria. The most common reasons for exclusion were that the item was too easy or had a low discrimination index. CONCLUSIONS: Crowdsourcing can efficiently yield high-quality assessment items that meet rigorous judgmental and statistical criteria. Similar models may be adopted by students and educators to augment item pools that support adaptive learning.

The Pediatrics Milestones Assessment Pilot: Development of Workplace-Based Assessment Content, Instruments, and Processes.

PURPOSE: To report on the development of content and user feedback regarding the assessment process and utility of the workplace-based assessment instruments of the Pediatrics Milestones Assessment Pilot (PMAP). METHOD: One multisource feedback instrument and two structured clinical observation instruments were developed and refined by experts in pediatrics and assessment to provide evidence for nine competencies based on the Pediatrics Milestones (PMs) and chosen to inform residency program faculty decisions about learners' readiness to serve as pediatric interns in the inpatient setting. During the 2012-2013 PMAP study, 18 U.S. pediatric residency programs enrolled interns and subinterns. Faculty, residents, nurses, and other observers used the instruments to assess learner performance through direct observation during a one-month rotation. At the end of the rotation, data were aggregated for each learner, milestone levels were assigned using a milestone classification form, and feedback was provided to learners. Learners and site leads were surveyed and/or interviewed about their experience as participants. RESULTS: Across the sites, 2,338 instruments assessing 239 learners were completed by 630 unique observers. Regarding end-of-rotation feedback, 93% of learners (128/137) agreed the assessments and feedback "helped me understand how those with whom I work perceive my performance," and 85% (117/137) agreed they were "useful for constructing future goals or identifying a developmental path." Site leads identified several benefits and challenges to the assessment process. CONCLUSIONS: PM-based instruments used in workplace-based assessment provide a meaningful and acceptable approach to collecting evidence of learner competency development. Learners valued feedback provided by PM-based assessment.

An Investigation of the Properties of Computer-Based Case Simulations.

Computer-based case simulations (CCS) require several development iterations, each involving a sizeable investment of time from physician volunteers and test development staff. Hence, a case that fails to demonstrate good measurement properties when administered to examinees at the pretest stage represents a costly loss. An earlier study investigated the relationship of case properties to case difficulty in part to obtain information that would permit (a) better instructions to case developers and (b) identification of problematic cases earlier in the developmental cycle. Some of the best predictors, however, were scoring points typically obtained at the pretest stage, making them inappropriate for meeting these two goals. The objective of this study was to determine if these scoring points could be predicted from cases properties that would be available early in case development. Case description variables and scoring points were available for 28 cases which were analyzed using regression procedures. Three models were identified that predicted the important scoring fairly well. These models differed primarily in the area of medicine: internal medicine, obstetrics/gynecology, or no medical area specified. The results were consistent with those found for predicting difficulty and appeared promising for better understanding of the nature of the simulations.

Exploring Alternative Models of Complex Patient Management with Artificial Neural Networks.

This study applied an unsupervised neural network modeling process to test data of the National Board of Medical Examiners (NBME) Computer-based Clinical Scenarios (CCS) to identify new performance categories and validate this process as a scoring technique. The classifications resulting from this neural network modeling were consistent with the NBME model in that highly rated NMBE performances (ratings of 7 or 8) were clustered together on the neural network output grid. Very low performance ratings appeared to share few common features and were accordingly classified at isolated nodes. This clustering was reproducible across three separately trained networks with greater than 80% agreement in two of the three networks trained. However, the neural network also contained performance clusters where disparate NBME-based ratings ranged from 1 (worst) to 8 (best). Here, agreement between networks was less than 60%. Through visualization of the search strategies (search path mapping), this neural network clustering was found to be sensitive to quantitative and qualitative test selections such as excessive usage of irrelevant tests reflecting broader behavioral classification in some instances. A disparity between NBME ratings and an independent human rating system was detected by the neural network model since disagreement among raters was also reflected by a lack of neural network performance clustering. Agreement between rating systems, however, was correlated with neural network clustering for 92% of the highly rated performances.

Which experiences in the hidden curriculum teach students about professionalism?

PURPOSE: To examine the relationship between learner experience in the "hidden curriculum" and student attribution of such experiences to professionalism categories. METHOD: Using the output of a thematic analysis of 272 consecutive narratives recorded by 135 students on a medical clerkship from June through November 2007, the authors describe the frequency of these experiences within and across student-designated Association of American Medical Colleges-National Board of Medical Examiners professionalism categories and employ logistic regression to link varieties of experience to specific professionalism categories. RESULTS: Thematic analysis uncovered two main domains of student experience: medical-clinical interaction and teaching-and-learning experiences. From a student perspective the critical incident stories evoked all professionalism categories. Most frequently checked off categories were caring/compassion/communication (77%) and respect (69%). Logistic regression suggested that student experiences within the teaching-and-learning environment were associated with professionalism categories of excellence, leadership, and knowledge and skills, whereas those involving medical-clinical interactions were associated with respect, responsibility and accountability, altruism, and honor and integrity. Experiences of communicating and working within teams had the broadest association with learning about professionalism. CONCLUSIONS: Student narratives touched on all major professionalism categories as well as illuminating the contexts in which critical experiences emerged. Linked qualitative and quantitative analysis identified those experiences that were associated with learning about particular aspects of professionalism. Experiences of teamwork were especially relevant to student learning about professionalism in action.

Medical students' professionalism narratives: a window on the informal and hidden curriculum.

PURPOSE: The aim of this study was to use medical students' critical incident narratives to deepen understanding of the informal and hidden curricula. METHOD: The authors conducted a thematic analysis of 272 stories of events recorded by 135 third-year medical students that "taught them something about professionalism and professional values." Students wrote these narratives in a "professionalism journal" during their internal medicine clerkships at Indiana University School of Medicine, June through November 2007. RESULTS: The majority of students' recorded experiences involved witnessing positive embodiment of professional values, rather than breaches. Attending physicians and residents were the central figures in the incidents. Analyses revealed two main thematic categories. The first focused on medical-clinical interactions, especially on persons who were role models interacting with patients, families, coworkers, and colleagues. The second focused on events in the teaching-and-learning environment, particularly on students' experiences as learners in the clinical setting. CONCLUSIONS: The findings strongly suggest that students' reflective narratives are a rich source of information about the elements of both the informal and hidden curricula, in which medical students learn to become physicians. Experiences with both positive and negative behaviors shaped the students' perceptions of the profession and its values. In particular, interactions that manifest respect and other qualities of good communication with patients, families, and colleagues taught powerfully.

Advancing resident assessment in graduate medical education.

BACKGROUND: The Outcome Project requires high-quality assessment approaches to provide reliable and valid judgments of the attainment of competencies deemed important for physician practice. INTERVENTION: The Accreditation Council for Graduate Medical Education (ACGME) convened the Advisory Committee on Educational Outcome Assessment in 2007-2008 to identify high-quality assessment methods. The assessments selected by this body would form a core set that could be used by all programs in a specialty to assess resident performance and enable initial steps toward establishing national specialty databases of program performance. The committee identified a small set of methods for provisional use and further evaluation. It also developed frameworks and processes to support the ongoing evaluation of methods and the longer-term enhancement of assessment in graduate medical education. OUTCOME: The committee constructed a set of standards, a methodology for applying the standards, and grading rules for their review of assessment method quality. It developed a simple report card for displaying grades on each standard and an overall grade for each method reviewed. It also described an assessment system of factors that influence assessment quality. The committee proposed a coordinated, national-level infrastructure to support enhancements to assessment, including method development and assessor training. It recommended the establishment of a new assessment review group to continue its work of evaluating assessment methods. The committee delivered a report summarizing its activities and 5 related recommendations for implementation to the ACGME Board in September 2008.