The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156.

The use of Artificial Intelligence (AI) in medical education has the potential to facilitate complicated tasks and improve efficiency. For example, AI could help automate assessment of written responses, or provide feedback on medical image interpretations with excellent reliability. While applications of AI in learning, instruction, and assessment are growing, further exploration is still required. There exist few conceptual or methodological guides for medical educators wishing to evaluate or engage in AI research. In this guide, we aim to: 1) describe practical considerations involved in reading and conducting studies in medical education using AI, 2) define basic terminology and 3) identify which medical education problems and data are ideally-suited for using AI.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement.

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

Educating for adaptive expertise: case examples along the medical education continuum.

Adaptive expertise represents the combination of both efficient problem-solving for clinical encounters with known solutions, as well as the ability to learn and innovate when faced with a novel challenge. Fostering adaptive expertise requires careful approaches to instructional design to emphasize deeper, more effortful learning. These teaching strategies are time-intensive, effortful, and challenging to implement in health professions education curricula. The authors are educators whose missions encompass the medical education continuum, from undergraduate through to organizational learning. Each has grappled with how to promote adaptive expertise development in their context. They describe themes drawn from educational experiences at these various learner levels to illustrate strategies that may be used to cultivate adaptive expertise.At Vanderbilt University School of Medicine, a restructuring of the medical school curriculum provided multiple opportunities to use specific curricular strategies to foster adaptive expertise development. The advantage for students in terms of future learning had to be rationalized against assessments that are more short-term in nature. In a consortium of emergency medicine residency programs, a diversity of instructional approaches was deployed to foster adaptive expertise within complex clinical learning environments. Here the value of adaptive expertise approaches must be balanced with the efficiency imperative in clinical care. At Mayo Clinic, an existing continuous professional development program was used to orient the entire organization towards an adaptive expertise mindset, with each individual making a contribution to the shift.The different contexts illustrate both the flexibility of the adaptive expertise conceptualization and the need to customize the educational approach to the developmental stage of the learner. In particular, an important benefit of teaching to adaptive expertise is the opportunity to influence individual professional identity formation to ensure that clinicians of the future value deeper, more effortful learning strategies throughout their careers.

A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation.

CONSTRUCT: For assessing the skill of visual diagnosis such as radiograph interpretation, competency standards are often developed in an ad hoc method, with a poorly delineated connection to the target clinical population. BACKGROUND: Commonly used methods to assess for competency in radiograph interpretation are subjective and potentially biased due to a small sample size of cases, subjective evaluations, or include an expert-generated case-mix versus a representative sample from the clinical field. Further, while digital platforms are available to assess radiograph interpretation skill against an objective standard, they have not adopted a data-driven competency standard which informs educators and the public that a physician has achieved adequate mastery to enter practice where they will be making high-stakes clinical decisions. APPROACH: Operating on a purposeful sample of radiographs drawn from the clinical domain, we adapted the Ebel Method, an established standard setting method, to ascertain a defensible, clinically relevant mastery learning competency standard for the skill of radiograph interpretation as a model for deriving competency thresholds in visual diagnosis. Using a previously established digital platform, emergency physicians interpreted pediatric musculoskeletal extremity radiographs. Using one-parameter item response theory, these data were used to categorize radiographs by interpretation difficulty terciles (i.e. easy, intermediate, hard). A panel of emergency physicians, orthopedic surgeons, and plastic surgeons rated each radiograph with respect to clinical significance (low, medium, high). These data were then used to create a three-by-three matrix where radiographic diagnoses were categorized by interpretation difficulty and significance. Subsequently, a multidisciplinary panel that included medical and parent stakeholders determined acceptable accuracy for each of the nine cells. An overall competency standard was derived from the weighted sum. Finally, to examine consequences of implementing this standard, we reported on the types of diagnostic errors that may occur by adhering to the derived competency standard. FINDINGS: To determine radiograph interpretation difficulty scores, 244 emergency physicians interpreted 1,835 pediatric musculoskeletal extremity radiographs. Analyses of these data demonstrated that the median interpretation difficulty rating of the radiographs was -1.8 logits (IQR -4.1, 3.2), with a significant difference of difficulty across body regions (p < 0.0001). Physician review classified the radiographs as 1,055 (57.8%) as low, 424 (23.1%) medium or 356 (19.1%) high clinical significance. The multidisciplinary panel suggested a range of acceptable scores between cells in the three-by-three table of 76% to 95% and the sum of equal-weighted scores resulted in an overall performance-based competency score of 85.5% accuracy. Of the 14.5% diagnostic interpretation errors that may occur at the bedside if this competency standard were implemented, 9.8% would be in radiographs of low-clinical significance, while 2.5% and 2.3% would be in radiographs of medium or high clinical significance, respectively. CONCLUSION(S): This study's novel integration of radiograph selection and a standard setting method could be used to empirically drive evidence-based competency standard for radiograph interpretation and can serve as a model for deriving competency thresholds for clinical tasks emphasizing visual diagnosis.

Multi-level longitudinal learning curve regression models integrated with item difficulty metrics for deliberate practice of visual diagnosis: groundwork for adaptive learning.

Visual diagnosis of radiographs, histology and electrocardiograms lends itself to deliberate practice, facilitated by large online banks of cases. Which cases to supply to which learners in which order is still to be worked out, with there being considerable potential for adapting the learning. Advances in statistical modeling, based on an accumulating learning curve, offer methods for more effectively pairing learners with cases of known calibrations. Using demonstration radiograph and electrocardiogram datasets, the advantages of moving from traditional regression to multilevel methods for modeling growth in ability or performance are demonstrated, with a final step of integrating case-level item-response information based on diagnostic grouping. This produces more precise individual-level estimates that can eventually support learner adaptive case selection. The progressive increase in model sophistication is not simply statistical but rather brings the models into alignment with core learning principles including the importance of taking into account individual differences in baseline skill and learning rate as well as the differential interaction with cases of varying diagnosis and difficulty. The developed approach can thus give researchers and educators a better basis on which to anticipate learners' pathways and individually adapt their future learning.

A think-aloud study to inform the design of radiograph interpretation practice.

Models for diagnostic reasoning in radiology have been based on the observed behaviors of experienced radiologists but have not directly focused on the thought processes of novices as they improve their accuracy of image interpretation. By collecting think-aloud verbal reports, the current study was designed to investigate differences in specific thought processes between medical students (novices) as they learn and radiologists (experts), so that we can better design future instructional environments. Seven medical students and four physicians with radiology training were asked to interpret and diagnose pediatric elbow radiographs where fracture is suspected. After reporting their diagnosis of a case, they were given immediate feedback. Participants were asked to verbalize their thoughts while completing the diagnosis and while they reflected on the provided feedback. The protocol analysis of their verbalizations showed that participants used some combination of four processes to interpret the case: gestalt interpretation, purposeful search, rule application, and reasoning from a prior case. All types of processes except reasoning from a prior case were applied significantly more frequently by experts. Further, gestalt interpretation was used with higher frequency in abnormal cases while purposeful search was used more often for normal cases. Our assessment of processes could help guide the design of instructional environments with well-curated image banks and analytics to facilitate the novice's journey to expertise in image interpretation.

Click-level Learning Analytics in an Online Medical Education Learning Platform.

THEORY: Learning in digital environments allows the collection of inexpensive, fine-grained process data across a large population of learners. Intentional design of the data collection can enable iterative testing of an instructional design. In this study, we propose that across a population of learners the information from multiple choice question responses can help to identify which design features are associated with positive learner engagement. Hypothesis: We hypothesized that, within an online module that presents serial knowledge content, measures of click-level behavior will show sufficient, but variable, association with a test-measure so as to potentially guide instructional design. Method: The Aquifer online learning platform employs interactive approaches to enable effective learning of health professions content. A multidisciplinary focus group of experts identified potential learning analytic measures within an Aquifer learning module, including: hyperlinks clicked (yes/no), magnify buttons clicked (yes/no), expert advice links clicked (yes/no), and time spent on each page (seconds). Learning analytics approaches revealed which click-level data was correlated with the subsequent relevant Case MCQ. We report regression coefficients where the dependent variable is student accuracy on the Case MCQ as a general indicator of successful engagement. Results: Clicking hyperlinks, magnifying images, clicking "expert" links, and spending >100 seconds on each page were learning analytic measures and were positively correlated with Case MCQ success; rushing through pages (<20 seconds) was inversely correlated with success. Conversely, for some measures, we failed to find expected associations. Conclusions: In online learning environments, the wealth of process data available offers insights for instructional designers to iteratively hone the effectiveness of learning. Learning analytic measures of engagement can provide feedback as to which interaction elements are effective.

Speed and quality goals in procedural skills learning: A randomized experiment.

Purpose: Compare time (speed) and product quality goals in a surgical procedural task.Methods: Secondary school students participating in a medical simulation-based training activity participated in a randomized experiment. Each participant completed eight repetitions of a blood vessel ligation. Once, between repetitions four and five, each participant received a randomly-assigned speed goal or quality goal. Outcomes included time and leak-free ligatures.Results: 80 students participated. The speed-goal group performed 18% faster on the final repetition than the quality-goal group, with adjusted fold change (FC) 0.82 (95% confidence interval [CI], 0.71, 0.94; p = 0.01). Conversely, the speed-goal group had fewer high-quality (leak-free) ligatures (odds ratio [OR] 0.36 [95% CI, 0.22, 0.58; p < 0.001]). For the quality-goal group, leaky ligatures took longer post-intervention than leak-free ligatures (FC 1.09 [95% CI, 1.02, 1.17; p = 0.01]), whereas average times for leaky and leak-free ligatures were similar for the speed-goal group (FC 0.97 [95% CI, 0.91, 1.04; p = 0.38]). For a given performance time, the speed-goal group had more leaks post-intervention than the quality-goal group (OR 3.35 [95% CI, 1.58, 7.10; p = 0.002]).Conclusions: Speed and quality goals promote different learning processes and outcomes among novices. Use of both speed and quality goals may facilitate more effective and efficient learning.

Pedagogical validity: The key to understanding different forms of 'good' teaching.

The interrelationship of pedagogical skills, educational ends, and underlying values and assumptions constitute a teacher's 'pedagogical validity' - who they are as a teacher and why they teach the way they do. If reflection, judgment, and improvement are to be helpful, they must have regard for a more complete understanding of what frames a teacher's pedagogical validity. This article briefly describes four kinds of pedagogical validity that teachers draw upon when explaining or justifying their notion of 'good' teaching. Teachers generally have some part of each, but most of us draw upon one or two more than all four as we define ourselves as teachers and make sense of our teaching.

Neonatal resuscitation experience curves: simulation based mastery learning booster sessions and skill decay patterns among pediatric residents.

Background Following neonatal resuscitation program (NRP) training, decay in clinical skills can occur. Simulation-based deliberate practice (SBDP) has been shown to maintain NRP skills to a variable extent. Our study objectives were (a) to determine whether a single 30 min simulation-based intervention that incorporates SBDP and mastery learning (ML) can effectively restore skills and prevent skill decay and (b) to compare different timing options. Methods Following NRP certification, pediatric residents were randomly assigned to receive a video-recorded baseline assessment plus SBDP-ML refresher education at between 6 and 9 months (early) or between 9 and 12 months (late). One year following initial certification, participants had repeat skill retention videotaped evaluations. Participants were scored by blinded NRP instructors using validated criteria scoring tools and assigned a global performance rating score (GRS). Results Twenty-seven participants were included. Residents in both early and late groups showed significant skill decay 7 and 10 months after initial NRP. SBDP-ML booster sessions significantly improved participants' immediate NRP performance scores (p<0.001), which persisted for 2 months, but were again lower 4 months later. Conclusions NRP skills may be boosted to mastery levels after a short SBDP-ML intervention and do not appear to significantly decline after 2 months. Brief booster training could potentially serve as a useful supplement to traditional NRP training for pediatric residents.

Learning to balance efficiency and innovation for optimal adaptive expertise.

It is critical for health professionals to continue to learn and this must be supported by health professions education (HPE). Adaptive expert clinicians are not only expert in their work but have the additional capacity to learn and improve in their practices. The authors review a selective aspect of learning to become an adaptive expert: the capacity to optimally balance routine approaches that maximize efficiency with innovative ones where energy and resources are used to customize actions for novel or difficult situations. Optimal transfer of learning, and hence the design of instruction, differs depending on whether the goal is efficient or innovative practice. However, the task is necessarily further complicated when the aspiration is an adaptive expert practitioner who can fluidly balance innovation with efficiency as the situation requires. Using HPE examples at both the individual and organizational level, the authors explore the instructional implications of learning to shift from efficient to innovative expert functioning, and back. They argue that the efficiency-innovation tension is likely to endure deep into the future and therefore warrants important consideration in HPE.

Development and Pilot Testing of a Computerized Asthma Kiosk to Initiate Chronic Asthma Care in a Pediatric Emergency Department.

OBJECTIVES: Emergency department (ED) visits are an opportunity to initiate chronic asthma care. Ideally, this care should be implemented in a fashion that limits utilization of scarce ED resources. We developed, iteratively refined, and pilot tested the feasibility of a computerized asthma kiosk to (1) capture asthma information, (2) deliver asthma education, and (3) facilitate guideline-based chronic asthma management. METHODS: The following are the 4 phases of this study: (1) developing the content and structure of a computerized asthma kiosk, (2) iterative refinement through heuristic testing by human-computer interface experts, (3) usability testing with ED providers (n = 4) and caregivers of children with asthma (n = 4), and (4) pilot testing the kiosk with caregivers (n = 31) and providers in the ED (n = 18). Outcome measures for the pilot-testing phase were the proportion of ED providers who prescribed long-term controller medication (LTCM) and asthma action plans (AsAPs) and the proportion of children who took LTCMs and attended primary care providers follow-up. RESULTS: After kiosk development and refinement, pilot implementation resulted in LTCMs prescribing and AsAP provision for 19 (61%) of 31 and 17 (55%) of 31 patients, respectively. Before kiosk use, the proportion of the 18 ED providers who reported prescribing LTCM was 1 (5%) of 18, and providing AsAPs was 0 (0%) of 18. Eighteen (58%) of the 31 caregivers reported that their children used LTCMs after kiosk use and 13 (42%) of 31 reported following up with the primary care provider within 1 month of the ED visits. CONCLUSIONS: A rigorously developed asthma kiosk showed promise for initiating chronic asthma care in the ED.

Asthma-Related Educational Needs of Families With Children With Asthma in an Urban Pediatric Emergency Department.

OBJECTIVE: The aim of this study was to identify the educational needs of inner-city children with persistent asthma and their caregivers who utilize the emergency department (ED) for asthma care as well as determine their guideline adherence, factors associated with ED use, and comfort with computers. METHODS: Cross-sectional survey of children aged 2 to 18 years with previous diagnosis of asthma presenting with asthma-related complaints or acute asthma exacerbations to an urban pediatric ED. Data on demographics, families' response to acute asthma, approach to asthma prevention, access to care, educational topics of interest, and sources of health information were collected. RESULTS: Of approximately 1500 asthma-related visits, 218 caregivers were approached, and 200 completed the survey. In the past 12 months, 31% had experienced at least 1 asthma-related hospitalization, and 55.5% had had at least 3 ED visits. Although 184 (92.9%) of 198 caregivers were able to identify a primary physician, 37% reported they were more likely to take their child to the ED in response to acute asthma during the day as opposed to their physician (17%). Approximately half of patients were not on any preventive medication, with 57% not having had received an Asthma Action Plan. Caregivers expressed the most interest in learning about long-term controller medications (44.2%), use of metered dose inhalers or nebulizers (44.2%), and trigger avoidance (35.2%). Most caregivers (approximately 68%) reported ease of use with computers and the Internet. CONCLUSION: There was discordance between caregivers' reports of primary care provider teaching on asthma management and the use of the controller medications and possession of the Asthma Action Plans for persistent asthma. Education could focus on caregiver concerns of the safety and benefits of the controller medications.

Script Concordance Testing to Determine Infant Lumbar Puncture Practice Variation.

OBJECTIVES: Script concordance testing (SCT) is used to assess clinical decision-making. We explore the use of SCT to (1) quantify practice variations in infant lumbar puncture (LP) and (2) analyze physician's characteristics affecting LP decision making. METHODS: Using standard SCT processes, a panel of pediatric subspecialty physicians constructed 15 infant LP case vignettes, each with 2 to 4 SCT questions (a total of 47). The vignettes were distributed to pediatric attending physicians and fellows at 10 hospitals within the INSPIRE Network. We determined both raw scores (tendency to perform LP) and SCT scores (agreement with the reference panel) as well as the variation with participant factors. RESULTS: Two hundred twenty-six respondents completed all 47 SCT questions. Pediatric emergency medicine physicians tended to select LP more frequently than did general pediatricians, with pediatric emergency medicine physicians showing significantly higher raw scores (20.2 ± 10.2) than general pediatricians (13 ± 15; 95% confidence interval for difference, 1, 13). Concordance with the reference panel varied among subspecialties and by the frequency with which practitioners perform LPs in their practices. CONCLUSION: Script concordance testing questions can be used as a tool to detect subspecialty practice variation. We are able to detect significant practice variation in the self-report of use of LP for infants among different pediatric subspecialties.

Sequential dependencies in categorical judgments of radiographic images.

Sequential context effects, the psychological interactions occurring between the events of successive trials when a sequence of similar stimuli are judged, have interested psychologists for decades. It has been well established that individuals exhibit sequential context effects in psychophysical experiments involving unidimensional stimuli. Recent evidence shows that these effects generalize to quantitative judgments of more complex multidimensional stimuli such as images of faces, chairs, and shoes. In this article, we test for the presence of sequential context effects by re-examining previously published data on categorical judgments of 234 complex radiographic images made by 20 experienced physicians and 20 medical students engaged in an online training task. We found that medical students, but not experienced physicians, displayed evidence of sequential context effects. We also found evidence suggesting that as the students learned over blocks of trials, they tended to shift from relative comparisons between consecutive images toward more independent comparisons of each image against (strengthening) internalized standards.

A primer on the statistical modelling of learning curves in health professions education.

Learning curves are a useful way of representing the rate of learning over time. Features include an index of baseline performance (y-intercept), the efficiency of learning over time (slope parameter) and the maximal theoretical performance achievable (upper asymptote). Each of these parameters can be statistically modelled on an individual and group basis with the resulting estimates being useful to both learners and educators for feedback and educational quality improvement. In this primer, we review various descriptive and modelling techniques appropriate to learning curves including smoothing, regression modelling and application of the Thurstone model. Using an example dataset we demonstrate each technique as it specifically applies to learning curves and point out limitations.

"Yes, and " Exploring the Future of Learning Analytics in Medical Education.

This Conversations Starter article presents a selected research abstract from the 2017 Association of American Medical Colleges Northeastern Region Group on Educational Affairs annual spring meeting. The abstract is paired with the integrative commentary of three experts who shared their thoughts stimulated by the study. Commentators brainstormed "what's next" with learning analytics in medical education, including advancements in interaction metrics and the use of interactivity analysis to deepen understanding of perceptual, cognitive, and social learning and transfer processes.

A survey of mindset theories of intelligence and medical error self-reporting among pediatric housestaff and faculty.

BACKGROUND: Intelligence theory research has illustrated that people hold either "fixed" (intelligence is immutable) or "growth" (intelligence can be improved) mindsets and that these views may affect how people learn throughout their lifetime. Little is known about the mindsets of physicians, and how mindset may affect their lifetime learning and integration of feedback. Our objective was to determine if pediatric physicians are of the "fixed" or "growth" mindset and whether individual mindset affects perception of medical error reporting.  METHODS: We sent an anonymous electronic survey to pediatric residents and attending pediatricians at a tertiary care pediatric hospital. Respondents completed the "Theories of Intelligence Inventory" which classifies individuals on a 6-point scale ranging from 1 (Fixed Mindset) to 6 (Growth Mindset). Subsequent questions collected data on respondents' recall of medical errors by self or others. RESULTS: We received 176/349 responses (50 %). Participants were equally distributed between mindsets with 84 (49 %) classified as "fixed" and 86 (51 %) as "growth". Residents, fellows and attendings did not differ in terms of mindset. Mindset did not correlate with the small number of reported medical errors. CONCLUSIONS: There is no dominant theory of intelligence (mindset) amongst pediatric physicians. The distribution is similar to that seen in the general population. Mindset did not correlate with error reports.