Image interpretation: Learning analytics-informed education opportunities.

OBJECTIVES: Using a sample of pediatric chest radiographs (pCXR) taken to rule out pneumonia, we obtained diagnostic interpretations from physicians and used learning analytics to determine the radiographic variables and participant review processes that predicted for an incorrect diagnostic interpretation. METHODS: This was a prospective cross-sectional study. A convenience sample of frontline physicians with a range of experience levels interpreted 200 pCXR presented using a customized online radiograph presentation platform. Participants were asked to determine absence or presence (with respective location) of pneumonia. The pCXR were categorized for specific image-based variables potentially associated with interpretation difficulty. We also generated heat maps displaying the locations of diagnostic error among normal pCXR. Finally, we compared image review processes in participants with higher versus lower levels of clinical experience. RESULTS: We enrolled 83 participants (20 medical students, 40 postgraduate trainees, and 23 faculty) and obtained 12,178 case interpretations. Variables that predicted for increased pCXR interpretation difficulty were pneumonia versus no pneumonia (ß = 8.7, 95% confidence interval [CI] = 7.4 to 10.0), low versus higher visibility of pneumonia (ß = -2.2, 95% CI = -2.7 to -1.7), nonspecific lung pathology (ß = 0.9, 95% CI = 0.40 to 1.5), localized versus multifocal pneumonia (ß = -0.5, 95% CI = -0.8 to -0.1), and one versus two views (ß = 0.9, 95% CI = 0.01 to 1.9). A review of diagnostic errors identified that bony structures, vessels in the perihilar region, peribronchial thickening, and thymus were often mistaken for pneumonia. Participants with lower experience were less accurate when they reviewed one of two available views (p < 0.0001), and accuracy of those with higher experience increased with increased confidence in their response (p < 0.0001). CONCLUSIONS: Using learning analytics, we identified actionable learning opportunities for pCXR interpretation, which can be used to allow for a customized weighting of which cases to practice. Furthermore, experienced-novice comparisons revealed image review processes that were associated with greater diagnostic accuracy, providing additional insight into skill development of image interpretation.

Prepubescent Female Genital Examination Images: Evidence-Informed Learning Opportunities.

OBJECTIVES: To determine diagnoses and image features that are associated with difficult prepubescent female genital image interpretations. DESIGN AND SETTING: This was a mixed-methods study conducted at a tertiary care pediatric center using images from a previously developed education platform. PARTICIPANTS: Participants comprised 107 medical students, residents, fellows, and attendings who interpreted 158 cases to derive case difficulty estimates. INTERVENTIONS: This was a planned secondary analysis of participant performance data obtained from a prospective multi-center cross-sectional study. An expert panel also performed a descriptive review of images with the highest frequency of diagnostic error. MAIN OUTCOME MEASURES: We derived the proportion of participants who interpreted an image correctly, and features that were common in images with the most frequent diagnostic errors. RESULTS: We obtained 16,906 image interpretations. The mean proportion correct scores for each diagnosis were as follows: normal/normal variants 0.84 (95% confidence interval [CI] 0.82, 0.87); infectious/dermatology pathology 0.59 (95% CI 0.45, 0.73); anatomic pathology 0.61 (95% CI 0.41, 0.81); and, traumatic pathology 0.64 (95% CI 0.49, 0.79). The mean proportion correct scores varied by diagnosis (P < .001). The descriptive review demonstrated that poor image quality, infant genitalia, normal variant anatomy, external material (eg, diaper cream) in the genital area, and nonspecific erythema were common features in images with lower accuracy scores. CONCLUSIONS: A quantitative and qualitative examination of prepubescent female genital examination image interpretations provided insight into diagnostic challenges for this complex examination. These data can be used to inform the design of teaching interventions to improve skill in this area.

The Variable Journey in Learning to Interpret Pediatric Point-of-care Ultrasound Images: A Multicenter Prospective Cohort Study.

OBJECTIVES: To complement bedside learning of point-of-care ultrasound (POCUS), we developed an online learning assessment platform for the visual interpretation component of this skill. This study examined the amount and rate of skill acquisition in POCUS image interpretation in a cohort of pediatric emergency medicine (PEM) physician learners. METHODS: This was a multicenter prospective cohort study. PEM physicians learned POCUS using a computer-based image repository and learning assessment system that allowed participants to deliberately practice image interpretation of 400 images from four pediatric POCUS applications (soft tissue, lung, cardiac, and focused assessment sonography for trauma [FAST]). Participants completed at least one application (100 cases) over a 4-week period. RESULTS: We enrolled 172 PEM physicians (114 attendings, 65 fellows). The increase in accuracy from the initial to final 25 cases was 11.6%, 9.8%, 7.4%, and 8.6% for soft tissue, lung, cardiac, and FAST, respectively. For all applications, the average learners (50th percentile) required 0 to 45, 25 to 97, 66 to 175, and 141 to 290 cases to reach 80, 85, 90, and 95% accuracy, respectively. The least efficient (95th percentile) learners required 60 to 288, 109 to 456, 160 to 666, and 243 to 1040 cases to reach these same accuracy benchmarks. Generally, the soft tissue application required participants to complete the least number of cases to reach a given proficiency level, while the cardiac application required the most. CONCLUSIONS: Deliberate practice of pediatric POCUS image cases using an online learning and assessment platform may lead to skill improvement in POCUS image interpretation. Importantly, there was a highly variable rate of achievement across learners and applications. These data inform our understanding of POCUS image interpretation skill development and could complement bedside learning and performance assessments.

A think-aloud study to inform the design of radiograph interpretation practice.

Models for diagnostic reasoning in radiology have been based on the observed behaviors of experienced radiologists but have not directly focused on the thought processes of novices as they improve their accuracy of image interpretation. By collecting think-aloud verbal reports, the current study was designed to investigate differences in specific thought processes between medical students (novices) as they learn and radiologists (experts), so that we can better design future instructional environments. Seven medical students and four physicians with radiology training were asked to interpret and diagnose pediatric elbow radiographs where fracture is suspected. After reporting their diagnosis of a case, they were given immediate feedback. Participants were asked to verbalize their thoughts while completing the diagnosis and while they reflected on the provided feedback. The protocol analysis of their verbalizations showed that participants used some combination of four processes to interpret the case: gestalt interpretation, purposeful search, rule application, and reasoning from a prior case. All types of processes except reasoning from a prior case were applied significantly more frequently by experts. Further, gestalt interpretation was used with higher frequency in abnormal cases while purposeful search was used more often for normal cases. Our assessment of processes could help guide the design of instructional environments with well-curated image banks and analytics to facilitate the novice's journey to expertise in image interpretation.

The effect of testing and feedback on the forgetting curves for radiograph interpretation skills.

Objectives: Forgetting curves plot skill decay over time. After exposure to a simulation-based radiograph interpretation learning system, we determined the rate of learning decay and how this was impacted by testing (with and without feedback). Further, we examined the association of initial learning parameters on the forgetting curve. Methods: This was a multicenter, four-arm randomized control trial. Medical trainees completed 80 elbow radiographs and a 20-case post-test. Group 1 had no testing until 12 months; Groups 2-4 had testing every 2 months until 12 months. At 6 months, Group 3 testing was feedback-enhanced, while Group 4 had feedback-enhanced testing at 2, 6, and 10 months. Results: There were 106 participants (n = 42 Group 1; n = 22 Groups 2 and 3; n = 20 Group 4). Group 1 showed an -8.1% learning decay at 12-months relative to other groups. In Groups 2, 3, and 4, there was no significant learning decay (+0.8%), and there were no differences in skill decay between these groups. Initial score and learning curve slope were predictive of retained skill. Conclusions: Learning decay was mitigated by exposure to 20 test cases (with and without feedback) every two months. Initial learning parameters predicted learning retention and may inform refresher education scheduling.

Sequential dependencies in categorical judgments of radiographic images.

Sequential context effects, the psychological interactions occurring between the events of successive trials when a sequence of similar stimuli are judged, have interested psychologists for decades. It has been well established that individuals exhibit sequential context effects in psychophysical experiments involving unidimensional stimuli. Recent evidence shows that these effects generalize to quantitative judgments of more complex multidimensional stimuli such as images of faces, chairs, and shoes. In this article, we test for the presence of sequential context effects by re-examining previously published data on categorical judgments of 234 complex radiographic images made by 20 experienced physicians and 20 medical students engaged in an online training task. We found that medical students, but not experienced physicians, displayed evidence of sequential context effects. We also found evidence suggesting that as the students learned over blocks of trials, they tended to shift from relative comparisons between consecutive images toward more independent comparisons of each image against (strengthening) internalized standards.

A primer on the statistical modelling of learning curves in health professions education.

Learning curves are a useful way of representing the rate of learning over time. Features include an index of baseline performance (y-intercept), the efficiency of learning over time (slope parameter) and the maximal theoretical performance achievable (upper asymptote). Each of these parameters can be statistically modelled on an individual and group basis with the resulting estimates being useful to both learners and educators for feedback and educational quality improvement. In this primer, we review various descriptive and modelling techniques appropriate to learning curves including smoothing, regression modelling and application of the Thurstone model. Using an example dataset we demonstrate each technique as it specifically applies to learning curves and point out limitations.

A Big Data and Learning Analytics Approach to Process-Level Feedback in Cognitive Simulations.

Collecting and analyzing large amounts of process data for the purposes of education can be considered a big data/learning analytics (BD/LA) approach to improving learning. However, in the education of health care professionals, the application of BD/LA is limited to date. The authors discuss the potential advantages of the BD/LA approach for the process of learning via cognitive simulations. Using the lens of a cognitive model of radiograph interpretation with four phases (orientation, searching/scanning, feature detection, and decision making), they reanalyzed process data from a cognitive simulation of pediatric ankle radiography where 46 practitioners from three expertise levels classified 234 cases online. To illustrate the big data component, they highlight the data available in a digital environment (time-stamped, click-level process data). Learning analytics were illustrated using algorithmic computer-enabled approaches to process-level feedback.For each phase, the authors were able to identify examples of potentially useful BD/LA measures. For orientation, the trackable behavior of re-reviewing the clinical history was associated with increased diagnostic accuracy. For searching/scanning, evidence of skipping views was associated with an increased false-negative rate. For feature detection, heat maps overlaid on the radiograph can provide a metacognitive visualization of common novice errors. For decision making, the measured influence of sequence effects can reflect susceptibility to bias, whereas computer-generated path maps can provide insights into learners' diagnostic strategies.In conclusion, the augmented collection and dynamic analysis of learning process data within a cognitive simulation can improve feedback and prompt more precise reflection on a novice clinician's skill development.

Interpretation difficulty of normal versus abnormal radiographs using a pediatric example.

BACKGROUND: Radiograph teaching files are usually dominated by abnormal cases, implying that normal radiographs are easier to interpret. Our main objective was to compare the interpretation difficulty of normal versus abnormal radiographs of a set of common pediatric radiographs. METHODS: We developed a 234-item digital case bank of pediatric ankle radiographs, recruited a convenience sample of participants, and presented the cases to each participant who then classified the cases as normal or abnormal. We determined and contrasted the interpretation difficulty of the normal and abnormal x-rays items using Rasch Measurement Theory. We also identified case features that were associated with item difficulty. RESULTS: 139 participants (86 medical students, 7 residents, 29 fellows, 5 emergency physicians, and 3 radiologists) rated a minimum of 50 cases each, which resulted in 16,535 total ratings. Abnormal cases were more difficult (+0.99 logits) than were normal ones (-0.58 logits), difference 1.57 logits (95% CI 1.2, 2.0), but there was considerable overlap in difficulty scores. Patient variables associated with a more difficult normal radiograph included younger patient age (ß = -0.16, 95% CI -0.22, -0.10), history of distal fibular tenderness (ß = 0.55, 95% CI 0.17, 0.93), and presence of a secondary ossification centre (ß = 0.84, 95% CI 0.27, 1.41). CONCLUSIONS: While abnormal images were more difficult to interpret, normal images did show a range of interpretation difficulties. Including a significant proportion of normal cases may be of benefit to learners.

Accuracy of self-monitoring during learning of radiograph interpretation.

CONTEXT: Despite calls for the improvement of self-assessment as a basis for self-directed learning, instructional designs that include reflection in practice are uncommon. Using data from a screen-based simulation for learning radiograph interpretation, we present validity evidence for a simple self-monitoring measure and examine how it can complement skill assessment. METHODS: Medical students learning ankle radiograph interpretation were given an online learning set of 50 cases which they were asked to classify as 'abnormal' (fractured) or 'normal' and to indicate the degree to which they felt certain about their response (Definitely or Probably). They received immediate feedback on each case. All students subsequently completed two 20-case post-tests: an immediate post-test (IPT), and a delayed post-test (DPT) administered 2 weeks later. We determined the degree to which certainty (Definitely versus Probably) correlated with accuracy of interpretation and how this relationship changed between the tests. RESULTS: Of 988 students approached, 115 completed both tests. Mean ± SD accuracy scores decreased from 59 ± 17% at the IPT to 53 ± 16% at the DPT (95% confidence interval [CI] for the difference: -2% to -10%). Mean self-assessed certainty did not decrease (rates of Definitely: IPT, 17.6%; DPT, 19.5%; 95% CI for difference: +7.2% to -3.4%). Regression modelling showed that accuracy was positively associated with choosing Definitely over Probably (odds ratio [OR] 1.63, 95% CI 1.27-2.09) and indicated a statistically significant interaction between test timing and certainty (OR 0.72, 95% CI 0.52-0.99); thus, the accuracy of self-monitoring decayed over the retention interval, leaving students relatively overconfident in their abilities. CONCLUSIONS: This study shows that, in medical students learning radiograph interpretation, the development of self-monitoring skills can be measured and should not be assumed to necessarily vary in the same way as the underlying clinical skill.

A hinting strategy for online learning of radiograph interpretation by medical students.

CONTEXT: We examined whether a 'hint' manoeuvre increases the time novice medical learners spend on reviewing a radiograph, thereby potentially increasing their interpretation accuracy. METHODS: Senior year medical students were recruited into a randomised control, three-arm, multicentre trial. Students reviewed an online 50-case learning set that varied in degree of 'hint' intervention. The 'hint' was a dialogue box that appeared after a student submitted an answer, encouraging the student to re-evaluate their interpretation. The students in the control group received no hints. In the weak intervention group, students received 'hints' with 66% of their incorrect interpretations and 33% of those that were correct. In the strong intervention group, the incorrect interpretation hint frequency was 80%, whereas for correct responses it was 20%. All students completed a 20-case post-test immediately and 2 weeks after the 50 cases. The primary outcome was student performance on the immediate post-test, measured as the ability to discriminate between normal and abnormal films (dPrime). Secondary outcomes included the probability of considering the hint, time spent on learning cases and knowledge retention at 2 weeks. RESULTS: We enrolled 117 medical students from three sites into the three study groups: control (36), weak intervention (40) and strong intervention (41) groups. The mean (standard deviation) dPrime in the control, weak and strong groups were 0.4 (1.1), 0.7 (1.1) and 0.4 (0.9), respectively (P = 0.4). In the weak and strong groups, participants reconsidered answers in 556 of 1944 (28.6%) hinting opportunities, and those who reconsidered their answers spent a mean (95% confidence interval) of 13.9 (11.9, 16.0) seconds longer on each case. There were no significant differences in knowledge retention at 2 weeks between the groups (P = 0.2). CONCLUSIONS: Although the implemented hinting strategy did result in students spending more time considering a proportion of the cases, overall it was not effective in improving student performance.

Experience curves as an organizing framework for deliberate practice in emergency medicine learning.

Deliberate practice is an important skill-training strategy in emergency medicine (EM) education. Learning curves display the relationship between practice and proficiency. Forgetting curves show the opposite, and demonstrate how skill decays over time when it is not reinforced. Using examples of published studies of deliberate practice in EM we list the properties of learning and forgetting curves and suggest how they can be combined to create experience curves: a longitudinal representation of the relationship between practice, skill acquisition, and decay over time. This framework makes explicit the need to avoid a piecemeal, episodic approach to skill practice and assessment in favor of more emphasis on what can be done to improve durability of competence over time. The authors highlight the implications for both educators and education researchers.

Prevalence of abnormal cases in an image bank affects the learning of radiograph interpretation.

OBJECTIVES: Using a large image bank, we systematically examined how the use of different ratios of abnormal to normal cases affects trainee learning. METHODS: This was a prospective, double-blind, randomised, three-arm education trial conducted in six academic training programmes for emergency medicine and paediatric residents in post-licensure years 2-5. We developed a paediatric ankle trauma radiograph case bank. From this bank, we constructed three different 50-case training sets, which varied in their proportions of abnormal cases (30%, 50%, 70%). Levels of difficulty and diagnoses were similar across sets. We randomly assigned residents to complete one of the training sets. Users classified each case as normal or abnormal, specifying the locations of any abnormalities. They received immediate feedback. All participants completed the same 20-case post-test in which 40% of cases were abnormal. We determined participant sensitivity, specificity, likelihood ratio and signal detection parameters. RESULTS: A total of 100 residents completed the study. The groups did not differ in accuracy on the post-test (p = 0.20). However, they showed considerable variation in their sensitivity-specificity trade-off. The group that received a training set with a high proportion of abnormal cases achieved the best sensitivity (0.69, standard deviation [SD] = 0.24), whereas the groups that received training sets with medium and low proportions of abnormal cases demonstrated sensitivities of 0.63 (SD = 0.21) and 0.51 (SD = 0.24), respectively (p < 0.01). Conversely, the group with a low proportion of abnormal cases demonstrated the best specificity (0.83, SD = 0.10) compared with the groups with medium (0.70, SD = 0.15) and high (0.66, SD = 0.17) proportions of abnormal cases (p < 0.001). The group with a low proportion of abnormal cases had the highest false negative rate and missed fractures one-third more often than the groups that trained on higher proportions of abnormal cases. CONCLUSIONS: Manipulating the ratio of abnormal to normal cases in learning banks can have important educational implications.

How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation.

PURPOSE: To demonstrate how learning curves can describe proficiency improvements associated with deliberate practice of radiograph interpretation. METHOD: This was a prospective, cross-sectional study of pediatric residents in two tertiary care programs. A 234-item digital case bank of pediatric ankle radiographs was developed. The authors gave participants a brief clinical summary of each case and asked them to consider three radiograph views of the ankle. Participants classified each case as either normal or abnormal and, if applicable, specified the location of the abnormality. They received immediate feedback and a radiologist's dictated report. The authors reviewed longitudinal learning curves, which were generated based on calculated test characteristics (e.g., accuracy, sensitivity, specificity). RESULTS: Eighteen participants (56.3% of those eligible) completed all 234 cases. The form of the participants' learning curves was similar across all test characteristics. The curves showed a period of "noise" until the participants completed an average of 20 cases. The slope of the learning curve was maximal from 21 to 50 cases during which cumulative sensitivity (95% CI) increased from 0.50 (0.45, 0.57) to 0.54 (0.47, 0.58). Then, the curves reached an inflection point after which learning slowed but did not stop even after 234 cases. The final cumulative sensitivity was 0.60 (0.54, 0.63). Applying a reference criterion, the authors classified learners into formative categories. CONCLUSIONS: Learning curves describing deliberate practice of radiograph interpretation allow medical educators to define at which point(s) practice is most efficient and how much practice is required to achieve a defined level of mastery.

Pediatric emergency physician opinions on ankle radiograph clinical decision rules.

OBJECTIVES: The Low Risk Ankle Rule (LRAR) is a validated clinical decision rule (CDR) about the indications for ankle radiographs in children with acute blunt ankle trauma. Although application of the LRAR has the potential to safely reduce the rate of ankle radiography by 60%, current x-ray rates in most emergency departments (EDs) in the United States and Canada remain unnecessarily high (85%-100%). To evaluate this gap between knowledge and practice, physicians who treat pediatric ankle injuries in EDs were surveyed to determine physician awareness and use of the LRAR, acceptability of the LRAR as measured by the Ottawa Acceptability for Decision Rules Scale (OADRS), and perceived barriers to the use of a validated pediatric ankle x-ray rule. METHODS: An on-line survey of members of two national pediatric emergency medicine (PEM) physician associations in the United States and Canada was conducted using a modified Dillman technique. RESULTS: Response rates were 75.6% (149/197) in Canada and 45.7% (352/770) in the United States, yielding an aggregate rate of 51.8%. Only 119 of 478 respondents (24.9%) had heard of the LRAR, and 53 of 432 (12.3%) were sufficiently familiar with the LRAR to apply it. The LRAR scored a mean (+/- standard deviation [SD]) OADRS score of 4.28 out of 6 (+/-0.67), comparable to published OADRS scores for two well-known CDRs used in adults. Of the respondents, 434 of 471 (92.1%) at least "slightly agreed" that ankle x-ray CDRs would be useful in their practice, with no significant differences between the two sides of the border (p = 0.28). Ankle x-ray rules were felt to save time by 342 (72.6%) of the participants, and the pediatric ankle exam was considered easy enough to apply a CDR by 306 (65.0%). The most common barriers reported for use of any ankle x-ray rule included perceived reduction in family satisfaction without imaging in 380 (80.7%), nurse-initiated x-ray protocols not based on ankle x-ray rules in 285 (60.5%), concerns about missing a significant fracture in 248 (52.7%), and a preference for own clinical judgment in 246 (52.2%). CONCLUSIONS: Although the LRAR had a high acceptability score among respondents in this survey, this validated CDR is not widely known and is even less frequently applied by PEM physicians in the United States and Canada. Barriers were identified that will guide efforts to improve the knowledge translation of the LRAR into pediatric EDs.

Using signal detection theory to model changes in serial learning of radiological image interpretation.

Signal detection theory (SDT) parameters can describe a learner's ability to discriminate (d') normal from abnormal and the learner's criterion (λ) to under or overcall abnormalities. To examine the serial changes in SDT parameters with serial exposure to radiological cases. 46 participants were recruited for this study: 20 medical students (MED), 6 residents (RES), 12 fellows (FEL), 5 staff pediatric emergency physicians (PEM), and 3 staff radiologists (RAD). Each participant was presented with 234 randomly assigned ankle radiographs using a web-based application. Participants were given a clinical scenario and considered 3 views of the ankle. They classified each case as normal or abnormal. For abnormal cases, they specified the location of the abnormality. Immediate feedback included highlighting on the images and the official radiologist's report. The low experience group (MED, RES, FEL) showed steady improvement in discrimination ability with each case, while the high experience group (PEM, RAD) had higher and stable discrimination ability throughout the exercise. There was also a difference in the way the high and low experience groups balanced sensitivity and specificity (λ) with the low experience group tending to make more errors calling positive radiographs negative. This tendency was progressively less evident with each increase in expertise level. SDT metrics provide valuable insight on changes associated with learning radiograph interpretation, and may be used to design more effective instructional strategies for a given learner group.

Teaching X-ray interpretation: selecting the radiographs by the target population.

CONTEXT: The unbiased selection of images representing a spectrum of diagnostic difficulty is an important first step in designing effective assessment and teaching interventions for X-ray interpretation. OBJECTIVES: This study aimed to develop a scale that would reliably differentiate more difficult X-rays from those that are easier to interpret. METHODS: After pilot testing, an X-ray difficulty scale (XRDS) was developed. Raters of different learner levels from two universities were presented with 20 chest X-rays (CXRs) and asked to read them and then to answer the scale questions that would help to differentiate the level of difficulty of interpretation of each film. Reliability of the scale was evaluated. Face validity of the scale was assessed and the construct validity of two hypotheses was tested. RESULTS: The final scale consisted of five questions in which a given X-ray could score from--10 (most difficult) to + 10 (easiest to interpret) by a single rater. Raters included 53 medical students, 10 paediatric residents and 10 emergency staff. The scale demonstrated excellent internal consistency (r = 0.94), inter-rater reliability (r = 0.95) and overall reliability (r = 0.90) in medical students. Construct validity testing demonstrated good correlation (r = 0.72) between diagnostic accuracy and mean XRDS score. Mean scores on the scale were significantly lower (indicating that CXRs were more difficult to interpret) for students than for resident and staff doctors (P < 0.0001). CONCLUSIONS: The scale developed in this study serves as a reliable and valid tool for categorising CXRs according to diagnostic difficulty, which reduces the bias inherent in the process of selecting radiographs by expert opinion alone.