Learner engagement and teaching effectiveness in livestreamed versus in-person CME.

INTRODUCTION: Engaging learners in continuing medical education (CME) is challenging. Recently, CME courses have transitioned to livestreamed CME, with learners viewing live, in-person courses online. The authors aimed to (1) compare learner engagement and teaching effectiveness in livestreamed with in-person CME and (2) determine how livestream engagement and teaching effectiveness is associated with (A) interactivity metrics, (B) presentation characteristics and (C) medical knowledge. METHODS: A 3-year, non-randomised study of in-person and livestream CME was performed. The course was in-person for 2018 but transitioned to livestream for 2020 and 2021. Learners completed the Learner Engagement Inventory and Teaching Effectiveness Instrument after each presentation. Both instruments were supported by content, internal structure and relations to other variables' validity evidence. Interactivity metrics included learner use of audience response, questions asked by learners and presentation views. Presentation characteristics included presentations using audience response, using pre/post-test format, time of day and words per slide. Medical knowledge was assessed by audience response. A repeated measures analysis of variance (anova) was used for comparisons and a mixed model approach for correlations. RESULTS: A total of 159 learners (response rate 27%) completed questionnaires. Engagement did not significantly differ between in-person or livestream CME. (4.56 versus 4.53, p = 0.64, maximum 5 = highly engaged). However, teacher effectiveness scores were higher for in-person compared with livestream (4.77 versus 4.71 p = 0.01, maximum 5 = highly effective). For livestreamed courses, learner engagement was associated with presentation characteristics, including presentation using of audience response (yes = 4.57, no = 4.45, p < .0001), use of a pre/post-test (yes = 4.62, no = 4.54, p < .0001) and time of presentation (morning = 4.58, afternoon = 4.53, p = .0002). Significant associations were not seen for interactivity metrics or medical knowledge. DISCUSSION: Livestreaming may be as engaging as in-person CME. Although teaching effectiveness in livestreaming was lower, this difference was small. CME course planners should consider offering livestream CME while exploring strategies to enhance teaching effectiveness in livestreamed settings.

Pre-medical majors in the humanities and social sciences: impact on communication skills and specialty choice.

CONTEXT: Medical school admissions committees use a variety of criteria to determine which candidates to admit to their programmes. Effective communication is increasingly considered a key requisite to the practice of effective medicine. Medical students with pre-medical backgrounds in the humanities and social sciences may be more likely to acquire skills relevant to patient-centred communication, either prior to or during medical school. OBJECTIVES: The purpose of this study was to investigate the relationship between pre-medical backgrounds in the humanities and social sciences and outcomes in medical school, including in communication and interpersonal skills (CIS), licensure examination results and postgraduate specialty choice (primary care versus non-primary care specialties). METHODS: The American Medical College Application Service database was used to identify pre-medical college majors, demographic characteristics, Medical College Admission Test scores and college grade point averages for medical students at a large, midwestern medical school. Data were obtained for 465 medical students across three cohorts (classes of 2014-2016). Correlation and regression analyses were used to examine relationships between pre-medical background, performance on graduation competency examination standardised patient encounter CIS scores and on United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores, and postgraduate specialty choice. RESULTS: Graduating medical students with pre-medical humanities or social sciences majors performed significantly better in terms of CIS than those with natural science majors (Cohen's d = 0.28, p = 0.011). There were no significant associations between pre-medical majors and USMLE Step 1 and Step 2 Clinical Knowledge scores or postgraduate specialty choice. CONCLUSIONS: These results suggest that considering humanistic factors as part of admissions criteria may promote the selection and training of physicians with good communication skills.

Validity Evidence for a Residency Admissions Standardized Assessment Letter for Pediatrics.

Construct: This study aims to provide validity evidence for the standardized Assessment Letter for Pediatrics as a measure of competencies expected of a 1st-year pediatrics resident as part of a pediatric residency admissions process. BACKGROUND: The Narrative Letter of Recommendation is a frequently used tool in the residency admissions process even though it has poor interrater reliability, lacks pertinent content, and does not correlate with residency performance. A newer tool, the Standardized Letter, has shown validity evidence for content and interrater reliability in other specialties. We sought to develop and provide validity evidence for the standardized Assessment Letter for Pediatrics. APPROACH: All 2012 and 2013 applicants invited to interview at the University of Geneva Pediatrics Residency Program provided 2 standardized Assessment Letters. Content for the letter was based on CanMEDS roles and ratings of 6 desired competencies and an overall assessment. Validity evidence was gathered for internal structure (Cronbach's alpha and generalizability), response process (interrater reliability with intraclass correlation), relations to other variables (Pearson's correlation coefficient), and consequences (logistic regression to predict admission). RESULTS: One hundred fourteen faculty completed 142 standardized Assessment Letters for 71 applicants. Average overall assessment was 3.0 of 4 (SD = 0.59). Cronbach's alpha was 0.93. The G-coefficient was 0.59. The decision study projected that four Assessment Letters are needed to attain a G-coefficient of 0.73. Applicant variance (28.5%) indicated high applicant differentiation. The Assessment Letter intraclass coefficient was 0.51, 95% confidence interval (CI) [0.43, 0.59]. Assessment Letter scores were correlated with the structured interview (r = .28), 95% CI [0.05, 0.51]; global rating (r = .36), 95% CI [0.13, 0.58]; and admissions decision (r = .25), 95% CI [0.02, 0.46]. Assessment Letter scores did not predict the admissions decision (odds ratio = 1.67, p = .37) after controlling for the unique contribution of the structured interview and global rating scores. CONCLUSION: Validity evidence supports use of the Assessment Letter for Pediatrics; future studies should refine items to improve predictive validity and explore how to best integrate the Assessment Letter into the residency admissions process.

Rescuing the Clinical Breast Examination: Advances in Classifying Technique and Assessing Physician Competency.

OBJECTIVE: Develop new performance evaluation standards for the clinical breast examination (CBE). SUMMARY BACKGROUND DATA: There are several, technical aspects of a proper CBE. Our recent work discovered a significant, linear relationship between palpation force and CBE accuracy. This article investigates the relationship between other technical aspects of the CBE and accuracy. METHODS: This performance assessment study involved data collection from physicians (n = 553) attending 3 different clinical meetings between 2013 and 2014: American Society of Breast Surgeons, American Academy of Family Physicians, and American College of Obstetricians and Gynecologists. Four, previously validated, sensor-enabled breast models were used for clinical skills assessment. Models A and B had solitary, superficial, 2 cm and 1 cm soft masses, respectively. Models C and D had solitary, deep, 2 cm hard and moderately firm masses, respectively. Finger movements (search technique) from 1137 CBE video recordings were independently classified by 2 observers. Final classifications were compared with CBE accuracy. RESULTS: Accuracy rates were model A = 99.6%, model B = 89.7%, model C = 75%, and model D = 60%. Final classification categories for search technique included rubbing movement, vertical movement, piano fingers, and other. Interrater reliability was (k = 0.79). Rubbing movement was 4 times more likely to yield an accurate assessment (odds ratio 3.81, P < 0.001) compared with vertical movement and piano fingers. Piano fingers had the highest failure rate (36.5%). Regression analysis of search pattern, search technique, palpation force, examination time, and 6 demographic variables, revealed that search technique independently and significantly affected CBE accuracy (P < 0.001). CONCLUSIONS: Our results support measurement and classification of CBE techniques and provide the foundation for a new paradigm in teaching and assessing hands-on clinical skills. The newly described piano fingers palpation technique was noted to have unusually high failure rates. Medical educators should be aware of the potential differences in effectiveness for various CBE techniques.

Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility study.

The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane's formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon signed-rank test. Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r = 0.25, P = .036), SI (r = 0.34, P = .004), and global ratings (r = 0.48, P < .001). Generalizability of the P-MEX using three cases was moderate (G-coefficient = 0.45). P-MEX scores had the greatest correlation with acceptance (r = 0.56, P < .001), were the strongest predictor of acceptance (OR 4.37, P < .001), and increased pseudo R-squared by 0.20 points. Including P-MEX scores increased composite score reliability from 0.51 to 0.74. Rank lists of applicants using composite score versus global rating differed significantly (z = 5.41, P < .001). Validity evidence supports the use of P-MEX scores to improve the reliability of the residency admissions process by improving applicant composite score reliability.

Inter-rater reliability and generalizability of patient note scores using a scoring rubric based on the USMLE Step-2 CS format.

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was .79 (Documentation = .63, Differential Diagnosis = .90, Justification = .48, and Workup = .54). The majority of score variance was due to case specificity (13 %) and case-task specificity (31 %), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5 %). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.

A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents.

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

Reducing the number of options on multiple-choice questions: response time, psychometrics and standard setting.

OBJECTIVES: Despite significant evidence supporting the use of three-option multiple-choice questions (MCQs), these are rarely used in written examinations for health professions students. The purpose of this study was to examine the effects of reducing four- and five-option MCQs to three-option MCQs on response times, psychometric characteristics, and absolute standard setting judgements in a pharmacology examination administered to health professions students. METHODS: We administered two versions of a computerised examination containing 98 MCQs to 38 Year 2 medical students and 39 Year 3 pharmacy students. Four- and five-option MCQs were converted into three-option MCQs to create two versions of the examination. Differences in response time, item difficulty and discrimination, and reliability were evaluated. Medical and pharmacy faculty judges provided three-level Angoff (TLA) ratings for all MCQs for both versions of the examination to allow the assessment of differences in cut scores. RESULTS: Students answered three-option MCQs an average of 5 seconds faster than they answered four- and five-option MCQs (36 seconds versus 41 seconds; p = 0.008). There were no significant differences in item difficulty and discrimination, or test reliability. Overall, the cut scores generated for three-option MCQs using the TLA ratings were 8 percentage points higher (p = 0.04). CONCLUSIONS: The use of three-option MCQs in a health professions examination resulted in a time saving equivalent to the completion of 16% more MCQs per 1-hour testing period, which may increase content validity and test score reliability, and minimise construct under-representation. The higher cut scores may result in higher failure rates if an absolute standard setting method, such as the TLA method, is used. The results from this study provide a cautious indication to health professions educators that using three-option MCQs does not threaten validity and may strengthen it by allowing additional MCQs to be tested in a fixed amount of testing time with no deleterious effect on the reliability of the test scores.

Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an Internal Medicine residency OSCE.

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.

Unannounced standardized patient assessment of the roter interaction analysis system: the challenge of measuring patient-centered communication.

BACKGROUND: Despite wide-spread endorsement of patient-centered communication (PCC) in health care, there has been little evidence that it leads to positive change in health outcomes. The lack of correlation may be due either to an overestimation of the value of PCC or to a measurement problem. If PCC measures do not capture elements of the interaction that determine whether the resulting care plan is patient-centered, they will confound efforts to link PCC to outcomes. OBJECTIVE: To evaluate whether one widely used measure of PCC, the Roter Interaction Analysis System (RIAS), captures patient-centered care planning. DESIGN: RIAS was employed in the coding of unannounced standardized patient (USP) encounters that were scripted so that the failure to address patient contextual factors would result in an ineffective plan of care. The design enabled an assessment of whether RIAS can differentiate between communication behavior that does and does not result in a care plan that takes into account a patient's circumstances and needs. PARTICIPANTS: Eight actors role playing four scripted cases (one African American and one Caucasian for each case) in 399 visits to 111 internal medicine attending physicians. MAIN MEASURES: RIAS measures included composites for physician utterance types and (in separate models) two different previously applied RIAS patient-centeredness summary composites. The gold standard comparison measure was whether the physician's treatment plan, as abstracted from the visit note, successfully addressed the patient's problem. Mixed effects regression models were used to evaluate the relationship between RIAS measures and USP measured performance, controlling for a variety of design features. KEY RESULTS: None of the RIAS measures of PCC differentiated encounters in which care planning was patient-centered from care planning in which it was not. CONCLUSIONS: RIAS, which codes each utterance during a visit into mutually exclusive and exhaustive categories, does not differentiate between conversations leading to and not leading to care plans that accommodate patients' circumstances and needs.

Assessing students' ability to detect melanomas using standardized patients and moulage.

BACKGROUND: Detection of melanoma by physicians via opportunistic surveillance during focused physical examinations may reduce mortality. Medical students may not encounter a clinical case of melanoma during a dermatology clerkship. OBJECTIVE: This study examined the proficiency of fourth-year University of Illinois at Chicago medical students at detecting melanomas. METHODS: Melanoma moulages were applied to the second digit of the left hand of standardized patients (SPs) participating in a wrist pain scenario during a required clinical skills examination. An observer reviewed videotapes of the examination, written SP checklists, and student notes for evidence that the student noticed the moulage, obtained a history, or provided counseling. RESULTS: Among the 190 fourth-year medical students, 56 students were observed noticing the lesion; however, 13 failed to write it in their notes or advise the patient. The detection rate was 22.6% (43 of 190 students). Students who detected the probable melanoma consistently inquired about changes in the lesion and symptoms, but did not examine the rest of the skin or regularly palpate for adenopathy. LIMITATIONS: Testing one class of students from a single medical school with a time-restricted SP encounter while focusing the students' attention toward a different presenting symptom may hinder exploration of medical issues. CONCLUSION: The low detection rate and failure of students who noticed the moulage to identify the lesion as atypical represents a lost opportunity to provide a patient intervention. Use of SP examinations may help physicians in training build confidence and competence in cutaneous malignancy screening.

Applying the Bookmark method to medical education: standard setting for an aseptic technique station.

INTRODUCTION: The purpose was to evaluate the Bookmark standard-setting method for use on a performance-based assessment in medical education. METHODS: We compared cutscores for Aseptic performance assessment using the modified Angoff, Hofstee and modified Bookmark methods. RESULTS: The Angoff produced a cutscore of 62%, SD=18 and a percent passing (pp)= 64%. The Hofstee cutscore was 71%, SD=7 and pp=46%. Bookmark mean cutscores were 65.9% SD=10.7 and pp=42% for advanced beginners; 83.6%, SD=9.2 and pp=17% for competent and the proficient category resulted in a cutscore of 96.4% SD=3.9 and pp=1%. Faculty judges found the Bookmark method to be an easy and acceptable method. CONCLUSIONS: The Bookmark method was acceptable to faculty, has reasonable quality metrics when compared to other methods and can be a practical tool for establishing standards in performance-based examinations. The Bookmark method could be useful for establishing multiple levels of competency using the Dreyfus criteria.

Development of a Rubric to Evaluate Implementation Quality of Simulation-Based Courses: A Consensus Study.

INTRODUCTION: Simulation-based education is a recognized way of developing medical competencies, and there is overwhelming scientific evidence to support its efficacy. However, it is still underused, which can often be related to poor implementation process. In addition, best practices for implementation of simulation-based courses based on implementation science are not widely known nor applied. The purpose of this study was to develop a rubric, the Implementation Quality Rubric for Simulation (IQR-SIM), to evaluate the implementation quality of simulation-based courses. METHODS: A 3-round, modified Delphi process involving international simulation and implementation experts was initiated to gather and converge opinions regarding criteria for evaluating the implementation quality of simulation-based courses. Candidate items for Round 1 were developed based on the Adapted Implementation Model for Simulation. Items were revised and expanded to include descriptive anchors for evaluation in Round 2. Criterion for inclusion was 70% of respondents selecting an importance rating of 4 or 5/5. Round 3 provided refinement and final approval of items and anchors. RESULTS: Thirty-three experts from 9 countries participated. The initial rubric of 32 items was reduced to 18 items after 3 Delphi rounds, resulting in the IQR-SIM: a 3-point rating scale, with nonscored options "Don't know/can't assess" and "Not applicable," and a comments section. CONCLUSIONS: The IQR-SIM is an operational tool that can be used to evaluate the implementation quality of simulation-based courses and aid in the implementation process to identify gaps, monitor the process, and promote the achievement of desired implementation and learning outcomes.

Telemedicine in Anesthesiology: Using Simulation to Teach Remote Preoperative Assessment.

Background: The move toward telemedicine has markedly accelerated with the COVID-19 pandemic. Anesthesia residents must learn to provide preoperative assessments on a virtual platform. We created a pilot telemedicine curriculum for postgraduate year-2 (PGY2) anesthesiology. Methods: The curriculum included a virtual didactic session and a simulated virtual preoperative assessment with a standardized patient (SP). A faculty member and the SP provided feedback using a checklist based on the American Medical Association Telehealth Visit Etiquette Checklist and the American Board of Anesthesiology Applied Examination Objective Structured Clinical Examination content outline. Residents completed surveys assessing their perceptions of the effectiveness and helpfulness of the didactic session and simulated encounter, as well as the cognitive workload of the encounter. Results: A total of 12 PGY2 anesthesiology residents in their first month of clinical anesthesia residency training participated in this study. Whereas most (11/12) residents felt confident, very confident, or extremely confident in being able to conduct a telemedicine preoperative assessment after the didactic session, only 42% ensured adequate lighting and only 33% ensured patient privacy before conducting the visit. Postencounter survey comments indicated that the SP encounter was of greater value (more effective and helpful) than the didactic session. Residents perceived the encounter as demanding, but they felt successful in accomplishing it and did not feel rushed. Faculty and SP indicated that the checklist guided them in providing clear and useful formative feedback. Conclusions: A virtual SP encounter can augment didactics to help residents learn and practice essential telemedicine skills for virtual preoperative assessments.

A Multimedia Evaluation of Pharmacy Faculty PowerPoint Slides in a Critical Care Course.

OBJECTIVES: To conduct a pilot investigation about the alignment between didactic multimedia materials utilized by pharmacy faculty, with Mayer's Principles for Multimedia Learning and faculty characteristics associated with greater alignment. METHODS: An investigatory systematic process was used which included a modified Learning Object Review Instrument (LORI) to evaluate the faculty video-recorded lectures for alignment with Mayer's Principles of Multimedia Learning, hence capturing the number and type of misalignments. Correlations were performed to evaluate the association between faculty characteristics; and ratings and proportions of misalignments. RESULTS: Five hundred fifty-five PowerPoint slides of 13 lectures from 13 faculty members were reviewed. The average (SD) LORI score per slide was 4.44 (0.84) out of 5 with an average score per lecture ranging from 3.83 (0.96) to 4.95 (0.53). Across all lecture slides, misalignments with multimedia principles were captured in 20.2% of slides. For each lecture, the average percentage of misalignments was 27.6% ranging from 0% to 49%. Principal misalignments included violation of the principles of coherence (66.1%), signaling (15.2%), and segmenting (8%). No faculty characteristics were significantly associated with LORI ratings or proportion of misalignments within lectures. CONCLUSIONS: Faculty had high LORI ratings for their multimedia material but these varied significantly between lectures. Misalignments with multimedia principles were identified and were related primarily to extraneous processing. These misalignments, when addressed, have the potential to improve learning, thus suggesting an opportunity for the faculty to develop ways to optimize multimedia educational delivery. Future investigation is needed to clarify how clinical pharmacy faculty can develop multimedia material and the impact of faculty development on the application of multimedia principles and learning outcomes.

Automated Patient Note Grading: Examining Scoring Reliability and Feasibility.

PURPOSE: Scoring postencounter patient notes (PNs) yields significant insights into student performance, but the resource intensity of scoring limits its use. Recent advances in natural language processing (NLP) and machine learning allow application of automated short answer grading (ASAG) for this task. This retrospective study evaluated psychometric characteristics and reliability of an ASAG system for PNs and factors contributing to implementation, including feasibility and case-specific phrase annotation required to tune the system for a new case. METHOD: PNs from standardized patient (SP) cases within a graduation competency exam were used to train the ASAG system, applying a feed-forward neural networks algorithm for scoring. Using faculty phrase-level annotation, 10 PNs per case were required to tune the ASAG system. After tuning, ASAG item-level ratings for 20 notes were compared across ASAG-faculty (4 cases, 80 pairings) and ASAG-nonfaculty (2 cases, 40 pairings). Psychometric characteristics were examined using item analysis and Cronbach's alpha. Inter-rater reliability (IRR) was examined using kappa. RESULTS: ASAG scores demonstrated sufficient variability in differentiating learner PN performance and high IRR between machine and human ratings. Across all items the ASAG-faculty scoring mean kappa was .83 (SE ± .02). The ASAG-nonfaculty pairings kappa was .83 (SE ± .02). The ASAG scoring demonstrated high item discrimination. Internal consistency reliability values at the case level ranged from a Cronbach's alpha of .65 to .77. Faculty time cost to train and supervise nonfaculty raters for 4 cases was approximately $1,856. Faculty cost to tune the ASAG system was approximately $928. CONCLUSIONS: NLP-based automated scoring of PNs demonstrated a high degree of reliability and psychometric confidence for use as learner feedback. The small number of phrase-level annotations required to tune the system to a new case enhances feasibility. ASAG-enabled PN scoring has broad implications for improving feedback in case-based learning contexts in medical education.

Optimizing Clinical Reasoning Assessments With Analytic and Holistic Ratings: Examining the Validity, Reliability, and Cost of a Simplified Patient Note Scoring Procedure.

PURPOSE: Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback. METHOD: Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time). RESULTS: Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores. CONCLUSIONS: Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams.

Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study.

OBJECTIVES: Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC. DESIGN: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions. SETTING: Academic surgical residency program. PARTICIPANTS: Forty-eight surgical residents (PGY-1 to PGY-5). RESULTS: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions. CONCLUSIONS: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.

Mastery skill assessment in hepato-pancreato-biliary surgical ultrasound: It's a Matter of Entrustment.

BACKGROUND: A formative hepato-pancreato-biliary (HPB) ultrasound (US) skills practicum is offered annually to graduating HPB fellows, using entrustment assessments for open (IOUS) and laparoscopic (LAPUS) US. It is hypothesized that validity evidence will support the use of these assessments to determine if graduating fellows are well prepared to perform HPB US independently. METHODS: Expert faculty were surveyed to set Mastery Entrustment standards for fellow performance. Standards were applied to fellow performances during two annual US skills practicums. RESULTS: 11 faculty questionnaires were included. Mean Entrustment cut scores across all items were 4.9/5.0 and 4.8/5.0 and Global Entrustment cut scores were 5.0/5.0 and 4.8/5.0 for IOUS and LAPUS, respectively. 78.5% (29/37) fellows agreed to have their de-identified data evaluated. Mean fellow Entrustments (across all skills) were 4.1 (SD 0.6; 2.6-4.9) and 3.9 (SD 0.7; 2.7-5), while the Global Entrustments were 3.6 (SD 0.8; 2-5) and 3.5 (SD 1.0; 2-5) for IOUS and LAPUS, respectively. CONCLUSIONS: Two cohorts of graduating HPB fellows are not meeting Mastery Standards for HPB US performance determined by a panel of expert faculty.