Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Cost analysis of the History, ECG, Age, Risk factors, and initial Troponin (HEART) Pathway randomized control trial.

INTRODUCTION: The HEART Pathway is a diagnostic protocol designed to identify low-risk patients presenting to the emergency department with chest pain that are safe for early discharge. This protocol has been shown to significantly decrease health care resource utilization compared with usual care. However, the impact of the HEART Pathway on the cost of care has yet to be reported. METHODS AND RESULTS: We performed a cost analysis of patients enrolled in the HEART Pathway trial, which randomized participants to either usual care or the HEART Pathway protocol. For low-risk patients, the HEART Pathway recommended early discharge from the emergency department without further testing. We compared index visit cost, cost at 30 days, and cardiac-related health care cost at 30 days between the 2 treatment arms. Costs for each patient included facility and professional costs. Cost at 30 days included total inpatient and outpatient costs, including the index encounter, regardless of etiology. Cardiac-related health care cost at 30 days included the index encounter and costs adjudicated to be cardiac-related within that period. Two hundred seventy of the 282 patients enrolled in the trial had cost data available for analysis. There was a significant reduction in cost for the HEART Pathway group at 30 days (median cost savings of $216 per individual), which was most evident in low-risk (Thrombolysis In Myocardial Infarction score of 0-1) patients (median savings of $253 per patient) and driven primarily by lower cardiac diagnostic costs in the HEART Pathway group. CONCLUSIONS: Using the HEART Pathway as a decision aid for patients with undifferentiated chest pain resulted in significant cost savings.

Validation of a performance checklist for ultrasound-guided internal jugular central lines for use in procedural instruction and assessment.

PURPOSE OF THE STUDY: Tools created to measure procedural competency must be tested in their intended environment against an established standard in order to be validated. We previously created a checklist for ultrasound-guided internal jugular central venous catheter (US IJ CVC) insertion using the modified Delphi method. We sought to further validate the checklist tool for use in an educational environment. STUDY DESIGN: This is a cohort study involving 15 emergency medicine interns being evaluated on their skill in US IJ CVC placement. We compared the checklist tool with a modified version of a clinically validated global rating scale (GRS) for procedural performance. RESULTS: The correlation between the GRS tool and the checklist tool was excellent, with a correlation coefficient (Pearson's r) of 0.90 (p<0.0001). CONCLUSIONS: This checklist represents a useful tool for measuring procedural competency.

Derivation of a Performance Checklist for Ultrasound-Guided Arthrocentesis Using the Modified Delphi Method.

OBJECTIVES: Arthrocentesis is an important skill for physicians in multiple specialties. Recent studies indicate a superior safety and performance profile for this procedure using ultrasound guidance for needle placement, and improving quality of care requires a valid measurement of competency using this modality. METHODS: We endeavored to create a validated tool to assess the performance of this procedure using the modified Delphi technique and experts in multiple disciplines across the United States. RESULTS: We derived a 22-item checklist designed to assess competency for the completion of ultrasound-guided arthrocentesis, which demonstrated a Cronbach's alpha of 0.89, indicating an excellent degree of internal consistency. CONCLUSIONS: Although we were able to demonstrate content validity for this tool, further validity evidence should be acquired after the tool is used and studied in clinical and simulated contexts.

Video-augmented feedback for procedural performance.

BACKGROUND: Resident programs must assess residents' achievement of core competencies for clinical and procedural skills. AIMS: Video-augmented feedback may facilitate procedural skill acquisition and promote more accurate self-assessment. METHODS: A randomized controlled study to investigate whether video-augmented verbal feedback leads to increased procedural skill and improved accuracy of self-assessment compared to verbal only feedback. Participants were evaluated during procedural training for ultrasound guided internal jugular central venous catheter (US IJ CVC) placement. All participants received feedback based on a validated 30-point checklist for US IJ CVC placement and validated 6-point procedural global rating scale. RESULTS: Scores in both groups improved by a mean of 9.6 points (95% CI: 7.8-11.4) on the 30-point checklist, with no difference between groups in mean score improvement on the global rating scale. In regards to self-assessment, participant self-rating diverged from faculty scoring, increasingly so after receiving feedback. Residents rated highly by faculty underestimated their skill, while those rated more poorly demonstrated increasing overestimation. CONCLUSIONS: Accuracy of self-assessment was not improved by addition of video. While feedback advanced the skill of the resident, video-augmented feedback did not enhance skill acquisition or improve accuracy of resident self-assessment compared to standard feedback.

North Carolina College of Emergency Physicians' Guidance Document on Emergency Medical Services.

The North Carolina College of Emergency Physicians (NCCEP) Emergency Medical Services (EMS) Committee uses an evidence-based approach in writing its protocols and procedures. The most recent revision of the NCCEP document, which was started in late 2010, lasted for more than 1 year and utilized committee members from across the state. Four meetings were held at locations across North Carolina. In addition, 2 surveys were sent to get input from EMS providers. Since 2010, the document has been updated on an ongoing basis, aligning it with the latest evidence-based medicine.

Implications of 25% to 50% coronary stenosis with cardiac computed tomographic angiography in ED patients.

OBJECTIVE: The aim of this study was to determine if patients presenting with symptoms of acute coronary syndrome and found to have 25% to 50% diameter reduction with coronary computed tomographic angiography (CCTA) are likely to benefit from further diagnostic testing. METHODS: A registry study of 213 subjects (median age, 51 years; 53% women) with symptoms concerning for possible acute coronary syndrome with low-risk features found to have 25% to 50% maximal diameter stenosis on CCTA was performed at 2 academic medical centers. The analysis was approved by an institutional review board and was conducted with waiver of consent. The potential contribution of additional testing was determined by measuring the major adverse cardiac events (MACEs) from presentation through 30 days. The MACEs included myocardial infarction, coronary revascularization, unstable angina, and cardiovascular death. Sample size calculations were predicated on a 0% MACE rate leading to upper bounds of a 2-sided exact 95% confidence interval less than 2%. RESULTS: Thrombolysis in myocardial infarction risk score of less than 2 was present in 92% subjects, 70% (150 of 213) had 2 or more serial cardiac markers performed, and 40% (87 of 213) had stress testing or cardiac catheterization. The MACEs occurred in 1 (0.5%) of 213 subjects (95% confidence interval, 0%-2.6%) and was identified by an elevation of serial cardiac markers during the index hospitalization. No patients experienced cardiovascular death or required revascularization. CONCLUSIONS: In patients with emergent low-risk chest pain and 25% to 50% diameter coronary stenosis by CCTA, the rate of near-term MACE is very low. Serial cardiac markers may be beneficial in this subgroup. Routine provocative testing is unlikely to be beneficial during the index visit.

The Impact of Shortened Clinical Clerkships on Medical Student Performance and Clerkship Assessment.

INTRO: Medical schools sometimes need to adjust the length of third-year clinical clerkships. The literature surrounding the effects of shortened clerkships on student experience and performance is mixed. METHODS: Our medical school shortened the third year by an average of 20% per clerkship to accommodate a curricular re-design in 2018-2019. We examined test scores and measures of clinical performance as well as student experience in order to understand the impact of this change. RESULTS: Two hundred and eight students were included in the analysis, 104 in each cohort. No statistically significant differences were noted between cohorts on NBME subject examination results. There were no significant differences on Step 2 CK scores between the traditional curriculum cohort (M = 249.4, SD = 13.7) and shortened curriculum cohort (M = 248.7, SD = 15.8). Student performance on OSCE cases was similar. Similar percentages of students rated each clerkship either "good" or "excellent" in the traditional (77%) and shortened (78%) curriculum. CONCLUSION: There was no significant impact on student test scores after shortening the curriculum. Measures of student satisfaction and experience also remained stable, likely related to emphasis on retaining patient care experiences and streamlining of didactics. Curricular shortening during the third year of medical school was feasible and safe from the student perspective in our experience.

Effect of Perceived Level of Interaction on Faculty Evaluations of 3rd Year Medical Students.

INTRODUCTION: Several factors are known to affect the way clinical performance evaluations (CPEs) of medical students are completed by supervising physicians. We sought to explore the effect of faculty perceived "level of interaction" (LOI) on these evaluations. METHODS: Our third-year CPE requires evaluators to identify perceived LOI with each student as low, moderate, or high. We examined CPEs completed during the academic year 2018-2019 for differences in (1) clinical and professionalism ratings, (2) quality of narrative comments, (3) quantity of narrative comments, and (4) percentage of evaluation questions left unrated. RESULTS: A total of 3682 CPEs were included in the analysis. ANOVA revealed statistically significant differences between LOI and clinical ratings (p ≤ .001), with mean ratings from faculty with a high LOI significantly higher than from faculty with a moderate or low LOI (p ≤ .001). Chi-squared analysis demonstrated differences based on faculty LOI and whether questions were left unrated (p ≤ .001), quantity of narrative comments (p ≤ .001), and specificity of narrative comments (p ≤ .001). CONCLUSIONS: Faculty who perceive higher LOI were more likely to assign that student higher ratings, complete more of the clinical evaluation and were more likely to provide narrative feedback with more specific, higher-quality comments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01307-w.

Practical Application of Assessment Principles in the Development of an Innovative Clinical Performance Evaluation in the Entrustable Professional Activity Era.

Educators have been challenged to create assessments that are competency-based and grounded in accepted standards such as the entrustable professional activities (EPAs). The clinical performance evaluation (CPE) is a commonly utilized assessment modality, which allows multiple evaluators to provide feedback on a learner's performance in the clinical workplace. In this paper, we describe the relevant principles that served as a guide as we developed a new CPE for medical students that fully incorporate EPAs. This may help ease the transition for other institutions looking to introduce a new student CPE.

Does Clerkship Rotation Sequence Affect Performance on National Board of Medical Examiners (NBME) Clinical Subject Examinations and United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) Examination?

PURPOSE: To investigate whether starting the clerkship year in family medicine (FM), internal medicine (IM), pediatrics, or surgery influences NBME shelf and USMLE Step 2 CK examination performance. METHODS: USMLE Step 1, Step 2 CK, and shelf examination scores for FM, IM, pediatrics, and surgery were collected. Sequences were selected on the following assignment criteria: rotation 1 (either FM or IM), rotation 5 (pediatrics), rotation 8 (surgery), rotation 1 (either pediatrics or surgery), and rotation 8 (IM). Multivariate analysis of covariance, with Step 1 as the covariate, was used to investigate rotation sequence on examination performance. RESULTS: Wilks's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in FM or IM) on the pediatrics, surgery, and Step 2 CK examinations (Λ = .95, F[3,51] = .93, p ≤ .432). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .488, F[3,51] = 17.827, p ≤ .001), indicating the two groups differ on Step 1 performance. Wilk's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in pediatrics or surgery) on the IM and Step 2 CK results (Λ = .925, F[2,75] = 3.036, p ≤ .054). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .309, F[2,75] = 83.915, p ≤ .001) indicating that the two groups differ on Step 1 performance. CONCLUSION: Starting the clerkship year in FM, IM, pediatrics, or surgery does not influence subsequent performance on shelf examinations or on Step 2 CK.

The HEART Pathway Randomized Controlled Trial One-year Outcomes.

OBJECTIVE: The objective was to determine the impact of the HEART Pathway on health care utilization and safety outcomes at 1 year in patients with acute chest pain. METHODS: Adult emergency department (ED) patients with chest pain (N = 282) were randomized to the HEART Pathway or usual care. In the HEART Pathway arm, ED providers used the HEART score and troponin measures (0 and 3 hours) to risk stratify patients. Usual care was based on American College of Cardiology/American Heart Association guidelines. Major adverse cardiac events (MACE-cardiac death, myocardial infarction [MI], or coronary revascularization), objective testing (stress testing or coronary angiography), and cardiac hospitalizations and ED visits were assessed at 1 year. Randomization arm outcomes were compared using Fisher's exact tests. RESULTS: A total of 282 patients were enrolled, with 141 randomized to each arm. MACE at 1 year occurred in 10.6% (30/282): 9.9% in the HEART Pathway arm (14/141; 10 MIs, four revascularizations without MI) versus 11.3% in usual care (16/141; one cardiac death, 13 MIs, two revascularizations without MI; p = 0.85). Among low-risk HEART Pathway patients, 0% (0/66) had MACE, with a negative predictive value (NPV) of 100% (95% confidence interval = 93%-100%). Objective testing through 1 year occurred in 63.1% (89/141) of HEART Pathway patients compared to 71.6% (101/141) in usual care (p = 0.16). Nonindex cardiac-related hospitalizations and ED visits occurred in 14.9% (21/141) and 21.3% (30/141) of patients in the HEART Pathway versus 10.6% (15/141) and 16.3% (23/141) in usual care (p = 0.37, p = 0.36). CONCLUSIONS: The HEART Pathway had a 100% NPV for 1-year safety outcomes (MACE) without increasing downstream hospitalizations or ED visits. Reduction in 1-year objective testing was not significant.

ENPP1-Fc prevents neointima formation in generalized arterial calcification of infancy through the generation of AMP.

Generalized arterial calcification of infancy (GACI) is associated with widespread arterial calcification and stenoses and is caused by mutations in ENPP1. ENPP1 encodes for ectonucleotide pyrophosphatase/phosphodiesterase 1 (ENPP1), which cleaves ATP to generate inorganic pyrophosphate (PPi) and adenosine monophosphate (AMP) extracellularly. The current study was designed to define the prevalence of arterial stenoses in GACI individuals and to identify the mechanism through which ENPP1 deficiency causes intimal proliferation. Furthermore, we aimed to effectively prevent and treat neointima formation in an animal model of GACI through the systemic administration of recombinant human (rh)ENPP1-Fc protein. Based on a literature review, we report that arterial stenoses are present in at least 72.4% of GACI cases. We evaluated the effect of rhENPP1-Fc on ENPP1-silenced human vascular smooth muscle cells (VSMCs) and on induced intimal proliferation in Enpp1-deficient ttw/ttw mice treated with carotid ligation. We demonstrate that silencing ENPP1 in VSMCs resulted in a tenfold increase in proliferation relative to that of cells transfected with negative control siRNA. The addition of rhENPP1-Fc, AMP or adenosine restored the silenced ENPP1-associated proliferation. In contrast, neither PPi nor etidronate, a current off-label treatment for GACI, had an effect on VSMC proliferation. Furthermore, subcutaneous rhENPP1-Fc protein replacement was effective in preventing and treating intimal hyperplasia induced by carotid ligation in an animal model of GACI. We conclude that ENPP1 inhibits neointima formation by generating  AMP. RhENPP1-Fc may serve as an approach for the effective prevention and treatment of arterial stenoses in GACI.

3 for the Price of 1: Teaching Chest Pain Risk Stratification in a Multidisciplinary, Problem-based Learning Workshop.

INTRODUCTION: Chest pain is a common chief complaint among patients presenting to health systems and often leads to complex and intensive evaluations. While these patients are often cared for by a multidisciplinary team (primary care, emergency medicine, and cardiology), medical students usually learn about the care of these patients in a fragmented, single-specialty paradigm. The present and future care of patients with chest pain is multidisciplinary, and the education of medical students on the subject should be as well. Our objective was to evaluate the effectiveness of a multidisciplinary, problem-based learning workshop to teach third-year medical students about risk assessment for patients presenting with chest pain, specifically focusing on acute coronary syndromes. METHODS: To create an educational experience consistent with multidisciplinary team-based care, we designed a multidisciplinary, problem-based learning workshop to provide medical students with an understanding of how patients with chest pain are cared for in a systems-based manner to improve outcomes. Participants included third-year medical students (n=219) at a single, tertiary care, academic medical center. Knowledge acquisition was tested in a pre-/post-retention test study design. RESULTS: Following the workshop, students achieved a 19.7% (95% confidence interval [CI] [17.3-22.2%]) absolute increase in scores on post-testing as compared to pre-testing. In addition, students maintained an 11.1% (95% CI [7.2-15.0%]) increase on a retention test vs. the pre-test. CONCLUSION: A multidisciplinary, problem-based learning workshop is an effective method of producing lasting gains in student knowledge about chest pain risk stratification.

Planning considerations for persons with access and functional needs in a disaster-Part 1: Overview and legal.

Although disasters can affect anyone in a given community or region, those with access and functional needs have the highest rates of morbidity and mortality during an emergency or disaster. There are many unique and complex issues that should be considered when planning and caring for these individuals in a disaster situation. Who are these individuals, what specific issues should be addressed when considering these members of the population, and what recommendations can be made to address their unique needs? How can we include them as part of the all-hazard, comprehensive approach to disaster management? The first part of this three-part series identifies who is included in this population and what are the legal considerations that arise in caring for, not only this unique group but also all the members of the community in a disaster. The second part considers evacuation, sheltering, sheltering in place, communication, medical needs, independence, supervision, and transportation (CMIST) with a focus on mental health. The third part deals with the medical aspect of CMIST and with recommendations that may aid disaster responders and planners in caring for these high-risk individuals in a disaster.

ENPP1 enzyme replacement therapy improves blood pressure and cardiovascular function in a mouse model of generalized arterial calcification of infancy.

Generalized arterial calcification of infancy (GACI) is a rare, life-threatening disorder caused by loss-of-function mutations in the gene encoding ectonucleotide pyrophosphatase phosphodiesterase 1 (ENPP1), which normally hydrolyzes extracellular ATP into AMP and pyrophosphate (PPi). The disease is characterized by extensive arterial calcification and stenosis of large- and medium-sized vessels, leading to vascular-related complications of hypertension and heart failure. There is currently no effective treatment available, but bisphosphonates - nonhydrolyzable PPi analogs - are being used off-label to reduce arterial calcification, although this has no reported impact on the hypertension and cardiac dysfunction features of GACI. In this study, the efficacy of a recombinant human ENPP1 protein therapeutic (rhENPP1) was tested in Enpp1asj-2J homozygous mice (Asj-2J or Asj-2J hom), a model previously described to show extensive mineralization in the arterial vasculature, similar to GACI patients. In a disease prevention study, Asj-2J mice treated with rhENPP1 for 3 weeks showed >95% reduction in aorta calcification. Terminal hemodynamics and echocardiography imaging of Asj-2J mice also revealed that a 6-week rhENPP1 treatment normalized elevated arterial and left ventricular pressure, which translated into significant improvements in myocardial compliance, contractility, heart workload and global cardiovascular efficiency. This study suggests that ENPP1 enzyme replacement therapy could be a more effective GACI therapeutic than bisphosphonates, treating not just the vascular calcification, but also the hypertension that eventually leads to cardiac failure in GACI patients.

Planning considerations for persons with access and functional needs in a disaster-Part 2: Evacuation and sheltering.

Although disasters can affect anyone in a given community or region, those with access and functional needs have the highest rates of morbidity and mortality during an emergency or disaster. There are many unique and complex issues that should be considered when dealing with these individuals in a disaster situation. Who are these individuals, what specific issues should be addressed, and what recommendations can be made in order to address their unique needs? How can we include them as part of the all-hazard, comprehensive approach to disaster management? The first part of this three-part series identifies who is included in this population and what are the legal considerations that arise in caring for not only this unique group but also all of the members of the community in a disaster. The second part considers evacuation, sheltering, sheltering in place, communication, medical needs, independence, supervision, and transportation (CMIST) with a focus on mental health. The third part deals with the medical aspect of CMIST and with recommendations that may aid disaster responders and planners in caring for these high-risk individuals in a disaster.

Planning considerations for persons with access and functional needs in a disaster-Part 3: Medical CMIST and recommendations.

Although disasters can affect anyone in a given community or region, those with access and functional needs have the highest rates of morbidity and mortality during an emergency or disaster. There are many unique and complex issues that should be considered when dealing with these individuals in a disaster situation. Who are these individuals, what specific issues should be addressed when considering these members of the population, and what recommendations can be made in order to address their unique needs? How can we include them as part of the all-hazard, comprehensive approach to disaster management? The first part of this three-part series identifies who is included in this population and what are the legal considerations that arise in caring for not only this unique group but also all of the members of the community in a disaster. The second part considers evacuation, sheltering, sheltering in place, communication, medical needs, independence, supervision, and transportation (CMIST) with a focus on mental health. The third part deals with the medical aspect of CMIST and with recommendations that may aid disaster responders and planners in caring for these high-risk individuals in a disaster.

Educational priorities of students in the entrustable professional activity era.

BACKGROUND: The Association of American Medical Colleges (AAMC) guidelines on the entrustable professional activities (EPAs) expected of graduating medical students were recently published. Although perceptions of educators, residents and programme directors have been described, the voice of senior medical students is lacking. METHODS: A single-institution cross-sectional study of senior medical students was performed. Student perceptions were collected and compared with: (1) national guidelines (i.e. the 13 newly developed undergraduate EPAs); (2) resident expectations (i.e. through comparison with a recently published survey from >28 000 residents); and (3) institutional objectives. Descriptive statistics were performed. RESULTS: A total of 113 students participated. The top three EPA-based educational priorities were 'recognising a patient requiring urgent/emergent care' (EPA10), 'performing procedures of a physician' (EPA12) and 'collaborating as an interprofessional' (EPA9). Over 80 per cent of students rated 'managing time efficiently' and 'communicating around care transitions' as very important pre-internship skills. Of the institutional objectives, 87 per cent rated 'recognising critically ill patients' and 'knowing when to ask for help' as the most important pre-internship skills. The voice of senior medical students is lacking CONCLUSIONS: Although the emphasis on knowing when to ask for help and communication around care transitions differed somewhat across stakeholders, educational priorities were shared by students, residents, educators and institutional objectives. These preliminary data support national assessments of perceptions and achievements of senior medical students to guide residency readiness in the EPA era.

A Multidisciplinary Self-Directed Learning Module Improves Knowledge of a Quality Improvement Instrument: The HEART Pathway.

We created and tested an educational intervention to support implementation of an institution wide QI project (the HEART Pathway) designed to improve care for patients with acute chest pain. Although online learning modules have been shown effective in imparting knowledge regarding QI projects, it is unknown whether these modules are effective across specialties and healthcare professions. Participants, including nurses, advanced practice clinicians, house staff and attending physicians (N = 486), were enrolled into an online, self-directed learning course exploring the key concepts of the HEART Pathway. The module was completed by 97% of enrollees (469/486) and 90% passed on the first attempt (422/469). Out of 469 learners, 323 completed the pretest, learning module and posttest in the correct order. Mean test scores across learners improved significantly from 74% to 89% from the pretest to the posttest. Following the intervention, the HEART Pathway was used for 88% of patients presenting to our institution with acute chest pain. Our data demonstrate that this online, self-directed learning module can improve knowledge of the HEART Pathway across specialties-paving the way for more efficient and informed care for acute chest pain patients.