Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Effect of Perceived Level of Interaction on Faculty Evaluations of 3rd Year Medical Students.

INTRODUCTION: Several factors are known to affect the way clinical performance evaluations (CPEs) of medical students are completed by supervising physicians. We sought to explore the effect of faculty perceived "level of interaction" (LOI) on these evaluations. METHODS: Our third-year CPE requires evaluators to identify perceived LOI with each student as low, moderate, or high. We examined CPEs completed during the academic year 2018-2019 for differences in (1) clinical and professionalism ratings, (2) quality of narrative comments, (3) quantity of narrative comments, and (4) percentage of evaluation questions left unrated. RESULTS: A total of 3682 CPEs were included in the analysis. ANOVA revealed statistically significant differences between LOI and clinical ratings (p ≤ .001), with mean ratings from faculty with a high LOI significantly higher than from faculty with a moderate or low LOI (p ≤ .001). Chi-squared analysis demonstrated differences based on faculty LOI and whether questions were left unrated (p ≤ .001), quantity of narrative comments (p ≤ .001), and specificity of narrative comments (p ≤ .001). CONCLUSIONS: Faculty who perceive higher LOI were more likely to assign that student higher ratings, complete more of the clinical evaluation and were more likely to provide narrative feedback with more specific, higher-quality comments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01307-w.

The Impact of Shortened Clinical Clerkships on Medical Student Performance and Clerkship Assessment.

INTRO: Medical schools sometimes need to adjust the length of third-year clinical clerkships. The literature surrounding the effects of shortened clerkships on student experience and performance is mixed. METHODS: Our medical school shortened the third year by an average of 20% per clerkship to accommodate a curricular re-design in 2018-2019. We examined test scores and measures of clinical performance as well as student experience in order to understand the impact of this change. RESULTS: Two hundred and eight students were included in the analysis, 104 in each cohort. No statistically significant differences were noted between cohorts on NBME subject examination results. There were no significant differences on Step 2 CK scores between the traditional curriculum cohort (M = 249.4, SD = 13.7) and shortened curriculum cohort (M = 248.7, SD = 15.8). Student performance on OSCE cases was similar. Similar percentages of students rated each clerkship either "good" or "excellent" in the traditional (77%) and shortened (78%) curriculum. CONCLUSION: There was no significant impact on student test scores after shortening the curriculum. Measures of student satisfaction and experience also remained stable, likely related to emphasis on retaining patient care experiences and streamlining of didactics. Curricular shortening during the third year of medical school was feasible and safe from the student perspective in our experience.

Practical Application of Assessment Principles in the Development of an Innovative Clinical Performance Evaluation in the Entrustable Professional Activity Era.

Educators have been challenged to create assessments that are competency-based and grounded in accepted standards such as the entrustable professional activities (EPAs). The clinical performance evaluation (CPE) is a commonly utilized assessment modality, which allows multiple evaluators to provide feedback on a learner's performance in the clinical workplace. In this paper, we describe the relevant principles that served as a guide as we developed a new CPE for medical students that fully incorporate EPAs. This may help ease the transition for other institutions looking to introduce a new student CPE.

The HEART Pathway Randomized Controlled Trial One-year Outcomes.

OBJECTIVE: The objective was to determine the impact of the HEART Pathway on health care utilization and safety outcomes at 1 year in patients with acute chest pain. METHODS: Adult emergency department (ED) patients with chest pain (N = 282) were randomized to the HEART Pathway or usual care. In the HEART Pathway arm, ED providers used the HEART score and troponin measures (0 and 3 hours) to risk stratify patients. Usual care was based on American College of Cardiology/American Heart Association guidelines. Major adverse cardiac events (MACE-cardiac death, myocardial infarction [MI], or coronary revascularization), objective testing (stress testing or coronary angiography), and cardiac hospitalizations and ED visits were assessed at 1 year. Randomization arm outcomes were compared using Fisher's exact tests. RESULTS: A total of 282 patients were enrolled, with 141 randomized to each arm. MACE at 1 year occurred in 10.6% (30/282): 9.9% in the HEART Pathway arm (14/141; 10 MIs, four revascularizations without MI) versus 11.3% in usual care (16/141; one cardiac death, 13 MIs, two revascularizations without MI; p = 0.85). Among low-risk HEART Pathway patients, 0% (0/66) had MACE, with a negative predictive value (NPV) of 100% (95% confidence interval = 93%-100%). Objective testing through 1 year occurred in 63.1% (89/141) of HEART Pathway patients compared to 71.6% (101/141) in usual care (p = 0.16). Nonindex cardiac-related hospitalizations and ED visits occurred in 14.9% (21/141) and 21.3% (30/141) of patients in the HEART Pathway versus 10.6% (15/141) and 16.3% (23/141) in usual care (p = 0.37, p = 0.36). CONCLUSIONS: The HEART Pathway had a 100% NPV for 1-year safety outcomes (MACE) without increasing downstream hospitalizations or ED visits. Reduction in 1-year objective testing was not significant.

Does Clerkship Rotation Sequence Affect Performance on National Board of Medical Examiners (NBME) Clinical Subject Examinations and United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) Examination?

PURPOSE: To investigate whether starting the clerkship year in family medicine (FM), internal medicine (IM), pediatrics, or surgery influences NBME shelf and USMLE Step 2 CK examination performance. METHODS: USMLE Step 1, Step 2 CK, and shelf examination scores for FM, IM, pediatrics, and surgery were collected. Sequences were selected on the following assignment criteria: rotation 1 (either FM or IM), rotation 5 (pediatrics), rotation 8 (surgery), rotation 1 (either pediatrics or surgery), and rotation 8 (IM). Multivariate analysis of covariance, with Step 1 as the covariate, was used to investigate rotation sequence on examination performance. RESULTS: Wilks's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in FM or IM) on the pediatrics, surgery, and Step 2 CK examinations (Λ = .95, F[3,51] = .93, p ≤ .432). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .488, F[3,51] = 17.827, p ≤ .001), indicating the two groups differ on Step 1 performance. Wilk's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in pediatrics or surgery) on the IM and Step 2 CK results (Λ = .925, F[2,75] = 3.036, p ≤ .054). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .309, F[2,75] = 83.915, p ≤ .001) indicating that the two groups differ on Step 1 performance. CONCLUSION: Starting the clerkship year in FM, IM, pediatrics, or surgery does not influence subsequent performance on shelf examinations or on Step 2 CK.

ENPP1-Fc prevents neointima formation in generalized arterial calcification of infancy through the generation of AMP.

Generalized arterial calcification of infancy (GACI) is associated with widespread arterial calcification and stenoses and is caused by mutations in ENPP1. ENPP1 encodes for ectonucleotide pyrophosphatase/phosphodiesterase 1 (ENPP1), which cleaves ATP to generate inorganic pyrophosphate (PPi) and adenosine monophosphate (AMP) extracellularly. The current study was designed to define the prevalence of arterial stenoses in GACI individuals and to identify the mechanism through which ENPP1 deficiency causes intimal proliferation. Furthermore, we aimed to effectively prevent and treat neointima formation in an animal model of GACI through the systemic administration of recombinant human (rh)ENPP1-Fc protein. Based on a literature review, we report that arterial stenoses are present in at least 72.4% of GACI cases. We evaluated the effect of rhENPP1-Fc on ENPP1-silenced human vascular smooth muscle cells (VSMCs) and on induced intimal proliferation in Enpp1-deficient ttw/ttw mice treated with carotid ligation. We demonstrate that silencing ENPP1 in VSMCs resulted in a tenfold increase in proliferation relative to that of cells transfected with negative control siRNA. The addition of rhENPP1-Fc, AMP or adenosine restored the silenced ENPP1-associated proliferation. In contrast, neither PPi nor etidronate, a current off-label treatment for GACI, had an effect on VSMC proliferation. Furthermore, subcutaneous rhENPP1-Fc protein replacement was effective in preventing and treating intimal hyperplasia induced by carotid ligation in an animal model of GACI. We conclude that ENPP1 inhibits neointima formation by generating  AMP. RhENPP1-Fc may serve as an approach for the effective prevention and treatment of arterial stenoses in GACI.

Planning considerations for persons with access and functional needs in a disaster-Part 1: Overview and legal.

Although disasters can affect anyone in a given community or region, those with access and functional needs have the highest rates of morbidity and mortality during an emergency or disaster. There are many unique and complex issues that should be considered when planning and caring for these individuals in a disaster situation. Who are these individuals, what specific issues should be addressed when considering these members of the population, and what recommendations can be made to address their unique needs? How can we include them as part of the all-hazard, comprehensive approach to disaster management? The first part of this three-part series identifies who is included in this population and what are the legal considerations that arise in caring for, not only this unique group but also all the members of the community in a disaster. The second part considers evacuation, sheltering, sheltering in place, communication, medical needs, independence, supervision, and transportation (CMIST) with a focus on mental health. The third part deals with the medical aspect of CMIST and with recommendations that may aid disaster responders and planners in caring for these high-risk individuals in a disaster.

ENPP1 enzyme replacement therapy improves blood pressure and cardiovascular function in a mouse model of generalized arterial calcification of infancy.

Generalized arterial calcification of infancy (GACI) is a rare, life-threatening disorder caused by loss-of-function mutations in the gene encoding ectonucleotide pyrophosphatase phosphodiesterase 1 (ENPP1), which normally hydrolyzes extracellular ATP into AMP and pyrophosphate (PPi). The disease is characterized by extensive arterial calcification and stenosis of large- and medium-sized vessels, leading to vascular-related complications of hypertension and heart failure. There is currently no effective treatment available, but bisphosphonates - nonhydrolyzable PPi analogs - are being used off-label to reduce arterial calcification, although this has no reported impact on the hypertension and cardiac dysfunction features of GACI. In this study, the efficacy of a recombinant human ENPP1 protein therapeutic (rhENPP1) was tested in Enpp1asj-2J homozygous mice (Asj-2J or Asj-2J hom), a model previously described to show extensive mineralization in the arterial vasculature, similar to GACI patients. In a disease prevention study, Asj-2J mice treated with rhENPP1 for 3 weeks showed >95% reduction in aorta calcification. Terminal hemodynamics and echocardiography imaging of Asj-2J mice also revealed that a 6-week rhENPP1 treatment normalized elevated arterial and left ventricular pressure, which translated into significant improvements in myocardial compliance, contractility, heart workload and global cardiovascular efficiency. This study suggests that ENPP1 enzyme replacement therapy could be a more effective GACI therapeutic than bisphosphonates, treating not just the vascular calcification, but also the hypertension that eventually leads to cardiac failure in GACI patients.

3 for the Price of 1: Teaching Chest Pain Risk Stratification in a Multidisciplinary, Problem-based Learning Workshop.

INTRODUCTION: Chest pain is a common chief complaint among patients presenting to health systems and often leads to complex and intensive evaluations. While these patients are often cared for by a multidisciplinary team (primary care, emergency medicine, and cardiology), medical students usually learn about the care of these patients in a fragmented, single-specialty paradigm. The present and future care of patients with chest pain is multidisciplinary, and the education of medical students on the subject should be as well. Our objective was to evaluate the effectiveness of a multidisciplinary, problem-based learning workshop to teach third-year medical students about risk assessment for patients presenting with chest pain, specifically focusing on acute coronary syndromes. METHODS: To create an educational experience consistent with multidisciplinary team-based care, we designed a multidisciplinary, problem-based learning workshop to provide medical students with an understanding of how patients with chest pain are cared for in a systems-based manner to improve outcomes. Participants included third-year medical students (n=219) at a single, tertiary care, academic medical center. Knowledge acquisition was tested in a pre-/post-retention test study design. RESULTS: Following the workshop, students achieved a 19.7% (95% confidence interval [CI] [17.3-22.2%]) absolute increase in scores on post-testing as compared to pre-testing. In addition, students maintained an 11.1% (95% CI [7.2-15.0%]) increase on a retention test vs. the pre-test. CONCLUSION: A multidisciplinary, problem-based learning workshop is an effective method of producing lasting gains in student knowledge about chest pain risk stratification.

RNaseT2 knockout rats exhibit hippocampal neuropathology and deficits in memory.

RNASET2 deficiency in humans is associated with infant cystic leukoencephalopathy, which causes psychomotor impairment, spasticity and epilepsy. A zebrafish mutant model suggests that loss of RNASET2 function leads to neurodegeneration due to the accumulation of non-degraded RNA in the lysosomes. The goal of this study was to characterize the first rodent model of RNASET2 deficiency. The brains of 3- and 12-month-old RNaseT2 knockout rats were studied using multiple magnetic resonance imaging modalities and behavioral tests. While T1- and T2-weighted images of RNaseT2 knockout rats exhibited no evidence of cystic lesions, the prefrontal cortex and hippocampal complex were enlarged in knockout animals. Diffusion-weighted imaging showed altered anisotropy and putative gray matter changes in the hippocampal complex of the RNaseT2 knockout rats. Immunohistochemistry for glial fibrillary acidic protein (GFAP) showed the presence of hippocampal neuroinflammation. Decreased levels of lysosome-associated membrane protein 2 (LAMP2) and elevated acid phosphatase and ß-N-acetylglucosaminidase (NAG) activities indicated that the RNASET2 knockout rats likely had altered lysosomal function and potential defects in autophagy. Object recognition tests confirmed that RNaseT2 knockout rats exhibited memory deficits. However, the Barnes maze, and balance beam and rotarod tests indicated there were no differences in spatial memory or motor impairments, respectively. Overall, patients with RNASET2 deficiency exhibited a more severe neurodegeneration phenotype than was observed in the RNaseT2 knockout rats. However, the vulnerability of the knockout rat hippocampus as evidenced by neuroinflammation, altered lysosomal function and cognitive defects indicates that this is still a useful in vivo model to study RNASET2 function.

Planning considerations for persons with access and functional needs in a disaster-Part 2: Evacuation and sheltering.

Although disasters can affect anyone in a given community or region, those with access and functional needs have the highest rates of morbidity and mortality during an emergency or disaster. There are many unique and complex issues that should be considered when dealing with these individuals in a disaster situation. Who are these individuals, what specific issues should be addressed, and what recommendations can be made in order to address their unique needs? How can we include them as part of the all-hazard, comprehensive approach to disaster management? The first part of this three-part series identifies who is included in this population and what are the legal considerations that arise in caring for not only this unique group but also all of the members of the community in a disaster. The second part considers evacuation, sheltering, sheltering in place, communication, medical needs, independence, supervision, and transportation (CMIST) with a focus on mental health. The third part deals with the medical aspect of CMIST and with recommendations that may aid disaster responders and planners in caring for these high-risk individuals in a disaster.

Planning considerations for persons with access and functional needs in a disaster-Part 3: Medical CMIST and recommendations.

Although disasters can affect anyone in a given community or region, those with access and functional needs have the highest rates of morbidity and mortality during an emergency or disaster. There are many unique and complex issues that should be considered when dealing with these individuals in a disaster situation. Who are these individuals, what specific issues should be addressed when considering these members of the population, and what recommendations can be made in order to address their unique needs? How can we include them as part of the all-hazard, comprehensive approach to disaster management? The first part of this three-part series identifies who is included in this population and what are the legal considerations that arise in caring for not only this unique group but also all of the members of the community in a disaster. The second part considers evacuation, sheltering, sheltering in place, communication, medical needs, independence, supervision, and transportation (CMIST) with a focus on mental health. The third part deals with the medical aspect of CMIST and with recommendations that may aid disaster responders and planners in caring for these high-risk individuals in a disaster.

A Multidisciplinary Self-Directed Learning Module Improves Knowledge of a Quality Improvement Instrument: The HEART Pathway.

We created and tested an educational intervention to support implementation of an institution wide QI project (the HEART Pathway) designed to improve care for patients with acute chest pain. Although online learning modules have been shown effective in imparting knowledge regarding QI projects, it is unknown whether these modules are effective across specialties and healthcare professions. Participants, including nurses, advanced practice clinicians, house staff and attending physicians (N = 486), were enrolled into an online, self-directed learning course exploring the key concepts of the HEART Pathway. The module was completed by 97% of enrollees (469/486) and 90% passed on the first attempt (422/469). Out of 469 learners, 323 completed the pretest, learning module and posttest in the correct order. Mean test scores across learners improved significantly from 74% to 89% from the pretest to the posttest. Following the intervention, the HEART Pathway was used for 88% of patients presenting to our institution with acute chest pain. Our data demonstrate that this online, self-directed learning module can improve knowledge of the HEART Pathway across specialties-paving the way for more efficient and informed care for acute chest pain patients.

Educational priorities of students in the entrustable professional activity era.

BACKGROUND: The Association of American Medical Colleges (AAMC) guidelines on the entrustable professional activities (EPAs) expected of graduating medical students were recently published. Although perceptions of educators, residents and programme directors have been described, the voice of senior medical students is lacking. METHODS: A single-institution cross-sectional study of senior medical students was performed. Student perceptions were collected and compared with: (1) national guidelines (i.e. the 13 newly developed undergraduate EPAs); (2) resident expectations (i.e. through comparison with a recently published survey from >28 000 residents); and (3) institutional objectives. Descriptive statistics were performed. RESULTS: A total of 113 students participated. The top three EPA-based educational priorities were 'recognising a patient requiring urgent/emergent care' (EPA10), 'performing procedures of a physician' (EPA12) and 'collaborating as an interprofessional' (EPA9). Over 80 per cent of students rated 'managing time efficiently' and 'communicating around care transitions' as very important pre-internship skills. Of the institutional objectives, 87 per cent rated 'recognising critically ill patients' and 'knowing when to ask for help' as the most important pre-internship skills. The voice of senior medical students is lacking CONCLUSIONS: Although the emphasis on knowing when to ask for help and communication around care transitions differed somewhat across stakeholders, educational priorities were shared by students, residents, educators and institutional objectives. These preliminary data support national assessments of perceptions and achievements of senior medical students to guide residency readiness in the EPA era.

Validation of the No Objective Testing Rule and Comparison to the HEART Pathway.

BACKGROUND: The no objective testing rule (NOTR) is a decision aid designed to safely identify emergency department (ED) patients with chest pain who do not require objective testing for coronary artery disease. OBJECTIVES: The objective was to validate the NOTR in a cohort of U.S. ED patients with acute chest pain and compare its performance to the HEART Pathway. METHODS: A secondary analysis of 282 participants enrolled in the HEART Pathway randomized controlled trial was conducted. Each patient was classified as low risk or at risk by the NOTR. Sensitivity for major adverse cardiac events (MACE) at 30 days was calculated in the entire study population. NOTR and HEART Pathways were compared among patients randomized to the HEART Pathway in the parent trial using McNemar's test and the net reclassification improvement (NRI). RESULTS: Major adverse cardiac events occurred in 22/282 (7.8%) participants, including no deaths, 16/282 (5.6%) with myocardial infarction (MI), and 6/282 (2.1%) with coronary revascularization without MI. NOTR was 100% (95% confidence interval [CI] = 84.6%-100%) sensitive for MACE and identified 78/282 patients (27.7%, 95% = CI 22.5-33.3%) as low risk. In the HEART Pathway arm (n = 141), both NOTR and HEART Pathway identified all patients with MACE as at risk. Compared to NOTR, the HEART Pathway was able to correctly reclassify 27 patients without MACE as low risk, yielding a NRI of 20.8% (95% CI = 11.3%-30.2%). CONCLUSIONS: Within a U.S. cohort of ED patients with chest pain, the NOTR and HEART Pathway were 100% sensitive for MACE at 30 days. However, the HEART Pathway identified more patients suitable for early discharge than the NOTR.

Erratum: This Article Corrects: "Trends in NRMP Data from 2007-2014 for U.S. Seniors Matching into Emergency Medicine".

[This corrects the article on p. 105 in vol. 18, PMID: 28116018.].

Derivation of a Performance Checklist for Ultrasound-Guided Arthrocentesis Using the Modified Delphi Method.

OBJECTIVES: Arthrocentesis is an important skill for physicians in multiple specialties. Recent studies indicate a superior safety and performance profile for this procedure using ultrasound guidance for needle placement, and improving quality of care requires a valid measurement of competency using this modality. METHODS: We endeavored to create a validated tool to assess the performance of this procedure using the modified Delphi technique and experts in multiple disciplines across the United States. RESULTS: We derived a 22-item checklist designed to assess competency for the completion of ultrasound-guided arthrocentesis, which demonstrated a Cronbach's alpha of 0.89, indicating an excellent degree of internal consistency. CONCLUSIONS: Although we were able to demonstrate content validity for this tool, further validity evidence should be acquired after the tool is used and studied in clinical and simulated contexts.

Trends in NRMP Data from 2007-2014 for U.S. Seniors Matching into Emergency Medicine.

INTRODUCTION: Since 1978, the National Residency Matching Program (NRMP) has published data demonstrating characteristics of applicants who have matched into their preferred specialty in the NRMP main residency match. These data have been published approximately every two years. There is limited information about trends within these published data for students matching into emergency medicine (EM). Our objective was to investigate and describe trends in NRMP data to include the following: the ratio of applicants to available EM positions; United State Medical Licensing Examination (USMLE) Step 1 and Step 2 scores (compared to the national means); number of programs ranked; and Alpha Omega Alpha Honor Medical Society (AOA) membership among U.S. seniors matching into EM. METHODS: This was a retrospective observational review of NRMP data published between 2007 and 2016. We analyzed the data using analysis of variance (ANOVA) or Kruskal-Wallis testing, and Fischer's exact or chi-squared testing, as appropriate to determine statistical significance. RESULTS: The ratio of applicants to available EM positions remained essentially stable from 2007 to 2014 but did increase slightly in 2016. We observed a net upward trend in overall Step 1 and Step 2 scores for EM applicants. However, this did not outpace the national trend increase in Step 1 and 2 scores overall. There was an increase in the mean number of programs ranked by EM applicants over the years studied from 7.8 (SD4.2) to 9.2 (SD5.0, p<0.001), driven predominantly by the cohort of U.S. students successful in the match. Among time intervals, there was a difference in the number of EM applicants with AOA membership (p=0.043) due to a drop in the number of AOA students in 2011. No sustained statistical trend in AOA membership was identified over the seven-year period studied. CONCLUSION: NRMP data demonstrate trends among EM applicants that are similar to national trends in other specialties for USMLE board scores, and a modest increase in number of programs ranked. AOA membership was largely stable. EM does not appear to have become more competitive relative to other specialties or previous years in these categories.