Missing Data in Research on Youth and Family Programs.

BACKGROUND: Multilevel data can be missing at the individual level or at a nested level, such as family, classroom, or program site. Increased knowledge of higher-level missing data is necessary to develop evaluation design and statistical methods to address it. METHODS: Participants included 9,514 individuals participating in 47 youth and family programs nationwide who completed multiple self-report measures before and after program participation. Data were marked as missing or not missing at the item, scale, and wave levels for both individuals and program sites. RESULTS: Site-level missing data represented a substantial portion of missing data, ranging from 0-46% of missing data at pre-test and 35-71% of missing data at post-test. Youth were the most likely to be missing data, although site-level data did not differ by the age of participants served. In this dataset youth had the most surveys to complete, so their missing data could be due to survey fatigue. CONCLUSIONS: Much of the missing data for individuals can be explained by the site not administering those questions or scales. These results suggest a need for statistical methods that account for site-level missing data, and for research design methods to reduce the prevalence of site-level missing data or reduce its impact. Researchers can generate buy-in with sites during the community collaboration stage, assessing problematic items for revision or removal and need for ongoing site support, particularly at post-test. We recommend that researchers conducting multilevel data report the amount and mechanism of missing data at each level.

Association between medical student debt and choice of specialty: a 6-year retrospective study.

BACKGROUND: The effect of rapidly increasing student debt on medical students' ultimate career plans is of particular interest to residency programs desiring to enhance recruitment, including primary care specialties. Previous survey studies of medical students indicate that amount of student debt influences choice of medical specialty. Research on this topic to date remains unclear, and few studies have included the average income of different specialties in analyses. The purpose of this study is to observe whether empirical data demonstrates an association between debt of graduating medical students and specialties into which students match. METHODS: This was a retrospective cross-sectional study of a public institution including data from graduation years 2010-2015. For each included student, total educational debt at graduation and matched specialty were obtained. Average income of each specialty was also obtained. Statistical hypothesis testing was performed to analyze any differences in average debt among specialties; subanalysis was performed assessing debt for primary care (PC) versus non-primary care (NPC) specialties. Correlation between student debt and average specialty income was also evaluated. RESULTS: One thousand three hundred ten students met the inclusion criteria and 178 were excluded for a final study population of 1132 (86%). The average debt was $182,590. Average debt was not significantly different among the different specialties (P = 0.576). There was no significant difference in average debt between PC and NPC specialties (PC $182,345 ± $64,457, NPC $182,868 ± $70,420, P = 0.342). There was no correlation between average specialty income and graduation debt (Spearman's rho = 0.021, P = 0.482). CONCLUSIONS: At our institution, student indebtedness did not appear to affect matched medical specialty, and no correlation between debt and average specialty income was observed. Different subspecialties and residency programs interested in recruiting more students or increasing diversity may consider addressing alternative factors which may have a stronger influence on student choices.

Treating and teaching: using publicly available data to explore the relationship between student and patient evaluations of teaching hospitals.

Introduction: Treating patients and teaching medical students are parallel activities that occur at teaching hospitals. However, the relationship between these activities is poorly understood. There have been multiple calls for assessing the quality of medical education by examining publicly available clinical data but there is minimal evidence linking these variables. Method: In this proof-of-principle study, the authors examined publicly available Hospital Consumer Assessment of Healthcare Providers and Systems (H-CAHPS)Ⓡ data collected during Calendar Year 2013 to explore the relationship between patient evaluations of their hospital experience and medical student evaluations of the educational experience at that site. Results: Pearson product-moment correlation coefficients were calculated for multiple variables. Patient ratings of doctor-patient communication correlated with student ratings of organization (R=0.882, p=0.048), educational value (R=0.882, p=0.048), teaching (R=0.963, p=0.008), and evaluation and feedback (R=0.920, p=0.027). Conclusion: These findings provide preliminary evidence for a relationship between patient experiences and the quality of education at that site. Further studies linking clinical and education outcomes are needed to explore this relationship in more depth. The contributions of specific hospital locations, providers, or clerkships need to be evaluated. Studies examining these relationships have the potential to improve both patient care and medical education.

Criterion-Based Assessment in a Norm-Based World: How Can We Move Past Grades?

In the United States, the medical education community has begun a shift from the Flexnerian time-based model to a competency-based medical education model. The graduate medical education (GME) community is substantially farther along in this transition than is the undergraduate medical education (UME) community.GME has largely adopted the use of competencies and their attendant milestones and increasingly is employing the framework of entrustable professional activities (EPAs) to assess trainee competence. The UME community faces several challenges to successfully navigating a similar transition. First is the reliance on norm-based reference standards in the UME-GME transition, comparing students' performance versus their peers' with grades, United States Medical Licensing Examination Step 1 and Step 2 score interpretation, and the structured Medical School Performance Evaluation, or dean's letter. Second is the reliance on proxy assessments rather than direct observation of learners. Third is the emphasis on summative rather than formative assessments.Educators have overcome a major barrier to change by establishing UME outcomes assessment criteria with the advent and general acceptance of the physician competency reference set and the Core EPAs for Entering Residency in UME. Now is the time for the hard work of developing assessments steeped in direct observation that can be accepted by learners and faculty across the educational continuum and can be shown to predict clinical performance in a much more meaningful way than the current measures of grades and examinations. The acceptance of such assessments will facilitate the UME transition toward competency-based medical education.