The effect of assessor professional background on interview evaluation during residency selection: A mixed-methods study.

BACKGROUND: Residency interviewer scores are greatly variable and seems to be influenced by personal characteristics of assessors, although factors contributing to variability remain unclear. The study sought to determine how different professional backgrounds influence assessors' scores. METHODS: Fifty-five general surgery applicants rotated through an interview station assessing teamwork. They were scored by surgeons, human-resource managers, pilots, athletes. Pearson's correlation and a repeated-measures ANOVA were used to determine correlations between professions. Structured interviews were used to probe for scoring rationale. RESULTS: Interview scores differed significantly between professions (F (3, 159) = 11.12, p < 0.001. Qualitative analysis revealed that due to the challenge of distinguishing between similarly performing candidates, assessors rely on global impressions informed by personal values. CONCLUSION: Assessor variability is ubiquitous, in part due to the subjective nature of interviews and is associated with personal values. When selecting assessors, programs should choose diverse assessors to assess to ensure a reliable selection process.

Self-Assessment in General Surgery Applicants: An Insight Into Interview Performance.

INTRODUCTION: Selecting medical students for residency is a competitive process, with a narrow range of scores separating middle-ranked applicants. Self-assessment is a fundamental skill for any competent physician with a demonstrated correlation to diagnostic ability, examination scores, and technical skills, but has yet to be investigated in residency selection. The objective of this study was to investigate the relationship between self-assessment and interview performance as a potential adjunct to discriminate between applicants. METHODS: At the University of Ottawa in 2020, 55 applicants completed a 9-station interview circuit assessing different characteristics or skills important for a career in general surgery, followed by a self-assessment questionnaire evaluating their perceived performance at each station. Pearson's correlation was used to determine the relationship between self-assessment scores (SASs) and interviewer scores (ISs). RESULTS: There was a negative correlation between SASs and ISs for all interview stations. High performers underestimated their interview performance, and low performers overestimated their performance. Seven of the nine stations reached statistical significance (r = 0.60-0.73, P < 0.001). There was significant variability in the SAS of middle-ranked applicants, with a range three times greater than the range of ISs and demonstrating distinct self-assessment skills in candidates with very similar scores. CONCLUSIONS: Although we strive to select applicants who will succeed in residency to become competent physicians, self-assessment skills may be a useful adjunct during the interview process to assist in discriminating between applicants with similar scores.

Objective Assessment of the Entrustable Professional Activity Handover in Undergraduate and Postgraduate Surgical Learners.

OBJECTIVE: This study used a virtual patient simulation (VPS) to quantifiably and objectively assess undergraduate (UG) to postgraduate (PG) medical learners' acquisition of the entrustable professional activity (EPA) "handover," focusing particularly on the transition to residency. This EPA is critical because it is part of a core competency for UG and PG training in both the United States and Canada, and is essential for patient safety and comprehensive professional communication. DESIGN: Data were collected from 3 separate groups of participants: 2 UG cohorts from an earlier study, as well as a PG cohort at the beginning of residency. All participants completed the same trauma VPS, which required a free text summary statement that was used as a surrogate for an oral handover. These were collected and scored independently, using previously developed validated rubrics, one procedural and the second semantic. SETTING: All study participants were from one site. The VPS case was completed online. PARTICIPANTS: Two different UG groups, one designated junior (N = 52), was studied at the beginning of their clerkship year, a second group, designated senior (N = 30), was studied at the end of their clerkship year. These groups were compared to a third group of PG learners (N = 31) during the initial 2 weeks of their residency. Informed consent was obtained from all participants. RESULTS: A procedural rubric assessed learners' cognitive knowledge of trauma care-management. A semantic rubric assessed their use of the professional language necessary for a safe and succinct clinical handover communication. An Analysis of Variance comparing scores on the procedural rubric was highly significant with Tukey LSD tests indicating that all 3 groups were significantly different. Students increased their scores on the procedural rubric at each stage of their training. A parallel Analysis of Variance comparing students' scores on the semantic rubric revealed no significant increase in scores, indicating that students did not improve in their capacity to communicate professionally as they progressed through their training. CONCLUSIONS: Taken together, these results demonstrate that training was successful in teaching cognitive-based procedures, but not effective in teaching professional communication, which is critical to the EPA handover. Greater emphasis needs to be placed on ensuring the acquisition of professional communication skills throughout the continuum of UG and PG clinical activities. Faculty development should serve as a support to assist medical educators to address this requirement. These results also demonstrate that VPS with associated objective and validated rubrics can be used as an assessment methodology to quantifiably measure learner performance with respect to the EPA handover. A similar strategy should be considered across the UG and PG continuum for other EPAs and could form the nexus for further research.

Planning Health Services for Seniors: Can We Use Patient's Own Perception?

OBJECTIVES: The objectives of this study were to identify needs and to estimate whether self-reported health can be used as an indicator of service needs among seniors. METHODS: This was a cross-sectional survey. Age- and sex-adjusted logistic regression was used to estimate the link between functional status indicators and fair or poor self-reported health. Forward stepwise logistic regression was performed to identify the strongest contributors of poor health. Positive predictive value (PPV), sensitivity, and specificity were calculated to identify whether health perception could be used to identify people in need of physical rehabilitation services. RESULTS: 142 seniors agreed to answer the survey, yielding a response rate of 73%. Among the respondents (mean age 79±7; 60% women), 40% rated their health as fair or poor. Seniors perceiving their health as fair or poor had higher odds of reporting impairments, activity limitations, and participation restrictions (OR ranging from 2.37 95%CI: 1.03-5-45 to 12.22 95%CI: 2.68-55.78) in comparison to those perceiving their health as good or better. The strongest contributors for poor/fair health were depression, difficulty performing household tasks, pain, and dizziness (c-statistic = 0.91 and a maximum adjusted r-squared of 0.60). Self-rated health used as single-item showed a positive predictive value (PPV) of 1, sensitivity of 52%, and specificity of 100%. CONCLUSION: Our results indicate that all seniors participating in this study and reporting fair or poor health have indicators of need for further rehabilitation services. Asking patients to rate their own health may be an alternate way of querying about need, as many older persons are afraid to report disability because of fear of further institutionalization.

Toward patient-centered care: a systematic review of how to ask questions that matter to patients.

Clinicians rarely systematically document how their patients are feeling. Single item questions have been created to help obtain and monitor patient relevant outcomes, a requirement of patient-centered care.The objective of this review was to identify the psychometric properties for single items related to health aspects that only the patient can report (health perception, stress, pain, fatigue, depression, anxiety, and sleep quality). A secondary objective was to create a bank of valid single items in a format suitable for use in clinical practice.Data sources used were Ovid MEDLINE (1948 to May 2013), EMBASE (1960 to May 2013), and the Cumulative Index to Nursing and Allied Health Literature (1960 to May 2013).For the study appraisal, 24 articles were systematically reviewed. A critical appraisal tool was used to determine the quality of articles.Items were included if they were tested as single items, related to the construct, measured symptom severity, and referred to recent experiences.The psychometric properties of each item were extracted. Validity and reliability was observed for the items when compared with clinical interviews or well-validated measures. The items for general health perception and anxiety showed weak to moderate strength correlations (r = 0.28-0.70). The depression and stress items showed good area under the receiver operating characteristic curve of 0.85 and 0.73-0.88, respectively, with high sensitivity and specificity. The fatigue item demonstrated a strong effect size and correlations up to r = 0.80. The 2 pain items and the sleep item showed high reliability (intraclass correlation coefficient [ICC] = 0.85, κ = 0.76, ICC = 0.9, respectively).The search targeted articles about psychometric properties of single items. Articles that did not have this as the primary objective may have been missed. Furthermore, not all the articles included had the complete set of psychometric properties for each item.There is sufficient evidence to warrant the use of single items in clinical practice. They are simple, easily implemented, and efficient and thus provide an alternative to multi-item questionnaires. To facilitate their use, the top performing items were combined into the visual analog health states, which provides a quick profile of how the patient is feeling. This information would be useful for regular long-term monitoring.