Should my recommendation letter be written by artificial intelligence?

SummaryLetters of recommendation are increasingly important for the residency match. We assessed whether an artificial intelligence (AI) tool could help in writing letters of recommendation by analyzing recommendation letters written by 3 academic staff and AI duplicate versions for 13 applicants. The preferred letters were selected by 3 blinded orthopedic program directors based on a pre-determined set of criteria. The first orthopedic program director selected the AI letter for 31% of applicants, and the 2 remaining program directors selected the AI letter for 38% of applicants, with the staff-written versions selected more often by all of the program directors (p < 0.05). The first program director recognized only 15% of the AI-written letters, the second was able to identify 92%, and the third director identified 77% of AI-written letters (p < 0.05).

Occupational health screening during Gurkha Central Selection: a retrospective cohort study.

OBJECTIVE: The selection process to the British Army's Brigade of Gurkhas is rigorous, demanding and competitive. The ethos of recruitment to the Gurkhas is grounded in an overarching tenant: that selection is free, fair and transparent. The aim of this study was to retrospectively review reasons for potential recruits (PRs) to be deemed medically unsuitable or deferred suitability on medical grounds for selection to the Brigade of Gurkhas. METHODS: A retrospective review was conducted by extracted data from published post-exercise reports for the past four years to ascertain numbers of PRs deemed medically fit, medically unsuitable or deferred suitability on medical grounds. The International Classification of Disease version 11 (ICD-11) codes were retrospectively assigned to code medical reasons for non-progression. Rates of medical non-progression were compared by year. RESULTS: A total of 3154 PRs were analysed between 2018 and 2021. There was no significant difference between PRs deemed medically fit and those deemed medically suitable or deferred on medical grounds over the study period (p=0.351). There was a significant difference in the ratio of PRs deferred on medical grounds and those deemed medically unsuitable over the study period (p<0.05). CONCLUSION: Selection to the Gurkhas is extremely competitive. These data demonstrate that, overall, reasons for medical deferral or unsuitability have remained constant despite the impact of a global pandemic. These data reinforce the central tenant of Gurkha selection; that it continues to be free, fair, and transparent.

Residency Application Selection Committee Discriminatory Ability in Identifying Artificial Intelligence-Generated Personal Statements.

OBJECTIVE: Advances in artificial intelligence (AI) have given rise to sophisticated algorithms capable of generating human-like text. The goal of this study was to evaluate the ability of human reviewers to reliably differentiate personal statements (PS) written by human authors from those generated by AI software. SETTING: Four personal statements from the archives of two surgical program directors were de-identified and used as the human samples. Two AI platforms were used to generate nine additional PS. PARTICIPANTS: Four surgeons from the residency selection committees of two surgical residency programs of a large multihospital system served as blinded reviewers. AI was also asked to evaluate each PS sample for authorship. DESIGN: Sensitivity, specificity and accuracy of the reviewers in identifying the PS author were calculated. Kappa statistic for correlation between the hypothesized author and the true author were calculated. Inter-rater reliability was calculated using the kappa statistic with Light's modification given more than two reviewers in a fully-crossed design. Logistic regression was performed with to model the impact of perceived creativity, writing quality, and authorship or the likelihood of offering an interview. RESULTS: Human reviewer sensitivity for identifying an AI-generated PS was 0.87 with specificity of 0.37 and overall accuracy of 0.55. The level of agreement by kappa statistic of the reviewer estimate of authorship and the true authorship was 0.19 (slight agreement). The reviewers themselves had an inter-rater reliability of 0.067 (poor), with only complete agreement (four out of four reviewers) on two PS, both authored by humans. The odds ratio of offering an interview (compared to a composite of "backup" status or no interview) to a perceived human author was 7 times that of a perceived AI author (95% confidence interval 1.5276 to 32.0758, p=0.0144). AI hypothesized human authorship for twelve of the PS, with the last one "unsure." CONCLUSIONS: The increasing pervasiveness of AI will have far-reaching effects including on the resident application and recruitment process. Identifying AI-generated personal statements is exceedingly difficult. With the decreasing availability of objective data to assess applicants, a review and potential restructuring of the approach to resident recruitment may be warranted.

The Utility of the Standardized Letter of Recommendation for Orthopedic Surgery Residency Applicants: A Systematic Review.

Despite widespread adoption for evaluating residency candidates, few studies have evaluated the orthopedic standardized letter of recommendation (SLOR). A systematic review using PubMed, Embase, and Web of Science was performed in June 2022. Study design and results from SLOR investigations were compiled. Common outcomes studied were summative rank statement scores and SLOR individual domains. Applicants were rated ranked to match or in the top one-third of rank lists in non-normally distributed frequencies. The association of summative rank statement score with match outcome was rarely studied. Applicants' ratings skew positively, the utility is reportedly limited, and influence on match outcome has been inadequately studied. [Orthopedics. 2024;47(1):e1-e5.].

Gender Differences in Letters of Recommendation for Gynecology Surgical Fellowship Applicants: Analysis of One Program's Recruitment Cycle.

STUDY OBJECTIVE: To examine whether there are gender differences in letters of recommendation (LORs) written for residents applying to gynecology surgical fellowships. DESIGN: Retrospective study. SETTING: Single, academic institution. PATIENTS: LORs for applicants to gynecology oncology, urogynecology, and minimally invasive gynecology fellowships during the 2019-2020 application cycle. INTERVENTIONS: Not applicable. MEASUREMENTS AND MAIN RESULTS: We analyzed the linguistic content of the letters for the presence of 4 summary variables and 21 word categories based on previous studies using validated computerized text analysis software. We used multivariable analysis using linear mixed models to compare linguistic characteristics of letters by applicant gender. We performed qualitative content analysis on letters and compared the frequency of code themes by gender. The mixed-method design was planned to allow for analysis of domains not captured in text analysis. Among 680 letters written for 186 applicants, 124 (18.2%) were written for men, and 556 (81.8%) were written for women. There were no differences in the least square mean (standard error) word counts for LORs written for men and women applicants, 465 (20.0) vs 458(9.4) words, p = .74. In multivariable analysis, LORs written for men were found to have higher authentic tone and more risk words (p = .005 and p = .03, respectively). LORs written for women contained more communal (relationship-oriented) words (p = .006). The qualitative analysis demonstrated that ability, interpersonal traits, surgical skills, and research were the most often mentioned themes. Comments about compassion/empathy, leadership potential, teaching, interpersonal skills, and patient rapport were found more often in letters for men. More doubt raisers (words that raise doubt or concern) were present in letters for men, but letters for both genders had similar levels of negative criticism. In contrast, comments on ability, being "drama-free," and self-awareness were found more often in letters for women. CONCLUSION: There were gender differences in LORs written for obstetrics and gynecology surgical subspecialty fellowship applicants indicating the presence of gender bias.

Two-Year Interview and Match Outcomes of Otolaryngology Preference Signaling.

OBJECTIVE: To present the first year-over-year data comparison of preference signaling for residency interviews in the otolaryngology application marketplace. STUDY DESIGN: Cross-sectional study conducted over 2 application cycles. SETTING: Otolaryngology training programs in the United States. METHODS: Otolaryngology residency applicants were invited to participate in preference signaling during the 2021 and 2022 application cycles. Submissions were collected using a web-based interface. The distribution of signals among programs was evaluated descriptively and in relationship to Doximity program reputation rankings. Surveys were sent to applicants to assess general attitudes and the number of interview invitations received from signaled and nonsignaled programs. Surveys were sent to programs to evaluate use of signals and the impact on match results. RESULTS: Programs received a range of signals, from 0 to 66, with 50% of signals going to 24% of programs in 2022, which was similarly found in 2021. Programs of higher rank tended to receive more signals. Overall, >87% of surveyed applicants received an interview offer from at least 1 program that they signaled. In 2021 and 2022, applicants were 2.6 times more likely to get an interview from a signaled program than a comparator nonsignaled program. A greater positive impact on interview offer rate was seen for less competitive applicants. Signaling was viewed favorably by the majority of surveyed applicants and programs. CONCLUSIONS: Preference signaling for otolaryngology residency interviews demonstrates a promising mechanism to improve applicant visibility to programs during the application cycle. This impact is consistent over 2 application cycles.

Exploring Characteristics of Academic General Surgery Residency Applicants: A Group Concept-Mapping Approach.

OBJECTIVE: Holistic review, which emphasizes qualitative attributes over objective measures, has been proposed as a method for selecting candidates for surgical residency in order to improve diversity in graduate medical education, and, ultimately, the field of surgery. This study seeks to articulate desirable traits of applicants as a first-step in standardizing the holistic review process. DESIGN: Using Group Concept Mapping, a web-based mixed-methods participatory research methodology, residency selection committee members were asked to 1) list desirable characteristics of applicants, 2) group these into categories, 3) rate their importance to academic/clinical success on a 5-point Likert scale (1 = not at all important, 5 = extremely important), and 4) rate the degree to which each characteristic is feasible to assess on a 3-point Likert scale (1 = not at all feasible, 3 = very feasible). Grouped characteristics submitted to hierarchical cluster analysis depicted committee's consensus about desirable qualities/criteria for applicants. Bivariate scatter-plots and pattern-matching graphics demonstrated which of these criteria were most important and reliably assessed. SETTING: A single academic general surgery residency training program in Western Pennsylvania. PARTICIPANTS: Members of the selection committee for the UPMC General Surgery Residency program who had participated in at least 1 prior cycle of applicant selection. RESULTS: Desirable characteristics of highly qualified applicants into an academic general surgery residency were clustered into domains of 1) scholarly work and research, 2) grades/formal assessments, 3) program fit, 4) behavioral assets, and 5) aspiration. Behavioral assets, which was felt to be the most important to clinical and academic success were considered to be the least feasible to reliably assess. Within this domain, initiative, being self-motivated, intellectual curiosity, work ethic, communication skills, maturity and self-awareness, and thoughtfulness were viewed as most frequently reliably assessed from the application and interview process. CONCLUSIONS: High quality applicants possess several behavioral assets that faculty deem are important to academic and clinical success. Adapting validated metrics for assessing these assets, may provide a solution for addressing subjectivity and other challenges scrutinized by critics of holistic review.

Linguistic Differences by Gender in Letters of Recommendation for Minimally Invasive Gynecologic Surgery Fellowship Applicants.

OBJECTIVE: Various surgical specialties have reported gender bias in letters of recommendation (LOR). We aimed to determine if linguistic differences exist in LOR for female and male physicians applying to Fellowship in Minimally Invasive Gynecologic Surgery (FMIGS). DESIGN: Retrospective cohort study including application cycles 2019 and 2020. SETTING: Academic university hospital. PARTICIPANTS: FMIGS applicants. RESULTS: Applicant demographic and baseline data included age, race, gender, geographical region of residency training, Step 1 and 2 scores, number of research and volunteer activities, Alpha Omega Alpha and Gold Humanism status and number of LOR, as well as the gender and academic rank of the letter writer. The Linguistic Inquiry and Word Count software, a validated text analysis program, was used to characterize LOR linguistic content. A total of 118 applications, including 391 letters, were analyzed. Seventy-six (64.4%) applicants were female and 42 (35.6%) were male. Most female applicants were white (46% vs. 36%, p = 0.04), had Alpha Omega Alpha status (13% vs. 0%, p = 0.01), higher Step 2 scores (239.7 vs. 230.4, p < 0.01), and more service activities (7.7 vs. 4.7, p < 0.01), compared to male applicants. Male applicants were more likely to graduate from international medical schools (45% vs. 16%, p < 0.01). Female authors wrote 159 LOR, and male authors wrote 232. Following multivariable analysis controlling for race, Step 1 score and letter writer gender, no significant differences in average LOR word count for female and male applicants (406.7 ± 24.2 words vs. 340.1 ± 35.4 words), or differences in Linguistic Inquiry and Word Count linguistic categories existed. CONCLUSIONS: Although Baseline differences were noted between female and male FMIGS applicants, no differences in LOR length or linguistic categories were noted. These results likely reflect the impact of female predominance and increased efforts to address gender bias within Obstetrics and Gynecology.

Prevalence, Bias, and Rank List Impact of Illegal Questions in Surgical Specialty Residency Interviews.

OBJECTIVE: The purpose of this study is to characterize illegal questions as defined by federal law and to assess their impact on applicants' rank lists across four surgical specialties. DESIGN: A survey was developed and sent to surgical specialty residency applicants. The survey asked demographics, the frequency of questions about age, gender, religion, sexual orientation, family status and impact on final rank list. Applicants were asked to respond anonymously based on their experience at all institutions at which they interviewed during the interview cycle. Results were compared by applicant specialty and gender. SETTING: A large university-affiliated academic medical center PARTICIPANTS: Survey was administered to 3854 applicants (comprising between 28.9% and 41.2% of applicants nationwide) to general surgery, orthopaedic surgery, urology, and otolaryngology residency programs at a single institution during the 2018 and 2019 cycles. A total of 1066 applicants completed the survey. RESULTS: A total of 789 (74.0%) of applicants reported being asked at least one illegal question during the interview process at any institution. Applicants to orthopaedic surgery programs were most likely to be asked illegal question (n = 315, 81.6%), and general surgery applicants were least likely to be asked illegal questions (n = 324, 66.8%, p < 0.001). Females were more likely than males to be asked about gender (n = 99, 26.3% vs. n = 18, 2.6%, p < 0.001) and plans for pregnancy (n = 78, 20.8% vs. n = 78, 11.4%, p < 0.001). 152 (19.4%). Applicants reported that being asked an illegal question lowered a program on their rank list. Female applicants were more likely to lower a program on their rank list as a result of an illegal question (n = 102, 35.4% vs. n = 50, 10.1%, p < 0.001). CONCLUSIONS: Illegal questions in surgical specialty residency interviews are common, vary by specialty and applicant gender, and lower programs on applicants' rank lists. This data should serve to inform larger and more inclusive studies in the future. Programs should focus on educating interviewers on illegal topics in an effort to minimize illegal topics that may alienate applicants and contribute to workplace discrimination.

Perspectives on virtual interviews-A follow-up study of the Comprehensive Endocrine Surgery Fellowship interview process.

BACKGROUND: The American Association of Endocrine Surgeons Comprehensive Endocrine Surgery Fellowship interview stakeholders previously favored in-person interviews, despite time and expense. This study assessed perception changes given mandated virtual interviews because of coronavirus disease 2019. METHODS: Immediately after the 2020 Match, anonymous surveys were distributed to applicants (n = 37) and program directors (n = 22). Mixed-methods analyses were used to evaluate responses. Results were compared to data from a prior study of the 2013 to 2018 in-person interview process. RESULTS: Response rates were 82% (program directors) and 60% (applicants). Compared with prior applicants, 2020 applicants attended similar numbers of interviews (1-10, 32% vs 37%; P = .61), used fewer vacation days (23% vs 56%; P = .01), and most reported 0 expenses. Burdens included lack of protected time for interviews. The virtual format did not compromise applicant ability to meet faculty (mean rank = 6.8/10) or make favorable impressions (mean rank = 6.8/10). Program directors reported equivalent or improved assessments of applicants. Program directors (72%) and applicants (77%) indicated that future interviews should be partially or completely virtual. CONCLUSION: In contrast to prior survey data, applicants and program directors now express interest in virtual or hybrid interview processes. Virtual interviews were less costly, less time-consuming, and met goals effectively. Integrating virtual interview components will require innovative strategies to reduce redundancies and promote equitable access.

Novel Strategies for Evaluating and Improving Plastic Surgery Applicant Selection.

SUMMARY: Residency applicant evaluation and selection is a critical part of developing and maintaining a high-quality plastic surgery residency program. Currently, many programs rely on objective measures such as the United States Medical Licensing Exam scores, number of research publications, grade point average, Alpha Omega Alpha Honor Medical Society status, or a combination of these objective metrics. However, there is a growing body of literature suggesting that the current means of residency applicant evaluation and selection may not be the best predictive factors of future resident success. The aim of this study was to identify nontraditional means of evaluating plastic surgery residency candidates and discuss how these means have been implemented at the authors' institution. After reviewing industry hiring practices, the authors propose that standardized interviewing and personality testing can help evaluate some of the previously intangible parts of an applicant that may play a role in teamwork, commitment, and dedication to patient care.

Resident Selection: The Role of Assessment of Emotional Intelligence and Grit.

SUMMARY: The ability of our resident selection process to identify individuals who will ultimately become competent plastic surgeons is crucial to the specialty's future. Current criteria in use are not productive of that outcome. The presence of emotional intelligence and the element of grit have been incorporated in business and the military as factors to be evaluated in potential candidates. Plastic surgery should initiate an investigation of inclusion of a similar assessment of resident applicants.

Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs?

BACKGROUND: Narrative letters of recommendation are an important component of the residency application process. However, because narrative letters of recommendation are almost always positive, it is unclear whether those reviewing the letters understand the writer's intended strength of support for a given applicant. QUESTIONS/PURPOSES: (1) Is the perception of letter readers for narrative letters of recommendation consistent with the intention of the letter's author? (2) Is there inter-reviewer consistency in selection committee members' perceptions of the narrative letters of recommendation? METHODS: Letter writers who wrote two or more narrative letters of recommendation for applicants to one university-based orthopaedic residency program for the 2014 to 2015 application cycle were sent a survey linked to a specific letter of recommendation they authored to assess the intended meaning regarding the strength of an applicant. A total of 247 unstructured letters of recommendation and accompanying surveys were sent to their authors, and 157 surveys were returned and form the basis of this study (response percentage 64%). The seven core members of the admissions committee (of 22 total reviewers) at a university-based residency program were sent a similar survey regarding their perception of the letter. To answer our research question about whether letter readers' perceptions about a candidate were consistent with the letter writer's intention, we used kappa values to determine agreement for survey questions involving discrete variables and Spearman correlation coefficients (SCCs) to determine agreement for survey questions involving continuous variables. To answer our research question regarding inter-reviewer consistency among the seven faculty members, we compared the letter readers' responses to each survey question using intraclass correlation coefficients (ICCs). RESULTS: There was a negligible to moderate correlation between the intended and perceived strength of the letters (SCC 0.26 to 0.57), with only one of seven letter readers scoring in the moderate correlation category. When stratifying the applicants into thirds, there was only slight agreement (kappa 0.07 to 0.19) between the writers and reviewers. There were similarly low kappa values for agreement about how the writers and readers felt regarding the candidate matching into their program (kappa 0.14 to 0.30). The ICC for each question among the seven faculty reviewers ranged from poor to moderate (ICC 0.42 to 0.52). CONCLUSION: Our results demonstrate that the reader's perception of narrative letters of recommendation did not correlate well with the letter writer's intended meaning and was not consistent between letter readers at a single university-based urban orthopaedic surgery residency program. CLINICAL RELEVANCE: Given the low correlation between the intended strength of the letter writers and the perceived strength of those letters, we believe that other options such as a slider bar or agreed-upon wording as is used in many dean's letters may be helpful.

In Search of Workforce Diversity? A Program's Successful Approach.

SUMMARY: The persistence of health care disparities along racial and ethnic lines highlights the complex and multifactorial nature of this national concern. The paucity of physicians ethnically underrepresented in medicine to treat an ever-growing heterogeneous population inherently contributes to these ongoing disparities. The authors proposed an approach to improve the representation of physicians underrepresented in medicine in their plastic surgery residency program. With a renewed commitment to ethnic diversity and inclusion, a multifaceted recruitment and retention approach was implemented at the University of Pennsylvania plastic and reconstructive surgery residency program from 2015 to 2020 (5 academic years). A retrospective review of the demographics of the program's residents was then assessed over the past 9 academic years for comparison (2011 to 2020). The representation of underrepresented-in-medicine residents within the plastic and reconstructive surgery residency program steadily increased with the implementation of this multifaceted approach, reaching an unprecedented high. Currently, 29 percent of all residents are underrepresented in medicine and 29 percent are female, some of whom are also underrepresented in medicine. Although the female representation is on par with the national average, the underrepresented-in-medicine representation is far greater than the national average. As a result of this multifaceted approach, the representation of African American and Latino plastic surgery residents at the University of Pennsylvania now far exceeds current national averages. Unfortunately, the representation of Native American and Alaskan Natives is still lacking, despite the program's broadened recruitment efforts. The success of this experience describes a successful strategy that institutions can implement to enhance underrepresented-in-medicine representation among its plastic surgery trainees.

An analysis of applicant competitiveness to general surgery, surgical subspecialties, and integrated programs.

BACKGROUND: General surgery was once the gateway into a career in surgery. Over time, surgical subspecialties developed separate residency programs, and recently, integrated programs have emerged. It is unknown what impact the presence of surgical subspecialties and integrated programs have had on general surgery. Our objective was to evaluate match trends and quantify competitiveness of the general surgery, integrated programs, and surgical subspecialties matches. METHODS: National Residency Matching Program match data and applicant characteristics from 2010 through 2020 were analyzed for US senior allopathic applicants. Integrated programs were defined as plastic and vascular surgery, and surgical subspecialties were defined as otolaryngology, orthopedic surgery, and neurosurgery. Trends were evaluated using linear regression, programs were compared on 10 metrics by Wilcoxon rank-sum tests, and a logistic regression was used to rank each specialty match. RESULTS: The number of US senior applicants per position to integrated programs decreased and approached that of general surgery and surgical subspecialties, but the median number of applicants per position to general surgery was lower than to surgical subspecialties or integrated programs (1.21 interquartile range). Our logistic regression showed United States Medical Licensing Examination scores, research experience, Alpha Omega Alpha Honor Society membership, and graduation from a top medical school to be the most important factors in the match, and our weighted rank score found general surgery (2.85) to be less competitive than surgical subspecialties (1.92) or integrated programs (1.17). CONCLUSION: Throughout the last decade, integrated programs and surgical subspecialties have matched more competitive applicants based on the most significant predictors of the match. Moving forward, it is important that general surgery strives to attract the best and brightest out of medical school.

Pearls and Pitfalls of the Virtual Interview: Perspectives From Both Sides of the Camera.

As the SARS-COV-2 pandemic created the need for social distancing and the implementation of nonessential travel bans, residency and fellowship programs have moved toward a web-based virtual process for applicant interviews. As part of the Society of Asian Academic Surgeons 5th Annual Meeting, an expert panel was convened to provide guidance for prospective applicants who are new to the process. This article provides perspectives from applicants who have successfully navigated the surgical subspecialty fellowship process, as well as program leadership who have held virtual interviews.