Using Unannounced Standardized Patients to Assess Clinician Telehealth and Communication Skills at an Urban Student Health Center.

PURPOSE: As the COVID-19 pandemic forced most colleges and universities to go online, student health centers rapidly shifted to telehealth platforms without frameworks for virtual care provision. An urban student health center implemented a needs assessment involving unannounced standardized patients (USPs) to evaluate the integration of a new telehealth workflow and clinicians' virtual communication skills. METHODS: From April to May 2021, USPs conducted two video visits with 12 primary care and four women's health clinicians (N = 16 clinicians; 32 visits). Cases included (1) a 21-year-old female presenting for birth control with a positive Patient Health Questionaire-9 and (2) a 21-year-old male, who vapes regularly, with questions regarding safe sex with men. Clinicians were evaluated using a checklist completed by the USP immediately following the visit and a systematic chart review of the electronic health record. RESULTS: USP feedback indicates most clinicians received high ratings for general communication skills but may benefit from educational intervention in several key telemedicine skills. Clinicians struggled with using nonverbal signals to enrich communication (47% well done), acknowledging emotions (34% well done), and using video for information gathering (34% well done). Low rates of standard screenings (e.g., 63% administered the PHQ-2, <50% asked about alcohol use) suggested protocols for in-person care were not easily incorporated into telehealth practices, and clinicians may benefit from enhanced care team support. Performance reports were shared with clinicians and leadership postvisit. DISCUSSION: Results suggest project design and implementation is scalable and feasible for use at other institutions, offering a structured methodology that can improve general student health care.

Scaffolding the Transition to Residency: A Qualitative Study of Coach and Resident Perspectives.

PURPOSE: This study explores coaching during transition from medical school to residency through the perspectives of residents and faculty coaches participating in a coaching program from residency match through the first year of residency. METHOD: From January to September 2020, 15 faculty coaches in internal medicine, obstetrics and gynecology, emergency medicine, orthopedics, and pathology participated in a synchronous, in-person coaching training course. All 94 postgraduate year 1 residents in these 5 training programs participated. Between November 2021 and March 2022, focus groups were held with interns from all residency programs participating in the program. Interviews were conducted with faculty coaches in February 2022. Faculty and residents discussed their experiences with and perceptions of coaching. De-identified transcripts were coded, and researchers organized these codes into broader categories, generated cross-cutting themes from the concepts described in both cohorts, and proposed a model for the potential of coaching to support the transition to residency. Descriptive themes were constructed and analytic themes developed by identifying concepts that crossed the data sets. RESULTS: Seven focus groups were held with 39 residents (42%). Residents discussed the goals of a coaching program, coach attributes, program factors, resident attributes, and the role of the coach. Coaches focused on productivity of coaching, coaching skills and approach, professional development, and scaffolding the coaching experience. Three analytic themes were created: (1) coaching as creating an explicit curriculum for growth through the transition to residency, (2) factors contributing to successful coaching, and (3) ways in which these factors confront graduate medical education norms. CONCLUSIONS: Learner and faculty perspectives on coaching through the transition to residency reveal the potential for coaching to make an explicit and modifiable curriculum for professional growth and development. Creating structures for coaching in graduate medical education may allow for individualized professional development, improved mindset, self-awareness, and self-directed learning.

The Biomedical Entrepreneurship Skills Development Program for the Advancement of Research Translation: Foundations of Biomedical Startups course, metrics, and impact.

Background/Objective: A growing number of biomedical doctoral graduates are entering the biotechnology and industry workforce, though most lack training in business practice. Entrepreneurs can benefit from venture creation and commercialization training that is largely absent from standard biomedical educational curricula. The NYU Biomedical Entrepreneurship Educational Program (BEEP) seeks to fill this training gap to prepare and motivate biomedical entrepreneurs to develop an entrepreneurial skill set, thus accelerating the pace of innovation in technology and business ventures. Methods: The NYU BEEP Model was developed and implemented with funding from NIDDK and NCATS. The program consists of a core introductory course, topic-based interdisciplinary workshops, venture challenges, on-line modules, and mentorship from experts. Here, we evaluate the efficacy of the core, introductory course, "Foundations of Biomedical Startups," through the use of pre/post-course surveys and free-response answers. Results: After 2 years, 153 participants (26% doctoral students, 23% post-doctoral PhDs, 20% faculty, 16% research staff, 15% other) have completed the course. Evaluation data show self-assessed knowledge gain in all domains. The percentage of students rating themselves as either "competent" or "on the way to being an expert" in all areas was significantly higher post-course (P < 0.05). In each content area, the percentages of participants rating themselves as "very interested" increased post-course. 95% of those surveyed reported the course met its objectives, and 95% reported a higher likelihood of pursuing commercialization of discoveries post-course. Conclusion: NYU BEEP can serve as a model to develop similar curricula/programs to enhance entrepreneurial activity of early-stage researchers.

Understanding medical student paths to communication skills expertise using latent profile analysis.

PURPOSE: To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. MATERIALS AND METHODS: Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. RESULTS: Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles. CONCLUSION: Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm.

PURPOSE: To explore whether a machine-learning algorithm could accurately perform the initial screening of medical school applications. METHOD: Using application data and faculty screening outcomes from the 2013 to 2017 application cycles (n = 14,555 applications), the authors created a virtual faculty screener algorithm. A retrospective validation using 2,910 applications from the 2013 to 2017 cycles and a prospective validation using 2,715 applications during the 2018 application cycle were performed. To test the validated algorithm, a randomized trial was performed in the 2019 cycle, with 1,827 eligible applications being reviewed by faculty and 1,873 by algorithm. RESULTS: The retrospective validation yielded area under the receiver operating characteristic (AUROC) values of 0.83, 0.64, and 0.83 and area under the precision-recall curve (AUPRC) values of 0.61, 0.54, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The prospective validation yielded AUROC values of 0.83, 0.62, and 0.82 and AUPRC values of 0.66, 0.47, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The randomized trial found no significant differences in overall interview recommendation rates according to faculty or algorithm and among female or underrepresented in medicine applicants. In underrepresented in medicine applicants, there were no significant differences in the rates at which the admissions committee offered an interview (70 of 71 in the faculty reviewer arm and 61 of 65 in the algorithm arm; P = .14). No difference in the rate of the committee agreeing with the recommended interview was found among female applicants (224 of 229 in the faculty reviewer arm and 220 of 227 in the algorithm arm; P = .55). CONCLUSIONS: The virtual faculty screener algorithm successfully replicated faculty screening of medical school applications and may aid in the consistent and reliable review of medical school applicants.

Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education.

Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

SMARTer Goalsetting: A Pilot Innovation for Coaches During the Transition to Residency.

PROBLEM: Ability to set goals and work with coaches can support individualized, self-directed learning. Understanding the focus and quality of graduating medical student and first-year resident goals and the influence of coaching on goal-setting can inform efforts to support learners through the transition from medical school to residency. APPROACH: This observational study examined goal-setting among graduating medical students and first-year residents from April 2021 to March 2022. The medical students set goals while participating in a Transition to Residency elective. The residents in internal medicine, obstetrics and gynecology, emergency medicine, orthopedics, and pathology set goals through meeting 1:1 with coaches. Raters assessed goals using a 3-point rubric on domains of specific, measurable, attainable, relevant, and timely (i.e., SMART goal framework) and analyzed descriptive statistics, Mann-Whitney U tests, and linear regressions. OUTCOMES: Among 48 medical students, 30 (62.5%) set 108 goals for early residency. Among 134 residents, 62 (46.3%) entered goals. Residents met with coaches 2.8 times on average (range 0-8 meetings, median = 3). Goal quality was higher in residents than medical students (average score for S: 2.71 vs 2.06, P < .001; M: 2.38 vs 1.66, P < .001; A: 2.92 vs 2.64, P < .001; R: 2.94 vs 2.86, P = .002; T: 1.71 vs 1.31, P < .001). The number of coaching meetings was associated with more specific, measurable goals (specific: F [1, 1.02] = 6.56, P = .01, R2 = .10; measurable: F [1, 1.49] = 4.74, P = .03, R2 = .07). NEXT STEPS: Learners set realistic, attainable goals through the transition to residency, but the goals could be more specific, measurable, and timely. The residents set SMARTer goals, with coaching improving goal quality. Understanding how best to scaffold coaching and support goal-setting through this transition may improve trainees' self-directed learning and well-being.

Bridging the Gap from Student to Doctor: Developing Coaches for the Transition to Residency.

BACKGROUND: A lack of educational continuity creates disorienting friction at the onset of residency. Few programs have harnessed the benefits of coaching, which can facilitate self-directed learning, competency development, and professional identity formation, to help ease this transition. OBJECTIVE: To describe the process of training faculty Bridge Coaches for the Transition to Residency Advantage (TRA) program for interns. METHODS: Nineteen graduate faculty educators participated in a coaching training course with formative skills assessment as part of a faculty development program starting in January 2020. Surveys (n = 15; 79%) and a focus group (n = 7; 37%) were conducted to explore the perceived impact of the training course on coaching skills, perceptions of coaching, and further program needs during the pilot year of the TRA program. RESULTS: Faculty had strong skills around establishing trust, authentic listening, and supporting goal-setting. They required more practice around guiding self-discovery and following a coachee-led agenda. Faculty found the training course to be helpful for developing coaching skills. Faculty embraced their new roles as coaches and appreciated having a community of practice with other coaches. Suggestions for improvement included more opportunities to practice and receive feedback on skills and additional structures to further support TRA program encounters with coaches. CONCLUSIONS: The faculty development program was feasible and had good acceptance among participants. Faculty were well-suited to serve as coaches and valued the coaching mindset. Adequate skills reinforcement and program structure were identified as needs to facilitate a coaching program in graduate medical education.

Addressing social determinants of health in primary care: a quasi-experimental study using unannounced standardised patients to evaluate the impact of audit/feedback on physicians' rates of identifying and responding to social needs.

BACKGROUND: Although efforts are underway to address social determinants of health (SDOH), little is known about physicians' SDOH practices despite evidence that failing to fully elicit and respond to social needs can compromise patient safety and undermine both the quality and effectiveness of treatment. In particular, interventions designed to enhance response to social needs have not been assessed using actual practice behaviour. In this study, we evaluate the degree to which providing primary care physicians with feedback on their SDOH practice behaviours is associated with increased rates of eliciting and responding to housing and social isolation needs. METHODS: Unannounced standardised patients (USPs), actors trained to consistently portray clinical scenarios, were sent, incognito, to all five primary care teams in an urban, safety-net healthcare system. Scenarios involved common primary care conditions and each included an underlying housing (eg, mould in the apartment, crowding) and social isolation issue and USPs assessed whether the physician fully elicited these needs and if so, whether or not they addressed them. The intervention consisted of providing physicians with audit/feedback reports of their SDOH practices, along with brief written educational material. A prepost comparison group design was used to evaluate the intervention; four teams received the intervention and one team served as a 'proxy' comparison (no intervention). Preintervention (February 2017 to December 2017) rates of screening for and response to the scripted housing and social needs were compared with intervention period (January 2018 to March 2019) rates for both intervention and comparison teams. RESULTS: 108 visits were completed preintervention and 183 during the intervention period. Overall, social needs were not elicited half of the time and fully addressed even less frequently. Rates of identifying the housing issue increased for teams that received audit/feedback reports (46%-60%; p=0.045) and declined for the proxy comparison (61%-42%; p=0.174). Rates of responding to housing needs increased significantly for intervention teams (15%-41%; p=0.004) but not for the comparison team (21%-29%; p=0.663). Social isolation was identified more frequently postintervention (53%) compared with baseline (39%; p=0.041) among the intervention teams but remained unchanged for the comparison team (39% vs 32%; p=0.601). Full exploration of social isolation remained low for both intervention and comparison teams. CONCLUSIONS: Results suggest that physicians may not be consistently screening for or responding to social needs but that receiving feedback on those practices, along with brief targeted education, can improve rates of SDOH screening and response.

Does it get better? An ongoing exploration of physician experiences with and acceptance of telehealth utilization.

INTRODUCTION: COVID-19 forced health systems to rapidly implement telehealth for routine practice, often without sufficient training or standards. We conducted a longitudinal survey of physicians to explore changes in their perceptions of the challenges and benefits of telehealth and identify recommendations for future practice. METHODS: An anonymous online survey was distributed to a cohort of internal medicine physicians in May to June 2020 and March to June 2021. Changes in responses between 2020 and 2021 and by site (private vs. public) were described. These findings, along with those of a thematic analysis of open-ended responses to questions on telehealth experiences, informed a set of recommendations. RESULTS: The survey response rate was 111/391 in 2021 compared to 122/378 in 2020. Fewer physicians reported that telehealth was more difficult than in person with regards to taking a history (49% in 2020, 33% in 2021, p= 0.015), maximizing patient adherence (33% in 2020, 19% in 2021, p = 0.028), and maintaining patient relationships (31%, 25%, p = 0.009) in 2021. Similar proportions of physicians reported continued challenges with building new patient relationships (75%, 77%, p = 0.075), educating patients (40%, 32%, p = 0.393), and working collaboratively with their team (38%, 41%, p = 0.794). Physicians reported increased satisfaction with tele-visits over in-person visits (13%, 27%, p = 0.006) and less worry over doing future tele-visits (45%, 31%, p = 0.027). Physicians' open-ended responses identified recommendations for further improving the design and use of telehealth. DISCUSSION: Results suggest that physician experience with telehealth improved but opportunities for training and improved integration remain. Longitudinal assessment can deepen understanding of the evolution of telehealth care.

"I Don't Trust It": Use of a Routine OSCE to Identify Core Communication Skills Required for Counseling a Vaccine-Hesitant Patient.

BACKGROUND: Vaccine hesitancy is challenging for clinicians and of increasing concern since COVID-19 vaccination rollout began. Standardized patients (SPs) provide an ideal method for assessing resident physicians' current skills, providing opportunity to practice and gain immediate feedback, while also informing evaluation of curriculum and training. As such, we designed and implemented an OSCE station where residents were tasked with engaging and educating a vaccine-hesitant patient. AIM: Describe residents' vaccine counseling practices, core communication and interpersonal skills, and effectiveness in meeting the objectives of the case. Explore how effectiveness in overcoming vaccine hesitancy may be associated with communication and interpersonal skills in order to inform educational efforts. SETTING: Annual OSCE at a simulation center. PARTICIPANTS: 106 internal medicine residents (51% PGY1, 49% PGY2). PROGRAM DESCRIPTION: Residents participated in an annual residency-wide, multi-station OSCE, one of which included a Black, middle-aged, vaccine-hesitant male presenting for a routine video visit. Residents had 10 min to complete the encounter, during which they sought to educate, explore concerns, and make a recommendation. After each encounter, faculty gave residents feedback on their counseling skills and reviewed best practices for effective communication on the topic. SPs completed a behaviorally anchored checklist (30 items across 7 clinical skill domains and 2 measures of trust in the vaccine's safety and resident) which will inform future curriculum. PROGRAM EVALUATION: Fifty-five percent (SD: 43%) of the residents performed well on the vaccine-specific education domain. PGY2 residents scored significantly higher on two of the seven domains compared to PGY1s (patient education/counseling-PGY1: 35% (SD: 36%) vs. PGY2: 52% (SD: 41%), p = 0.044 and activation-PGY1: 37% (SD: 45%) vs. PGY2: 59% (SD: 46%), p = 0.016). In regression analyses, education/counseling and vaccine-specific communication skills were strongly, positively associated with trust in the resident and in the vaccine's safety. A review of qualitative data from the SPs' perspective suggested that low performers did not use patient-centered communication skills. DISCUSSION: This needs assessment suggests that many residents needed in-the-moment feedback, additional education, and vaccine-specific communication practice. Our program plans to reinforce evidence-based practices physicians can implement for vaccine hesitancy through ongoing curriculum, practice, and feedback. This type of needs assessment is replicable at other institutions and can be used, as we have, to ultimately shed light on next steps for programmatic improvement.

A Preliminary Evaluation of Students' Learning and Performance Outcomes in an Accelerated 3-Year MD Pathway Program.

BACKGROUND: Little outcome data exist on 3-year MD (3YMD) programs to guide residency program directors (PDs) in deciding whether to select these graduates for their programs. OBJECTIVE: To compare performance outcomes of 3YMD and 4-year MD (4YMD) students at New York University Grossman School of Medicine. METHODS: In 2020, using the Kirkpatrick 4-level evaluation model, outcomes from 3 graduating cohorts of 3YMD students (2016-2018) were compared with the 4YMD counterparts. RESULTS: Descriptive statistics compared outcomes among consented student cohorts: 92% (49 of 53) 3YMD, 87% (399 of 459) 4YMD-G, and 84% (367 of 437) 4YMD-S. Student survey response rates were 93% (14 of 15), 74% (14 of 19), and 89% (17 of 19) from 2016 to 2018. PDs' response rates were 58% (31 of 53, 3YMD) and 51% (225 of 441, 4YMD). Besides age, 3YMD and 4YMD cohorts did not differ significantly in admissions variables. Other than small statistically significant differences in the medicine shelf examination (3YMD mean 74.67, SD 7.81 vs 4YMD-G mean 78.18, SD 7.60; t test=3.02; P=.003) and USMLE Step 1 (3YMD mean 235.13, SD 17.61 vs 4YMD-S mean 241.70, SD 15.92; t test=2.644; P=.009 and vs 4YMD-G mean 242.39, SD 15.65; t test=2.97; P=.003) and Step 2 CK scores (3YMD mean 242.57, SD 15.58 vs 4YMD-S mean 248.55, SD 15.33; t test=2.55; P=.01 and vs 4YMD-G mean 247.83, SD 15.38; t test=2.97; P=.03), other metrics and overall intern ratings did not differ by pathway. CONCLUSIONS: Exploratory findings from a single institution suggest that 3YMD students performed similarly to 4YMD students in medical school and the first year of residency.

The Telemedicine Takeover: Lessons Learned During an Emerging Pandemic.

Introduction: Health care systems rose to the challenges of COVID-19 by creating or expanding telehealth programs to ensure that patients could access care without an in-person appointment. Traditionally, physicians receive limited formal telemedicine training, making preparedness for this transition uneven. To describe challenges to and attitudes toward providing virtual patient care, we distributed a survey to physicians within our system who largely had no formal prior training/experience with telemedicine, but transitioned to routine telemedicine use. Data collected are then used to offer actionable recommendations for health system leaders and medical educators. Materials and Methods: Surveys were distributed to all faculty outpatient general internal medicine physicians working at any New York University Langone Health, New York City Health + Hospitals/Bellevue and Gouverneur, and the VA NY Harbor Health System (n = 378) in mid-2020. Survey items consisted of Likert and open-ended questions related to experience with televisits (13 items) and attitudes toward care (24 items). Results: Telehealth-related challenges varied by site and modality. Primary challenges included establishing a connection from the patient's (98%) or physician's end (84%) and difficulty in the following domains: working with team members (39%), physical examinations (95%), establishing new patient relationships (70%), and taking history (40%), among others. In thematic analysis, significant themes with illustrative qualitative commentary emerged related to technological challenges, new systems issues, and new patient/physician communication experiences. Discussion: Experience differences were rooted in the type of technology employed. Safety-net practices conducted mostly telephonic visits, whereas private outpatient sites utilized video, despite both using identical electronic medical records. As we consider a "new normal" and prolonged community transmission of COVID-19 infection, it is essential to establish telemedicine training, tools, and protocols that meet the needs of both patients and physicians.

Accelerated 3-Year MD Pathway Programs: Graduates' Perspectives on Education Quality, the Learning Environment, Residency Readiness, Debt, Burnout, and Career Plans.

PURPOSE: To compare perception of accelerated and traditional medical students, with respect to satisfaction with education quality, and the learning environment, residency readiness, burnout, debt, and career plans. METHOD: Customized 2017 and 2018 Medical School Graduation Questionnaires (GQs) were analyzed using independent samples t tests for means and chi-square tests for percentages, comparing responses of accelerated MD program graduates (accelerated pathway [AP] students) from 9 schools with those of non-AP graduates from the same 9 schools and non-AP graduates from all surveyed schools. RESULTS: GQ completion rates for the 90 AP students, 2,573 non-AP students from AP schools, and 38,116 non-AP students from all schools in 2017 and 2018 were 74.4%, 82.3%, and 83.3%, respectively. AP students were as satisfied with the quality of their education and felt as prepared for residency as non-AP students. AP students reported a more positive learning climate than non-AP students from AP schools and from all schools as measured by the student-faculty interaction (15.9 vs 14.4 and 14.3, respectively; P < .001 for both pairwise comparisons) and emotional climate (10.7 vs 9.6 and 9.6, respectively; P = .004 and .003, respectively) scales. AP students had less debt than non-AP students (P < .001), and more planned to care for underserved populations and practice family medicine than non-AP students from AP schools (55.7% vs 33.9% and 37.7% vs 9.4%; P = .002 and < .001, respectively). Family expectations were a more common influence on career plans for AP students than for non-AP students from AP schools and from all schools (26.2% vs 11.3% and 11.7%, respectively; P < .001 for both pairwise comparisons). CONCLUSIONS: These findings support accelerated programs as a potentially important intervention to address workforce shortages and rising student debt without negative impacts on student perception of burnout, education quality, or residency preparedness.

Cultivating a New Generation of Biomedical Entrepreneurs.

In recent years, scientific and technological advances have brought great innovation within the life sciences industry, introducing the need for entrepreneurship training for medical and engineering graduates. With this in mind, Michal Gilon-Yanai, Dr Robert Schneider and their collaborators developed an academic program designed to provide students and faculty members with the skills they need to become successful entrepreneurs. The team of collaborators includes Dr Gabrielle Gold-von Simson, an expert in implementing academic programs, and Dr Colleen Gillespie, who specialises in education, evaluation and dissemination science. Their pioneering program trains students on how to bring new biomedical technologies to the market.

Gasping for air: measuring patient education and activation skillsets in two clinical assessment contexts.

Objective structured clinical examinations (OSCEs) provide a controlled, simulated setting for competency assessments, while unannounced simulated patients (USPs) measure competency in situ or real-world settings. This exploratory study describes differences in primary care residents' skills when caring for the same simulated patient case in OSCEs versus in a USP encounter. Data reported describe a group of residents (n=20) who were assessed following interaction with the same simulated patient case in two distinct settings: an OSCE and a USP visit at our safety-net clinic from 2009 to 2010. In both scenarios, the simulated patient presented as an asthmatic woman with limited understanding of illness management. Residents were rated through a behaviourally anchored checklist on visit completion. Summary scores (mean % well done) were calculated by domain and compared using paired sample t-tests. Residents performed significantly better with USPs on 7 of 10 items and in two of three aggregate assessment domains (p<0.05). OSCE structure may impede assessment of activation and treatment planning skills, which are better assessed in real-world settings. This exploration of outcomes from our two assessments using the same clinical case lays a foundation for future research on variation in situated performance. Using both assessments during residency will provide a more thorough understanding of learner competency.