Using Unannounced Standardized Patients to Assess Clinician Telehealth and Communication Skills at an Urban Student Health Center.

PURPOSE: As the COVID-19 pandemic forced most colleges and universities to go online, student health centers rapidly shifted to telehealth platforms without frameworks for virtual care provision. An urban student health center implemented a needs assessment involving unannounced standardized patients (USPs) to evaluate the integration of a new telehealth workflow and clinicians' virtual communication skills. METHODS: From April to May 2021, USPs conducted two video visits with 12 primary care and four women's health clinicians (N = 16 clinicians; 32 visits). Cases included (1) a 21-year-old female presenting for birth control with a positive Patient Health Questionaire-9 and (2) a 21-year-old male, who vapes regularly, with questions regarding safe sex with men. Clinicians were evaluated using a checklist completed by the USP immediately following the visit and a systematic chart review of the electronic health record. RESULTS: USP feedback indicates most clinicians received high ratings for general communication skills but may benefit from educational intervention in several key telemedicine skills. Clinicians struggled with using nonverbal signals to enrich communication (47% well done), acknowledging emotions (34% well done), and using video for information gathering (34% well done). Low rates of standard screenings (e.g., 63% administered the PHQ-2, <50% asked about alcohol use) suggested protocols for in-person care were not easily incorporated into telehealth practices, and clinicians may benefit from enhanced care team support. Performance reports were shared with clinicians and leadership postvisit. DISCUSSION: Results suggest project design and implementation is scalable and feasible for use at other institutions, offering a structured methodology that can improve general student health care.

Implementing an Experiential Telehealth Training and Needs Assessment for Residents and Faculty at a Veterans Affairs Primary Care Clinic.

Background: The transition to telehealth during the COVID-19 pandemic revealed a lack of preexisting telehealth training for clinicians. As a workplace-based simulation methodology designed to improve virtual clinical skills, announced standardized patients (ASPs) may help meet evolving educational needs to sustain quality telehealth care. Objective: We describe the development and implementation of an ASP program to assess and provide feedback to resident and faculty clinicians in virtual practice, and report on performance, feasibility, and acceptability. Methods: From June 2021 to April 2022, resident and faculty clinicians at a VA primary care clinic participated in a video visit in which an ASP portrayed either a 70-year-old man with hearing loss and hypertension or a 60-year-old man with hypertension and financial stress. Following the visit, ASPs provided verbal feedback and completed a behaviorally anchored checklist to rate telehealth and communication skills, chronic disease management, and use of resources. Domain summary scores were calculated as the mean percentage of "well done" items. Participants completed a feedback survey on their experience. Results: Seventy-six televisits (60 primary care residents [postgraduate year 1-3], 16 internal medicine faculty) were conducted from August 2021 to April 2022. Clinicians performed well in communication skills: information gathering (79%, 60 of 76, well done), relationship development (67%, 51 of 76), education and counseling (71%, 54 of 76), and patient satisfaction (86%, 65 of 76). They performed less well in telemedicine skills (38%, 29 of 76). Participants agreed that the experience was a good use of their time (88%, 67 of 76). Conclusions: An ASP-facilitated training for resident and faculty clinicians assessed telehealth skills and clinical practice and identified areas for intervention. Clinicians responded well to the training and feedback.

Educational training to improve opioid overdose response among health center staff: a quality improvement initiative.

BACKGROUND: There were seven opioid overdoses in this New York City (NYC) federally qualified health center from December 2018 through February 2019, reflecting the rising rate of overdose deaths in NYC overall at the time. In response to these overdoses, we sought to increase the readiness of health center staff to recognize and respond to opioid overdoses and decrease stigmatizing attitudes around opioid use disorder (OUD). METHODS: An hour-long training focusing on opioid overdose response was administered to clinical and non-clinical staff of all levels at the health center. This training included didactic education on topics such as the overdose epidemic, stigma around OUD, and opioid overdose response, as well as discussion. A structured assessment was administered immediately before and following the training to evaluate change in knowledge and attitudes. Additionally, participants completed a feedback survey immediately after the training to assess acceptability. Paired t-tests and analysis of variance tests were used to assess changes in pre- and post-test scores. RESULTS: Over 76% of the health center staff participated in the training (N = 310). There were large and significant increases in mean knowledge and attitudinal scores from pre- to post-test (p < .001 and p < .001, respectively). While there was no significant effect of profession on attitudinal change scores, profession did have a significant effect on knowledge change scores, with administrative staff, non-clinical support staff, other healthcare staff, and therapists learning significantly more than providers (p < .001). The training had high acceptability among participants from diverse departments and levels. CONCLUSIONS: An interactive educational training increased staff's knowledge and readiness to respond to an overdose as well as improved attitudes toward individuals living with OUD. TRIAL REGISTRATION: This project was undertaken as a quality improvement initiative at the health center and as such was not formally supervised by the Institutional Review Board per their policies. Further, per the guidelines of the International Committee of Medical Journal Editors, registration is not necessary for clinical trials whose sole purpose is to assess an intervention's effect on providers.

Comparison of Primary Care Patients' and Unannounced Standardized Patients' Perceptions of Care.

The objective of this study was to compare unannounced standardized patient (USP) and patient reports of care. Patient satisfaction surveys and USP checklist results collected at an urban, public hospital were compared to identify items included in both surveys. Qualitative commentary was reviewed to better understand USP and patient satisfaction survey data. Analyses included χ2 and Mann-Whitney U test. Patients provided significantly higher ratings on 10 of the 11 items when compared to USPs. USPs may provide a more objective perspective on a clinical encounter than a real patient, reinforcing the notion that real patients skew overly positive or negative.

Understanding medical student paths to communication skills expertise using latent profile analysis.

PURPOSE: To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. MATERIALS AND METHODS: Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. RESULTS: Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles. CONCLUSION: Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

Addressing social determinants of health in primary care: a quasi-experimental study using unannounced standardised patients to evaluate the impact of audit/feedback on physicians' rates of identifying and responding to social needs.

BACKGROUND: Although efforts are underway to address social determinants of health (SDOH), little is known about physicians' SDOH practices despite evidence that failing to fully elicit and respond to social needs can compromise patient safety and undermine both the quality and effectiveness of treatment. In particular, interventions designed to enhance response to social needs have not been assessed using actual practice behaviour. In this study, we evaluate the degree to which providing primary care physicians with feedback on their SDOH practice behaviours is associated with increased rates of eliciting and responding to housing and social isolation needs. METHODS: Unannounced standardised patients (USPs), actors trained to consistently portray clinical scenarios, were sent, incognito, to all five primary care teams in an urban, safety-net healthcare system. Scenarios involved common primary care conditions and each included an underlying housing (eg, mould in the apartment, crowding) and social isolation issue and USPs assessed whether the physician fully elicited these needs and if so, whether or not they addressed them. The intervention consisted of providing physicians with audit/feedback reports of their SDOH practices, along with brief written educational material. A prepost comparison group design was used to evaluate the intervention; four teams received the intervention and one team served as a 'proxy' comparison (no intervention). Preintervention (February 2017 to December 2017) rates of screening for and response to the scripted housing and social needs were compared with intervention period (January 2018 to March 2019) rates for both intervention and comparison teams. RESULTS: 108 visits were completed preintervention and 183 during the intervention period. Overall, social needs were not elicited half of the time and fully addressed even less frequently. Rates of identifying the housing issue increased for teams that received audit/feedback reports (46%-60%; p=0.045) and declined for the proxy comparison (61%-42%; p=0.174). Rates of responding to housing needs increased significantly for intervention teams (15%-41%; p=0.004) but not for the comparison team (21%-29%; p=0.663). Social isolation was identified more frequently postintervention (53%) compared with baseline (39%; p=0.041) among the intervention teams but remained unchanged for the comparison team (39% vs 32%; p=0.601). Full exploration of social isolation remained low for both intervention and comparison teams. CONCLUSIONS: Results suggest that physicians may not be consistently screening for or responding to social needs but that receiving feedback on those practices, along with brief targeted education, can improve rates of SDOH screening and response.

Does it get better? An ongoing exploration of physician experiences with and acceptance of telehealth utilization.

INTRODUCTION: COVID-19 forced health systems to rapidly implement telehealth for routine practice, often without sufficient training or standards. We conducted a longitudinal survey of physicians to explore changes in their perceptions of the challenges and benefits of telehealth and identify recommendations for future practice. METHODS: An anonymous online survey was distributed to a cohort of internal medicine physicians in May to June 2020 and March to June 2021. Changes in responses between 2020 and 2021 and by site (private vs. public) were described. These findings, along with those of a thematic analysis of open-ended responses to questions on telehealth experiences, informed a set of recommendations. RESULTS: The survey response rate was 111/391 in 2021 compared to 122/378 in 2020. Fewer physicians reported that telehealth was more difficult than in person with regards to taking a history (49% in 2020, 33% in 2021, p= 0.015), maximizing patient adherence (33% in 2020, 19% in 2021, p = 0.028), and maintaining patient relationships (31%, 25%, p = 0.009) in 2021. Similar proportions of physicians reported continued challenges with building new patient relationships (75%, 77%, p = 0.075), educating patients (40%, 32%, p = 0.393), and working collaboratively with their team (38%, 41%, p = 0.794). Physicians reported increased satisfaction with tele-visits over in-person visits (13%, 27%, p = 0.006) and less worry over doing future tele-visits (45%, 31%, p = 0.027). Physicians' open-ended responses identified recommendations for further improving the design and use of telehealth. DISCUSSION: Results suggest that physician experience with telehealth improved but opportunities for training and improved integration remain. Longitudinal assessment can deepen understanding of the evolution of telehealth care.

A novel simulation-based approach to training for recruitment of older adults to clinical trials.

BACKGROUND: The need to engage adults, age 65 and older, in clinical trials of conditions typical in older populations, (e.g. hypertension, diabetes mellitus, Alzheimer's disease and related dementia) is exponentially increasing. Older adults have been markedly underrepresented in clinical trials, often exacerbated by exclusionary study criteria as well as functional dependencies that preclude participation. Such dependencies may further exacerbate communication challenges. Consequently, the evidence of what works in subject recruitment is less generalizable to older populations, even more so for those from racial and ethnic minority and low-income communities. METHODS: To support capacity of research staff, we developed a virtual, three station simulation (Group Objective Structured Clinical Experience-GOSCE) to teach research staff communication skills. This 2-h course included a discussion of challenges in recruiting older adults; skills practice with Standardized Participants (SPs) and faculty observer who provided immediate feedback; and debrief to highlight best practices. Each learner had opportunities for active learning and observational learning. Learners completed a retrospective pre-post survey about the experience. SP completed an 11-item communication checklist evaluating the learner on a series of established behaviorally anchored communication skills (29). RESULTS: In the research staff survey, 92% reported the overall activity taught them something new; 98% reported it provided valuable feedback; 100% said they would like to participate again. In the SP evaluation there was significant variation: the percent well-done of items by case ranged from 25-85%. CONCLUSIONS: Results from this pilot suggest that GOSCEs are a (1) acceptable; (2) low cost; and (3) differentiating mechanism for training and assessing research staff in communication skills and structural competency necessary for participant research recruitment.

Fasting Serum IGFBP-1 as a Marker of Insulin Resistance in Diverse School Age Groups.

Introduction: The known markers of insulin resistance in obese children are well studied. However, they require serial measurements and complicated calculations. The objective is to study IGFBP-1 and its relation with other known risk measures. Materials and Methods: The study included 98 New York City school students of diverse ethnic/racial backgrounds (57 males and 41 females), 11-15 years of age. Subjects were enrolled in a cross-sectional study, and anthropometric measures were collected. They underwent fasting intravenous glucose tolerance tests (IVGTT), and glucose, insulin, lipids, IGFBP-1, adiponectin and inflammatory markers were collected. Results: The subjects were stratified into 3 groups based upon the BMI Z-score. Out of all the subjects, 65.3% were in the group with a BMI Z-score <1 SDS, 16.3% subjects were in the group with a BMI Z-score of 1 to 2 SDS, and 18.4% of the subjects were in the group with a BMI Z-score of more than 2 SDS. The group with a BMI Z-score of more than 2 SDS had increased waist circumference (WC), body fat, increased fasting insulin, and triglycerides (TG). This group had decreased levels of adiponectin and HDL and low IGFBP-1 as compared to the group with BMI <1 SDS. The group with a BMI Z-score of 1 to 2 SDS had a decreased level of IGFBP-1 as compared to the group with a BMI Z-score less than 1 SDS. IGFBP-1 inversely correlated with age, WC, BMI, body fat, TG, and insulin levels. IGFBP-1 positively correlated with adiponectin and HDL levels. Conclusion: IGFBP-1 in children can identify the presence of insulin resistance in the group with BMI 1 to 2 SDS, even before the known markers of insulin resistance such as elevated triglycerides and even before decreased HDL and adiponectin levels are identified.

The Telemedicine Takeover: Lessons Learned During an Emerging Pandemic.

Introduction: Health care systems rose to the challenges of COVID-19 by creating or expanding telehealth programs to ensure that patients could access care without an in-person appointment. Traditionally, physicians receive limited formal telemedicine training, making preparedness for this transition uneven. To describe challenges to and attitudes toward providing virtual patient care, we distributed a survey to physicians within our system who largely had no formal prior training/experience with telemedicine, but transitioned to routine telemedicine use. Data collected are then used to offer actionable recommendations for health system leaders and medical educators. Materials and Methods: Surveys were distributed to all faculty outpatient general internal medicine physicians working at any New York University Langone Health, New York City Health + Hospitals/Bellevue and Gouverneur, and the VA NY Harbor Health System (n = 378) in mid-2020. Survey items consisted of Likert and open-ended questions related to experience with televisits (13 items) and attitudes toward care (24 items). Results: Telehealth-related challenges varied by site and modality. Primary challenges included establishing a connection from the patient's (98%) or physician's end (84%) and difficulty in the following domains: working with team members (39%), physical examinations (95%), establishing new patient relationships (70%), and taking history (40%), among others. In thematic analysis, significant themes with illustrative qualitative commentary emerged related to technological challenges, new systems issues, and new patient/physician communication experiences. Discussion: Experience differences were rooted in the type of technology employed. Safety-net practices conducted mostly telephonic visits, whereas private outpatient sites utilized video, despite both using identical electronic medical records. As we consider a "new normal" and prolonged community transmission of COVID-19 infection, it is essential to establish telemedicine training, tools, and protocols that meet the needs of both patients and physicians.

Teaching and Assessing Communication Skills in Pediatric Residents: How Do Parents Think We Are Doing?

OBJECTIVE: Curricula designed to teach and assess the communication skills of pediatric residents variably integrates the parent perspective. We compared pediatric residents' communication skills in an objective structured clinical exam (OSCE) case as assessed by Family Faculty (FF), parents of pediatric patients, versus standardized patients (SP). METHODS: Residents participated in an OSCE case with a SP acting as a patient's parent. We compared resident performance as assessed by FF and SP with a behaviorally-anchored checklist. Items were rated as not done, partly done or well done, with well-done indicating mastery. The residents evaluated the experience. RESULTS: 42 residents consented to study participation. FF assessed a lower percentage of residents as demonstrating skill mastery as compared to SP in 19 of the 23 behaviors. There was a significant difference between FF and SP for Total Mastery Score and Mastery of the Competency Scores in three domains (Respect and Value, Information Sharing and Participation in Care and Decision Making). The majority of residents evaluated the experience favorably. CONCLUSION: Involving parents of pediatric patients in the instructive and assessment components of a communication curriculum for pediatric residents adds a unique perspective and integrates the true stakeholders in parent-physician communication.

Gasping for air: measuring patient education and activation skillsets in two clinical assessment contexts.

Objective structured clinical examinations (OSCEs) provide a controlled, simulated setting for competency assessments, while unannounced simulated patients (USPs) measure competency in situ or real-world settings. This exploratory study describes differences in primary care residents' skills when caring for the same simulated patient case in OSCEs versus in a USP encounter. Data reported describe a group of residents (n=20) who were assessed following interaction with the same simulated patient case in two distinct settings: an OSCE and a USP visit at our safety-net clinic from 2009 to 2010. In both scenarios, the simulated patient presented as an asthmatic woman with limited understanding of illness management. Residents were rated through a behaviourally anchored checklist on visit completion. Summary scores (mean % well done) were calculated by domain and compared using paired sample t-tests. Residents performed significantly better with USPs on 7 of 10 items and in two of three aggregate assessment domains (p<0.05). OSCE structure may impede assessment of activation and treatment planning skills, which are better assessed in real-world settings. This exploration of outcomes from our two assessments using the same clinical case lays a foundation for future research on variation in situated performance. Using both assessments during residency will provide a more thorough understanding of learner competency.

A novel method of assessing clinical preparedness for COVID-19 and other disasters.

QUALITY ISSUE: The emergence of coronavirus disease of 2019 (COVID-19) highlights the necessity of rapidly identifying and isolating potentially infected individuals. Evaluating this preparedness requires an assessment of the full clinical system, from intake to isolation. INITIAL ASSESSMENT: Unannounced Standardized Patients (USPs) present a nimble, sensitive methodology for assessing this readiness. CHOICE OF SOLUTION: Pilot the USP methodology, which employs an actor trained to present as a standardized, incognito potentially infected patient, to assess clinical readiness for potential COVID-19 patients at an urban, community safety-net clinic. IMPLEMENTATION: The USP was trained to present at each team's front desk with the complaint of feeling unwell (reporting a fever of 101 degrees Fahrenheit in the past 24 hours) and exposure to a roommate recently returned from Beijing. The USP was trained to complete a behaviorally anchored assessment of the care she received from the clinical system. EVALUATION: There was clear variation in care USPs received; some frontline clerical staff followed best practices; others did not. Signage and information on disease spread prevention publicly available was inconsistent. Qualitative comments shared by the USPs and those gathered during group debrief reinforced the experiences of the USPs and hospital leadership. LESSONS LEARNED: USPs revealed significant variation in care practices within a clinical system. Utilization of this assessment methodology can provide just-in-time clinical information about readiness and safety practices, particularly during emerging outbreaks. USPs will prove especially powerful as clinicians and systems return to outpatient visits while remaining vigilant about potentially infected individuals.

Clinical problem solving and social determinants of health: a descriptive study using unannounced standardized patients to directly observe how resident physicians respond to social determinants of health.

Objectives While the need to address patients' social determinants of health (SDoH) is widely recognized, less is known about physicians' actual clinical problem-solving when it comes to SDoH. Do physicians include SDoH in their assessment strategy? Are SDoH incorporated into their diagnostic thinking and if so, do they document as part of their clinical reasoning? And do physicians directly address SDoH in their "solution" (treatment plan)? Methods We used Unannounced Standardized Patients (USPs) to assess internal medicine residents' clinical problem solving in response to a patient with asthma exacerbation and concern that her moldy apartment is contributing to symptoms - a case designed to represent a clear and direct link between a social determinant and patient health. Residents' clinical practices were assessed through a post-visit checklist and systematic chart review. Patterns of clinical problem solving were identified and then explored, in depth, through review of USP comments and history of present illness (HPI) and treatment plan documentation. Results Residents fell into three groups when it came to clinical problem-solving around a housing trigger for asthma: those who failed to ask about housing and therefore did not uncover mold as a potential trigger (neglectors ­ 21%; 14/68); those who asked about housing in negative ways that prevented disclosure and response (negative elicitors ­ 23%, 16/68); and those who elicited and explored the mold issue (full elicitors ­ 56%; 38/68) [corrected]. Of the full elicitors 53% took no further action, 26% only documented the mold; and 21% provided resources/referral. In-depth review of USP comments/explanations and residents' notes (HPI, treatment plan) revealed possible influences on clinical problem solving. Failure to ask about housing was associated with both contextual factors (rushed visit) and interpersonal skills (not fully engaging with patient) and with possible differences in attention ("known" vs. unknown/new triggers, usual symptoms vs. changes, not attending to relocation, etc.,). Use of close-ended questions often made it difficult for the patient to share mold concerns. Negative responses to sharing of housing information led to missing mold entirely or to the patient not realizing that the physician agreed with her concerns about mold. Residents who fully elicited the mold situation but did not take action seemed to either lack knowledge or feel that action on SDoH was outside their realm of responsibility. Those that took direct action to help the patient address mold appeared to be motivated by an enhanced sense of urgency. Conclusions Findings provide unique insight into residents' problem solving processes including external influences (e.g., time, distractions), the role of core communication and interpersonal skills (eliciting information, creating opportunities for patients to voice concerns, sharing clinical thinking with patients), how traditional cognitive biases operate in practice (premature closure, tunneling, and ascertainment bias), and the ways in which beliefs about expectancies and scope of practice may color clinical problem-solving strategies for addressing SDoH.

Development and maintenance of a medical education research registry.

BACKGROUND: Medical Education research suffers from several methodological limitations including too many single institution, small sample-sized studies, limited access to quality data, and insufficient institutional support. Increasing calls for medical education outcome data and quality improvement research have highlighted a critical need for uniformly clean and easily accessible data. Research registries may fill this gap. In 2006, the Research on Medical Education Outcomes (ROMEO) unit of the Program for Medical Innovations and Research (PrMEIR) at New York University's (NYU) Robert I. Grossman School of Medicine established the Database for Research on Academic Medicine (DREAM). DREAM is a database of routinely collected, de-identified undergraduate (UME, medical school leading up to the Medical Doctor degree) and graduate medical education (GME, residency also known as post graduate education leading to eligibility for specialty board certification) outcomes data available, through application, to researchers. Learners are added to our database through annual consent sessions conducted at the start of educational training. Based on experience, we describe our methods in creating and maintaining DREAM to serve as a guide for institutions looking to build a new or scale up their medical education registry. RESULTS: At present, our UME and GME registries have consent rates of 90% (n = 1438/1598) and 76% (n = 1988/2627), respectively, with a combined rate of 81% (n = 3426/4225). 7% (n = 250/3426) of these learners completed both medical school and residency at our institution. DREAM has yielded a total of 61 individual studies conducted by medical education researchers and a total of 45 academic journal publications. CONCLUSION: We have built a community of practice through the building of DREAM and hope, by persisting in this work the full potential of this tool and the community will be realized. While researchers with access to the registry have focused primarily on curricular/ program evaluation, learner competency assessment, and measure validation, we hope to expand the output of the registry to include patient outcomes by linking learner educational and clinical performance across the UME-GME continuum and into independent practice. Future publications will reflect our efforts in reaching this goal and will highlight the long-term impact of our collaborative work.