Reliability of Multiple Mini-Interviews and traditional interviews within and between institutions: a study of five California medical schools.

BACKGROUND: Many medical schools use admissions Multiple Mini-Interviews (MMIs) rather than traditional interviews (TIs), partly because MMIs are thought to be more reliable. Yet prior studies examined single-school samples of candidates completing either an MMI or TI (not both). Using data from five California public medical schools, the authors examined the within- and between-school reliabilities of TIs and MMIs. METHODS: The analyses included applicants interviewing at ≥1 of the five schools during 2011-2013. Three schools employed TIs (TI1, TI2, TI3) and two employed MMIs (MMI1, MMI2). Mixed linear models accounting for nesting of observations within applicants examined standardized TI and MMI scores (mean = 0, SD = 1), adjusting for applicant socio-demographics, academic metrics, year, number of interviews, and interview date. RESULTS: A total of 4993 individuals (completing 7516 interviews [TI = 4137, MMI = 3379]) interviewed at ≥1 school; 428 (14.5%) interviewed at both MMI schools and 687 (20.2%) at more than one TI school. Within schools, inter-interviewer consistency was generally qualitatively lower for TI1, TI2, and TI3 (Pearson's r 0.07, 0.13, and 0.29, and Cronbach's α, 0.40, 0.44, and 0.61, respectively) than for MMI1 and MMI 2 (Cronbach's α 0.68 and 0.60, respectively). Between schools, the adjusted intraclass correlation coefficient was 0.27 (95% CI 0.20-0.35) for TIs and 0.47 (95% CI 0.41-0.54) for MMIs. CONCLUSIONS: Within and between-school reliability was qualitatively higher for MMIs than for TIs. Nonetheless, TI reliabilities were higher than anticipated from prior literature, suggesting TIs may not need to be abandoned on reliability grounds if other factors favor their use.

Physician communication regarding prostate cancer screening: analysis of unannounced standardized patient visits.

PURPOSE: Prostate cancer screening with prostate-specific antigen (PSA) is a controversial issue. The present study aimed to explore physician behaviors during an unannounced standardized patient encounter that was part of a randomized controlled trial to educate physicians using a prostate cancer screening, interactive, Web-based module. METHODS: Participants included 118 internal medicine and family medicine physicians from 5 health systems in California, in 2007-2008. Control physicians received usual education about prostate cancer screening (brochures from the Center for Disease Control and Prevention). Intervention physicians participated in the prostate cancer screening module. Within 3 months, all physicians saw unannounced standardized patients who prompted prostate cancer screening discussions in clinic. The encounter was audio-recorded, and the recordings were transcribed. Authors analyzed physician behaviors around screening: (1) engagement after prompting, (2) degree of shared decision making, and (3) final recommendations for prostate cancer screening. RESULTS: After prompting, 90% of physicians discussed prostate cancer screening. In comparison with control physicians, intervention physicians showed somewhat more shared decision making behaviors (intervention 14 items vs control 11 items, P <.05), were more likely to mention no screening as an option (intervention 63% vs control 26%, P <.05), to encourage patients to consider different screening options (intervention 62% vs control 39%, P <.05) and seeking input from others (intervention 25% vs control 7%, P<.05). CONCLUSIONS: A brief Web-based interactive educational intervention can improve shared decision making, neutrality in recommendation, and reduce PSA test ordering. Engaging patients in discussion of the uses and limitations of tests with uncertain value can decrease utilization of the tests.

Pairing physician education with patient activation to improve shared decisions in prostate cancer screening: a cluster randomized controlled trial.

BACKGROUND: Most expert groups recommend shared decision making for prostate cancer screening. Most primary care physicians, however, routinely order a prostate-specific antigen (PSA) test with little or no discussion about whether they believe the potential benefits justify the risk of harm. We sought to assess whether educating primary care physicians and activating their patients to ask about prostate cancer screening had a synergistic effect on shared decision making, rates and types of discussions about prostate cancer screening, and the physician's final recommendations. METHODS: Our study was a cluster randomized controlled trial among primary care physicians and their patients, comparing usual education (control), with physician education alone (MD-Ed), and with physician education and patient activation (MD-Ed+A). Participants included 120 physicians in 5 group practices, and 712 male patients aged 50 to 75 years. The interventions comprised a Web-based educational program for all intervention physicians and MD-Ed+A patients compared with usual education (brochures from the Centers for Disease Control and Prevention). The primary outcome measure was patients' reported postvisit shared decision making regarding prostate cancer screening; secondary measures included unannounced standardized patients' reported shared decision making and the physician's recommendation for prostate cancer screening. RESULTS: Patients' ratings of shared decision making were moderate and did not differ between groups. MD-Ed+A patients reported that physicians had higher prostate cancer screening discussion rates (MD-Ed+A = 65%, MD-Ed = 41%, control=38%; P <.01). Standardized patients reported that physicians seeing MD-Ed+A patients were more neutral during prostate cancer screening recommendations (MD-Ed+A=50%, MD-Ed=33%, control=15%; P <.05). Of the male patients, 80% had had previous PSA tests. CONCLUSIONS: Although activating physicians and patients did not lead to significant changes in all aspects of physician attitudes and behaviors that we studied, interventions that involved physicians did have a large effect on their attitudes toward screening and in the discussions they had with patients, including their being more likely than control physicians to engage in prostate cancer screening discussions and more likely to be neutral in their final recommendations.

Screening mammography beliefs and recommendations: a web-based survey of primary care physicians.

BACKGROUND: The appropriateness and cost-effectiveness of screening mammography (SM) for women younger than 50 and older than 74 years is debated in the clinical research community, among health care providers, and by the American public. This study explored primary care physicians' (PCPs) perceptions of the influence of clinical practice guidelines for SM; the recommendations for SM in response to hypothetical case scenarios; and the factors associated with perceived SM effectiveness and recommendations in the US from June to December 2009 before the United States Preventive Services Task Force (USPSTF) recently revised guidelines. METHODS: A nationally representative sample of 11,922 PCPs was surveyed using a web-based questionnaire. The response rate was 5.7% (684); (41%) 271 family physicians (FP), (36%) 232 general internal medicine physicians (IM), (23%) 150 obstetrician-gynaecologists (OBG), and (0.2%) 31 others. Cross-sectional analysis examined PCPs perceived effectiveness of SM, and recommendation for SM in response to hypothetical case scenarios. PCPs responses were measured using 4-5 point adjectival scales. Differences in perceived effectiveness and recommendations for SM were examined after adjusting for PCPs specialty, race/ethnicity, and the US region. RESULTS: Compared to IM and FP, OBG considered SM more effective in reducing breast cancer mortality among women aged 40-49 years (p = 0.003). Physicians consistently recommended mammography to women aged 50-69 years with no differences by specialty (p = 0.11). However, 94% of OBG "always recommended" SM to younger and 86% of older women compared to 81% and 67% for IM and 84% and 59% for FP respectively (p = < .001). In ordinal regression analysis, OBG specialty was a significant predictor for perceived higher SM effectiveness and recommendations for younger and older women. In evaluating hypothetical scenarios, overall PCPs would recommend SM for the 80 year woman with CHF with a significant variation by specialty (38% of OBG, 18% of FP, 17% of IM; p = < .001). CONCLUSIONS: A majority of physicians, especially OBG, favour aggressive breast cancer screening for women from 40 through 79 years of age, including women with short life expectancy. Policy interventions should focus on educating providers to provide tailored recommendations for mammography based on individualized cancer risk, health status, and preferences.

Building an institutional K awardee program at UC Davis through utilization of CTSA resources.

NIH offers multiple mentored career development award mechanisms. By building on the UC Davis Clinical and Translational Science Center (CTSC) from its initial NIH funding in 2006, we created an institution-wide K scholar resource. We investigated subsequent NIH funding for K scholars and to what extent CTSC research resources were used. Using NIH RePORTER, we created a database of UC Davis investigators who obtained K01, K08, K23, K25, or K99, as well as institutional KL2 or K12 awards and tracked CTSC research resource use and subsequent funding success. Overall, 94 scholars completed K training between 2007 and 2020, of which 70 participated in one of four institutional, NIH-funded K programs. An additional 103 scholars completed a mentored clinical research training program. Of 94 K awardees, 61 (65%) later achieved NIH funding, with the majority receiving a subsequent individual K award. A higher proportion (73%) of funded scholars used CTSC resources compared to unfunded (48%). Biostatistics and Biomedical Informatics were most commonly used and 55% of scholars used one or more CTSC resource. We conclude that institutional commitment to create a K scholar platform and use of CTSC research resources is associated with high NIH funding rates for early career investigators.

Wildfire Smoke Exposure: Awareness and Safety Responses in the Agricultural Workplace.

Objective: The study examines how wildfire smoke exposure may impact health and safety in the agricultural workplace. Methods: Semi-structured interviews were conducted with agricultural employers and focus group discussions were held with farmworkers in three regions of California. Results: Agricultural employers had varying knowledge about and experience responding to poor air quality due to wildfire smoke. Respirators or masks were not mentioned as a potential protective measure when describing their safety practices. Farmworkers reported experiencing poor air quality due to wildfire smoke, although knowledge of safety precautions varied. Farmworkers reported employer and supervisors' attitudes toward safety as having the greatest impact on the implementation of workplace safety measures. Conclusion: Adapting health promotion and workplace safety strategies to meet the multiple vulnerabilities and diverse needs of farmworkers is critical to successful implementation of workplace protection and safety measures. Given limited familiarity with the topic, wildfire smoke exposure resources are needed to assist employers and supervisors in their compliance with a new wildfire smoke safety regulation in California. To the best of our knowledge, this is the first study to explore agricultural employer and farmworker perceptions of the health and safety impacts of wildfire smoke and workplace exposure.

Erratum: Opportunities for strengthening CTSA evaluation - CORRIGENDUM.

[This corrects the article DOI: 10.1017/cts.2019.387.].

Exploration of Mistreatment and Burnout Among Resident Physicians: a Cross-Specialty Observational Study.

PURPOSE: Resident physician mistreatment and burnout are widespread issues in medical training, but the association between the two remains unclear. This study examines the prevalence and types of mistreatment among resident physicians in core specialties and its association with burnout syndrome as well as feelings of depression/anxiety. METHODS: A cross-sectional, survey-based observational study of medical residents was conducted at the University of California, Davis Medical Center in 2014. Current residents (PGY2 or higher) in the internal medicine, family medicine, obstetrics/gynecology, surgery, and pediatrics programs completed anonymous questionnaires addressing topics such as workplace mistreatment, feelings of depression/anxiety, and stress management. Burnout was measured using the Maslach Burnout Inventory. RESULTS: Forty-four out of 105 residents (41.9%) witnessed mistreatment of their co-residents while 26 residents (24.8%) disclosed personal accounts of mistreatment. Seventy-one percent of residents met the criteria for burnout. Residents who had been personally mistreated were almost eight times more likely to report burnout (OR 7.6, 95% CI = 1.7-34.4) and almost four times more likely to report symptoms of anxiety and depression (OR 3.8, 95% CI = 1.6-9.1). Public belittlement or humiliation was the most common type of mistreatment. CONCLUSION: Encountering mistreatment was associated with higher rates of burnout, as well as depression/anxiety. While it is uncertain if mistreatment in the workplace has a causative impact on burnout syndrome, the findings reveal the need to address work-related environmental factors that may contribute to both resident physician mistreatment and burnout.

I-Corps@NCATS trains clinical and translational science teams to accelerate translation of research innovations into practice.

INTRODUCTION: A key barrier to translation of biomedical research discoveries is a lack of understanding among scientists regarding the complexity and process of implementation. To address this challenge, the National Science Foundation's Innovation Corps™ (I-Corps™) program trains researchers in entrepreneurship. We report results from the implementation of an I-Corps™ training program aimed at biomedical scientists from institutions funded by the National Center for Advancing Translational Sciences (NCATS). METHODS: National/regional instructors delivered 5-week I-Corps@NCATS short courses to 62 teams (150 individuals) across six institutions. Content included customer discovery, value proposition, and validating needs. Teams interviewed real-life customers and presented the value of innovations for specific end-users weekly, culminating in a "Finale" featuring their refined business thesis and business model canvas. Methodology was developed to evaluate the newly adapted program. National mixed-methods evaluation assessed program implementation, reach, effectiveness using observations of training delivery and surveys at Finale (n = 55 teams), and 3-12 months post-training (n = 34 teams). RESULTS: Innovations related to medical devices (33%), drugs/biologics (20%), software applications (16%), and diagnostics (8%). An average of 24 interviews was conducted. Teams reported increased readiness for commercialization over time (83%, 9 months; 14%, 3 months). Thirty-nine percent met with institutional technology transfer to pursue licensing/patents and 24% pursued venture capital/investor funding following the short courses. CONCLUSIONS: I-Corps@NCATS training provided the NCATS teams a rigorous and repeatable process to aid development of a business model based on customer needs. Outcomes of this pilot program support the expansion of I-Corps™ training to biomedical scientists for accelerating research translation.

Opportunities for strengthening CTSA evaluation.

The purpose of the article is to describe the progress of the Clinical and Translational Science Award (CTSA) Program to address the evaluation-related recommendations made by the 2013 Institute of Medicine's review of the CTSA Program and guidelines published in CTS Journal the same year (Trochim et al., Clinical and Translational Science 2013; 6(4): 303-309). We utilize data from a 2018 national survey of evaluators administered to all 64 CTSA hubs and a content analysis of the role of evaluation in the CTSA Program Funding Opportunity Announcements to document progress. We present four new opportunities for further strengthening CTSA evaluation efforts: (1) continue to build the collaborative evaluation infrastructure at local and national levels; (2) make better use of existing data; (3) strengthen and augment the common metrics initiative; and (4) pursue internal and external opportunities to evaluate the CTSA program at the national level. This article will be of significant interest to the funders of the CTSA Program and the multiple stakeholders in the larger consortium and will promote dialog from the broad range of CTSA stakeholders about further strengthening the CTSA Program's evaluation.

Building a comprehensive mentoring academy for schools of health.

Formal mentoring programs are increasingly recognized as critical for faculty career development. We describe a mentoring academy (MA) developed for faculty across tracks (i.e., researchers, clinicians, educators) within a "school of health" encompassing schools of medicine and nursing. The program is anchored dually in a clinical and translational science center and a school of health. The structure includes the involvement of departmental and center mentoring directors to achieve widespread uptake and oversight. A fundamental resource provided by the MA includes providing workshops to enhance mentoring skills. Initiatives for junior faculty emphasize establishing and maintaining strong mentoring relationships and implementing individual development plans (IDPs) for career planning. We present self-report data on competency improvement from mentor workshops and data on resources and barriers identified by junior faculty (n = 222) in their IDPs. Mentors reported statistically significantly improved mentoring competency after workshop participation. Junior faculty most frequently identified mentors (61%) and collaborators (23%) as resources for goal attainment. Top barriers included insufficient time and time-management issues (57%), funding limitations (18%), work-life balance issues (18%), including inadequate time for self-care and career development activities. Our MA can serve as a model and roadmap for providing resources to faculty across traditional tracks within medical schools.

Medical School Applicant Characteristics Associated With Performance in Multiple Mini-Interviews Versus Traditional Interviews: A Multi-Institutional Study.

PURPOSE: To examine applicant characteristics associated with multiple mini-interview (MMI) or traditional interview (TI) performance at five California medical schools. METHOD: Of the five California Longitudinal Evaluation of Admission Practices consortium schools, three used TIs and two used MMIs. Schools provided retrospective data on 2011-2013 admissions cycle interviewees: age, gender, race/ethnicity (underrepresented in medicine [UIM] or not), disadvantaged (DA) status, undergraduate GPA, Medical College Admission Test (MCAT) score, and interview score (standardized as z score; mean = 0; SD = 1). Adjusted linear regression analyses, stratified by interview type, examined associations with interview performance. RESULTS: The 4,993 applicants who completed 7,516 interviews included 931 (18.6%) UIM and 962 (19.3%) DA individuals; 3,226 (64.6%) had only 1 interview. Mean age was 24.4 (SD = 2.7); mean GPA and MCAT score were 3.72 (SD = 0.22) and 33.6 (SD = 3.7), respectively. Older age, female gender, and number of prior interviews were associated with better performance on both MMIs and TIs. Higher GPA was associated with lower MMI scores (z score, per unit GPA = -0.26; 95% CI = -0.45, -0.06) but unrelated to TI scores. DA applicants had higher TI scores (z score = 0.17; 95% CI = 0.07, 0.28) but lower MMI scores (z score = -0.18; 95% CI = -0.28, -0.08) than non-DA applicants. Neither UIM status nor MCAT score was associated with interview performance. CONCLUSIONS: These findings have potentially important workforce implications, particularly regarding MMI performance of DA applicants, and illustrate the need for other multi-institutional studies.

Use of a Needs Assessment in the Development of an Interprofessional Faculty Development Program.

A needs assessment was conducted regarding an interprofessional faculty development program for promoting excellence in education. Nursing and medical faculty and administrators (N = 156) were surveyed about perceived need, program curriculum, and delivery. The results indicated strong support for the program, particularly related to teaching/learning strategies, leadership, and scholarship. Nursing faculty rated some topical areas significantly higher than did the medical faculty, including innovative classroom teaching, educational technology, interprofessional education, diversity/inclusion, and mentoring graduate students.

Training Medical Students for Rural, Underserved Areas: A Rural Medical Education Program in California.

The Association of American Medical Colleges projects an increasing shortage of physicians in rural areas. Medical schools have developed specialty track programs to improve the recruitment and retention of physicians who can serve rural populations. One such program in California includes a variety of unique elements including outreach, admissions, rural clinical experiences, focused mentorship, scholarly and leadership opportunities, and engagement with rural communities. Preliminary outcomes demonstrate that this rural track program has achieved some success in the recruitment, retention, and training of students interested in future rural practice and in the placement of students in primary care residencies. Long-term outcomes, such as graduates entering rural practice, are still unknown, but will be monitored to assess the impact and sustainability of the rural program. This article illustrates the opportunities and challenges of training medical students for rural practice and provides lessons learned to inform newly-established and long standing rural medical education programs.

Developing the Translational Research Workforce: A Pilot Study of Common Metrics for Evaluating the Clinical and Translational Award KL2 Program.

PURPOSE: This pilot study describes the career development programs (i.e., NIH KL2 awards) across five Clinical and Translational Science Award (CTSA) institutions within the University of California (UC) system, and examines the feasibility of a set of common metrics for evaluating early outcomes. METHODS: A survey of program administrators provided data related to the institutional environment within which each KL2 program was implemented. Application and progress report data yielded a combined data set that characterized KL2 awardees, their initial productivity, and early career outcomes. RESULTS: The pilot project demonstrated the feasibility of aggregating common metrics data across multiple institutions. The data indicated that KL2 awardees were an accomplished set of investigators, both before and after the award period, representing a wide variety of disciplines. Awardees that had completed their trainee period overwhelmingly remained active in translational research conducted within an academic setting. Early indications also suggest high rates of success with obtaining research funding subsequent to the KL2 award. CONCLUSION: This project offers a model for how to collect and analyze common metrics related to the education and training function of the CTSA Consortium. Next steps call for expanding participation to other CTSA sites outside of the University of California system.

Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

What data do California HMOs use to select hospitals for contracting?

OBJECTIVE: To explore whether health maintenance organization (HMO) executives in a mature market are familiar with hospital report cards, whether they find the report cards useful (and if not, why not), and how they weight such data relative to other factors. STUDY DESIGN: Cross-sectional survey of HMO executives in 1999. PATIENTS AND METHODS: We contacted all 47 licensed HMOs and the sponsors of all 90 employee medical benefit plans in California with at least 1000 participants. Thirty of the 47 (63.8%) eligible HMOs provided usable responses: 19 in writing, 11 by telephone. RESULTS: HMO executives reported basing their contracting decisions primarily on hospital accreditation, location, and price. Although hospital quality is considered important, HMO executives rely primarily on accreditation, government disciplinary actions, reputation, and member satisfaction as measures of quality. Respondents voiced multiple concerns about the validity and usefulness of currently available process and outcome data. Accredited plans are more likely than unaccredited plans to perform independent analyses of hospital performance. CONCLUSIONS: Although HMO executives are interested in information on hospital quality, and are confident that such information will improve care, they are concerned about the limitations of available data and uncomfortable weighting these data heavily in selecting network hospitals. Prior empirical evidence suggests that HMOs may rely on surrogate quality measures and informal evaluation mechanisms to steer their members toward better-performing hospitals. Policy makers and producers of hospital report cards will need to address these problems by providing more timely data with longitudinal follow-up and external validation.

Can administrative data be used to ascertain clinically significant postoperative complications?

The purpose of this study is to assess whether postoperative complications can be ascertained using administrative data. We randomly sampled 991 adults who underwent elective open diskectomies at 30 nonfederal acute care hospitals in California. Postoperative complications were specified by reviewing medical literature and by consulting clinical experts. We compared hospital-reported ICD-9-CM data and independently recoded ICD-9-CM data with complications abstracted by clinicians using detailed criteria. Recoded ICD-9-CM data were more likely than hospital-reported ICD-9-CM data to capture true complications, when they occurred, but they also mislabeled more patients who never experienced clinically significant complications. This finding was most evident for mild or ambiguous complications, such as atelectasis, posthemorrhagic anemia, and hypotension. Overall, recoded ICD-9-CM data captured 47% and 56% of all mild and severe complications, respectively, whereas hospital-reported ICD-9-CM data captured only 37% and 44%, respectively, of all mild and severe complications. These findings raise questions about the validity of using administrative data to ascertain postoperative complications, even if coders are carefully hired, trained, and supervised. ICD-9-CM complication codes are more promising as a tool to help providers identify their own adverse outcomes than as a tool for comparing performance.

Training the translational research teams of the future: UC Davis-HHMI Integrating Medicine into Basic Science program.

There is a need for successful models of how to recruit, train, and retain bench scientists at the earliest stages of their careers into translational research. One recent, promising model is the University of California Davis Howard Hughes Medical Institute Integrating Medicine into Basic Science (HHMI-IMBS) program, part of the HHMI Med into Grad initiative. This paper outlines the HHMI-IMBS program's logic, design, and curriculum that guide the goal of research that moves from bedside to bench. That is, a curriculum that provides graduate students with guided translational training, clinical exposure, team science competencies, and mentors from diverse disciplines that will advance the students careers in clinical translational research and re-focusing of research to answer clinical dilemmas. The authors have collected data on 55 HHMI-IMBS students to date. Many of these students are still completing their graduate work. In the current study the authors compare the initial two cohorts (15 students) with a group of 29 control students to examine the program success and outcomes. The data indicate that this training program provides an effective, adaptable model for training future translational researchers. HHMI-IMBS students showed improved confidence in conducting translational research, greater interest in a future translational career, and higher levels of research productivity and collaborations than a comparable group of predoctoral students.

State policies and language access in California's HMOs: public reporting and regulation of HMOs' language services.

OBJECTIVE: Describe the status of language access services for California's limited English proficient (LEP) health maintenance organization (HMO) members within the context of voluntary public reporting and anticipated state regulation. STUDY DESIGN: Descriptive analysis of longitudinal data provided by health plans. METHODS: Retrospective, descriptive analysis of responses to voluntary annual/biennial surveys of commercial, Medicare, Medicaid, and Healthy Families HMOs, 2003-2008. RESULTS: From 2003 to 2008, during California's public reporting of HMO language services, the percentage of HMOs providing in-person interpretation free of charge to consumers increased to over 80% across all product lines. By 2008, the percentage of commercial HMO plans matching new LEP members to providers by language increased to 70%, while 79% provided telephone interpretation services in at least seven languages. CONCLUSIONS: Increasingly HMOs provide language access services for LEP Californians, perhaps resulting from public reporting and anticipated state regulation, although similar trends in other states cannot be ruled out.