A landscape assessment of CTSA evaluators and their work in the CTSA consortium, 2021 survey findings.

This article presents a landscape assessment of the findings from the 2021 Clinical and Translational Science Award (CTSA) Evaluators Survey. This survey was the most recent iteration of a well established, national, peer-led systematic snapshot of the CTSA evaluators, their skillsets, listed evaluation resources, preferred methods, and identified best practices. Three questions guided our study: who are the CTSA evaluators, what competencies do they share and how is their work used within hubs. We describe our survey process (logistics of development, deployment, and differences in historical context with prior instruments); and present its main findings. We provide specific recommendations for evaluation practice in two main categories (National vs Group-level) including, among others, the need for a national, strategic plan for evaluation as well as enhanced mentoring and training of the next generation of evaluators. Although based on the challenges and opportunities currently within the CTSA Consortium, takeaways from this study constitute important lessons with potential for application in other large evaluation consortia. To our knowledge, this is the first time 2021 survey findings are disseminated widely, to increase transparency of the CTSA evaluators' work and to motivate conversations within hub and beyond, as to how best to leverage existent evaluative capacity.

Evaluation of the long-term outcomes of the University of Michigan's Practice-Oriented Research Training (PORT) program.

The University of Michigan created the Practice-Oriented Research Training (PORT) program and implemented it between 2008 and 2018. The PORT program provided research training and funding opportunities for allied healthcare professionals. The program consisted of weekly didactics and group discussion related to topics relevant to developing specific research ideas into projects and funding for a mentored research project for those who submitted a competitive grant application. The goal of this evaluation was to assess the long-term impact of the PORT program on the research careers of the participants. Ninety-two participants (74 staff and 18 faculty) participated in both phases of the program. A mixed-methods approach to evaluation was used; 25 participants who received funding for their research completed surveys, and semi-structured interviews were conducted with eight program participants. In addition, data were collected on participants' publication history. Fifteen out of the 74 staff participants published 31 first-authored papers after participating in PORT. Twelve out of 15 staff participants who published first-authored papers did so for the first time after participating in the PORT program. Results of quantitative and qualitative analyses suggest that the PORT program had positive impacts on both participants and the research community.

Updating and evaluating a research best practices training course for social and behavioral research professionals.

Introduction: The clinical and translational research workforce involved in social and behavioral research (SBR) needs to keep pace with clinical research guidance and regulations. Updated information and a new module on community and stakeholder engagement were added to an existing SBR training course. This article presents evaluation findings of the updated course for the Social and Behavioral Workforce. Methods and Materials: Participants working across one university were recruited. Course completers were sent an online survey to evaluate the training. Some participants were invited to join in a focus group to discuss the application of the training to their work. We performed descriptive statistics and conducted a qualitative analysis on focus group data. Results: There were 99 participants from diverse backgrounds who completed the survey. Most reported the training was relevant to their work or that of the study teams they worked with. Almost half (46%) indicated they would work differently after participating. Respondents with community or stakeholder engaged research experience vs. those without were more likely to report that the new module was relevant to study teams they worked with (t = 5.61, p = 0.001), and that they would work differently following the training (t = 2.63, p = 0.01). Open-ended survey responses (n = 99) and focus group (n = 12) data showed how participants felt their work would be affected by the training. Conclusion: The updated course was rated highly, particularly by those whose work was related to the new course content. This course provides an up-to-date resource for the training and development for the Social and Behavioral Workforce.

Implementing a mentoring program for clinical research professionals: A novel professional development initiative for university health research staff.

Clinical and translational research relies on a well-trained workforce, but mentorship programs designed expressly for this workforce are lacking. This paper presents the development of a mentoring program for research staff and identifies key programmatic outcomes. Research staff participating in this program were matched with a senior mentor. Focus groups were conducted to identify key program outcomes. Surveys were administered throughout the program period to assess participants' experience, gains in skill, and subsequent careers. Analysis of the resultant qualitative and quantitative data are used to characterize the implementation and impact of the program. A total of 47 mentees and 30 mentors participated in program between 2018 and 2023. A comprehensive logic model of short-, intermediate- and long-term outcomes was developed. Participants reported positive valuations of every programmatic outcome assessed including their program experience, learning and research careers. The pool of available mentors also grew as new mentors were successfully recruited for each cohort. This mentorship program developed and implemented by senior research staff successfully provided junior research staff with professional development support, mentorship, and professional development opportunities. Junior and senior health research staff built mentoring relationships that advanced their clinical and translational research careers.

Evaluation of an online research best practices training for community health workers and promotoras.

Introduction: Community health workers and promotoras (CHW/Ps) have a fundamental role in facilitating research with communities. However, no national standard training exists as part of the CHW/P job role. We developed and evaluated a culturally- and linguistically tailored online research best practices course for CHW/Ps to meet this gap. Methods: After the research best practices course was developed, we advertised the opportunity to CHW/Ps nationwide to complete the training online in English or Spanish. Following course completion, CHW/Ps received an online survey to rate their skills in community-engaged research and their perceptions of the course using Likert scales of agreement. A qualitative content analysis was conducted on open-ended response data. Results: 104 CHW/Ps completed the English or Spanish course (n = 52 for each language; mean age 42 years SD ± 12); 88% of individuals identified as female and 56% identified as Hispanic, Latino, or Spaniard. 96%-100% of respondents reported improvement in various skills. Nearly all CHW/Ps (97%) agreed the course was relevant to their work, and 96% felt the training was useful. Qualitative themes related to working more effectively as a result of training included enhanced skills, increased resources, and building bridges between communities and researchers. Discussion: The CHW/P research best practices course was rated as useful and relevant by CHW/Ps, particularly for communicating about research with community members. This course can be a professional development resource for CHW/Ps and could serve as the foundation for a national standardized training on their role related to research best practices.

Developing relevant assessments of community-engaged research partnerships: A community-based participatory approach to evaluating clinical and health research study teams.

Background/Objective: In 2017, the Michigan Institute for Clinical and Health Research (MICHR) and community partners in Flint, Michigan collaborated to launch a research funding program and evaluate the dynamics of those research partnerships receiving funding. While validated assessments for community-engaged research (CEnR) partnerships were available, the study team found none sufficiently relevant to conducting CEnR in the context of the work. MICHR faculty and staff along with community partners living and working in Flint used a community-based participatory research (CBPR) approach to develop and administer a locally relevant assessment of CEnR partnerships that were active in Flint in 2019 and 2021. Methods: Surveys were administered each year to over a dozen partnerships funded by MICHR to evaluate how community and academic partners assessed the dynamics and impact of their study teams over time. Results: The results suggest that partners believed that their partnerships were engaging and highly impactful. Although many substantive differences between community and academic partners' perceptions over time were identified, the most notable regarded the financial management of the partnerships. Conclusion: This work contributes to the field of translational science by evaluating how the financial management of community-engaged health research partnerships in a locally relevant context of Flint can be associated with these teams' scientific productivity and impact with national implications for CEnR. This work presents evaluation methods which can be used by clinical and translational research centers that strive to implement and measure their use of CBPR approaches.

Leadership and administration to advance translational science: Environmental scan of adaptive capacity and preparedness of Clinical and Translational Science Award Program hubs.

COVID-19 reinforced the need for effective leadership and administration within Clinical and Translational Science Award (CTSA) program hubs in response to a public health crisis. The speed, scale, and persistent evolution of the pandemic forced CTSA hubs to act quickly and remain nimble. The switch to virtual environments paired with supporting program operations, while ensuring the safety and well-being of their team, highlight the critical support role provided by leadership and administration. The pandemic also illustrated the value of emergency planning in supporting organizations' ability to quickly pivot and adapt. Lessons learned from the pandemic and from other cases of adaptive capacity and preparedness can aid program hubs in promoting and sustaining the overall capabilities of their organizations to prepare for future events.

Parental Perspectives Regarding the Impact of the COVID-19 Pandemic on Their Children.

Background: The COVID-19 pandemic has brought profound changes to the health of families worldwide. Yet, there is limited research regarding its impact on children. The pandemic may exacerbate factors associated with excess weight, which is particularly concerning due to the potential association between excess weight and severity of COVID-19 infection. This study investigates parental perspectives of changes in fruit/vegetable (FV) intake, processed food (PF) intake, outdoor playtime (OP), physical activity (PA) levels, and recreational screen time (RST) among children living in Michigan during the pandemic. Methods: The study team and community partners developed and distributed a survey using snowball sampling to reach families living largely in Central and Southeastern Michigan. Nonlinear mixed-effects proportional odds models were used to examine associations between child weight status along with demographic/household factors and changes in five weight-related behaviors. Results: Parents (n = 1313; representing 2469 children) reported a decrease in OP, FV, and PA levels, while there was an increase in RST and PF intake among their children. Household income was protective against a decrease in OP, PA, and FV but was associated with increased RST. Children's weight status was associated with decreased FV. Age was negatively associated with OP and PA, and positively associated with RST. Conclusions: These findings suggest an adverse influence of the pandemic on weight-related behaviors, particularly among adolescents in families with lower incomes and those with excess weight. Further work is needed to measure any impact on BMI trajectory and to identify interventions to reverse negative effects.

Adapting an Expanded Access program to enable investigational treatments for COVID-19.

Retrospective case studies of initiatives supported by the National Institutes of Health's Clinical and Translational Science Award (CTSA) hubs can be used to identify facilitators and barriers of translational science. This case study investigates how a CTSA Expanded Access program adapted to changing FDA guidance issued in 2020 to support clinicians' treatment of COVID-19 patients in Michigan. We studied how this program changed throughout the pandemic to support physicians' requests for remdesivir, convalescent plasma, and other uses of unapproved drugs and novel medical devices. A protocol for retrospective translational science case studies of health interventions developed by CTSA evaluators was used for this case study. Data collection methods included seven interviews and a review of institutional data, peer-reviewed publications, news stories, and other public records. The barriers identified include evolving guidance, misalignment of organizational operations, and the complexity of the research infrastructure. The facilitators of translation include collaboration between research and care teams, increasing engagement with a broad network of supporters, and ongoing professional development for research staff. The findings of this case study can be used to inform future investigations of the principles underlying the translational process.

Self-assessed Competencies of Clinical Research Professionals and Recommendations for Further Education and Training.

The Joint Task Force for Clinical Trial Competency (JTF) conducted a global survey of clinical research professionals requesting respondents to self-assess their competencies in each of the eight domains of its Core Competency Framework version 3.1. The results were analyzed based upon role, years of experience, educational level, professional certification, institutional affiliation, and continuing education participation. Respondents with professional certification self-assessed their competencies in all domains at higher levels than those without professional certification. The survey demonstrated that irrespective of role, experience, or educational level, training curricula in both pre-professional and continuing professional education should include additional content relating to research methods, protocol design, medical product development and regulation, and data management and informatics. These results validate and extend the recommendations of a similar 2016 JTF and other surveys. We further recommend that clinical and translational research organizations and clinical sites assess training needs locally, using both subjective and objective measures of skill and knowledge.

Community health workers and promotoras' perspectives of a research best practice course: A focus group study.

Introduction: Community Health Workers and Promotoras (CHW/Ps) are valued for their role in helping to engage community members in research. CHW/Ps have traditionally received variable training in research fundamentals, including importance and promotion of research rigor to establish consistency in the methods used over time. Research best practices training exists for research professionals, but no standard training is provided as part of the CHW/P job role. To develop this CHW/P research best practices training, our team engaged English- and Spanish-speaking CHW/Ps to watch an early version of an online module and to examine perceptions of the relevance of such a training and optimal delivery methods. Methods: Six virtual focus group discussions were conducted (three in English and three in Spanish) across different US geographic regions with currently employed CHW/Ps. Results: Forty CHW/Ps participated (95% female, mean age 44 years, 58% identifying as Hispanic/Latino). Four themes emerged: relevance of training, benefits of providing a certificate of completion, flexible training delivery modalities, and peer-led training. Discussion: With participation from representatives of the intended learner group of CHW/Ps, our team found that CHW/Ps valued learning about research best practices. They perceived culturally- and linguistically appropriate health research training to be highly relevant to their role, particularly for communicating key information to community members about their participation in health research. Additionally, participants provided input on effective dissemination of the training including the benefit of having proof of course completion, involvement of peer trainers, and value of providing the option to participate in online training.

Assessments of Research Competencies for Clinical Investigators: A Systematic Review.

Although there is extensive research literature on clinical skill competencies and the use of competency-based frameworks for clinical research, the appropriate methods to assess these competencies are not as well understood. Our goal in this systematic literature review is to identify, compare, and critique assessments of clinical research competencies. Articles were included in this review if they examined clinical investigators or clinical investigators in training, focused on research-based skills, and included some form of assessment of research-based competencies. A total of 76 articles were identified as part of the initial search; 16 met the criteria for inclusion. Two types of assessments of clinical research competence were identified: subjective self-assessments (n = 13) and objective tests (n = 6). These assessments covered a wide range of competencies, but there were no competency domains common to all. Most assessments had limited validation. Training was consistently associated with self-assessed competence but had little relationship to objective measures of competence. In contrast, experience was consistently associated with objectively assessed competence but not with self-assessed competence. These findings have important implications for those interested in assessing medical education programs. We describe a recommended standard for validity for assessments used for the purposes of summative program assessment.

The Research Objective Structured Clinical Exam (R-OSCE): An Innovative Tool to Assess Clinical and Translational Research Competencies.

This article was migrated. The article was marked as recommended. Introduction The Research Objective Structured Clinical Exam (R-OSCE) described in this paper was designed as part of a comprehensive program to assess competency in specific domains of clinical and translational research (CTR) for students enrolled in a 12-week introductory summer research program. Methods The program curriculum was mapped to core competencies developed by the National Center for Translational Science (NCATS) and used to develop R-OSCE stations. Twelve stations were developed, with five administered during orientation as a practice test and seven administered post-program. A scoring rubric using an anchored scale of 1-5 was developed and six qualified raters were trained in its use. The exam was self-paced and delivered through a secure online computer-based platform. Results Forty-seven students (three cohorts) completed the post-program R-OSCE. Most respondents scored at 3 (developing competence) or higher on most stations for both the practice and post-program exams, the exceptions being the stations involving writing research questions and engaging communities in research. Students indicated they liked demonstrating CTR skills through the R-OSCE. Most participants agreed that exam tasks were related to stated program competencies and that stations were realistic. Discussion The R-OSCE is best used as part of a comprehensive assessment program and may be useful in providing formative feedback to trainees that they can share with their mentors. Additionally, this study demonstrated that it could feasibly be used to evaluate the effectiveness of research education programs. However, additional time was needed to train raters and score the R-OSCE. Modifications were made to administer the exam through use of an online format with a modest budget. The computer-based format provides a solution to the current need for assessments that can be administered remotely.

The DIAMOND portal: An innovative platform for sharing translational research workforce training and assessment resources.

Although several initiatives have produced core competency domains for training the translational science workforce, training resources to help clinical research professionals advance these skills reside primarily within local departments or institutions. The Development, Implementation, and AssessMent of Novel Training in Domain (DIAMOND) project was designed to make this training more readily and publicly available. DIAMOND includes a digital portal to catalog publicly available educational resources and an ePortfolio to document professional development. DIAMOND is a nationally crowdsourced, federated, online catalog providing a platform for practitioners to find and share training and assessment materials. Contributors can share their own educational materials using a simple intake form that creates an electronic record; the portal enables users to browse or search this catalog of digital records and access the resources. Since September 2018, the portal has been visited more than 5,700 times and received over 280 contributions from professionals. The portal facilitates opportunities to connect and collaborate regarding future applications of these resources. Consequently, growing the collection and increasing numbers of both contributors and users remains a priority. Results from a small subset of users indicated over half accomplished their purpose for visiting the site, while qualitative results showed that users identified several benefits and helpful features of the ePortfolio.

Indices of Clinical Research Coordinators' Competence.

INTRODUCTION: There is a clear need to educate and train the clinical research workforce to conduct scientifically sound clinical research. Meeting this need requires the creation of tools to assess both an individual's preparedness to function efficiently in the clinical research enterprise as well as tools to evaluate the quality and effectiveness of programs that are designed to educate and train clinical research professionals. Here we report the development and validation of a competency self-assessment entitled the Competency Index for Clinical Research Professionals, version II (CICRP-II). METHODS: CICRP-II was developed using data collected from clinical research coordinators (CRCs) participating in the "Development, Implementation and Assessment of Novel Training In Domain-Based Competencies" project at four clinical and translational science award (CTSA) hubs and partnering institutions. RESULTS: An exploratory factor analysis identified a two-factor structure: the first factor measures self-reported competence to perform routine clinical research functions while the second factor measures competence to perform advanced clinical functions. We demonstrate the between groups validity by comparing CRCs working in different research settings. DISCUSSION: The excellent psychometric properties of CICRP-II, its ability to distinguish between experienced CRCs at research intensive CTSA hubs and CRCs working in less intensive community-based sites coupled with the simplicity of alternative methods for scoring respondents make it a valuable tool for gauging an individual's perceived preparedness to function in the role of CRC as well as an equally valuable tool to evaluate the value and effectiveness of clinical research education and training programs.

Guidelines for Evaluating Clinical Research Training using Competency Assessments.

This article was migrated. The article was marked as recommended. Effective training programs in clinical and translational research (CTR) are critical to the development of the research workforce. The evolution of global CTR competencies frameworks motivates many CTR institutions to align their training offerings with these professional standards. Guidelines for integrating competency-based frameworks and assessments into rigorous program evaluations are needed in order to promote the quality and impact of these training programs. These guidelines provide practical suggestions for how to ensure that subjective and objective assessments of CTR knowledge and skill can be effectively integrated in the evaluations used to improve these essential training programs. The approach presented here necessarily involves the systematic and deliberate incorporation of these particular types of assessments into comprehensive evaluation plans. While these guidelines are broadly applicable to the work of those charged with developing, administering and evaluating CTR training programs, they have been specifically designed for use by program directors.

Best Practices in Social and Behavioral Research: A multisite pilot evaluation of the good clinical practice online training course.

INTRODUCTION: The Best Practices in Social and Behavioral Research Course was developed to provide instruction on good clinical practice for social and behavioral trials. This study evaluated the new course. METHODS: Participants across 4 universities took the course (n=294) and were sent surveys following course completion and 2 months later. Outcomes included relevance, how engaging the course was, and working differently because of the course. Open-ended questions were posed to understand how work was impacted. RESULTS: Participants rated the course as relevant and engaging (6.4 and 5.8/7 points) and reported working differently (4.7/7 points). Participants with less experience in social and behavioral trials were most likely to report working differently 2 months later. DISCUSSION: The course was perceived as relevant and engaging. Participants described actions taken to improve rigor in implementing trials. Future studies with a larger sample and additional participating sites are recommended.

Implementing and measuring the impact of a clinical and translational research mentor recognition program.

INTRODUCTION: Research shows incentives can motivate faculty to increase their engagement in mentoring, despite a myriad of institutional barriers. One such incentive may be the implementation of a university-wide mentor award program to promote a culture of mentorship. METHODS: A new mentorship award was created at a research-intensive university and faculty recipients were surveyed to assess their perceptions of the award's impact on their mentoring practices and career. RESULTS: Sixty-two percent of awardees (n=21) completed the survey and felt the recognition incentivized them to engage in further mentoring and participate in formal mentorship training. Most awardees referenced the award in their CVs, performance evaluations, and grant proposals. Additionally, they felt the award effectively promoted mentoring among the broader faculty community. CONCLUSION: Growth of clinical and translational research depends in part on the mentorship received by early career faculty. Therefore, other research universities may benefit from implementing such awards.