A landscape assessment of CTSA evaluators and their work in the CTSA consortium, 2021 survey findings.

This article presents a landscape assessment of the findings from the 2021 Clinical and Translational Science Award (CTSA) Evaluators Survey. This survey was the most recent iteration of a well established, national, peer-led systematic snapshot of the CTSA evaluators, their skillsets, listed evaluation resources, preferred methods, and identified best practices. Three questions guided our study: who are the CTSA evaluators, what competencies do they share and how is their work used within hubs. We describe our survey process (logistics of development, deployment, and differences in historical context with prior instruments); and present its main findings. We provide specific recommendations for evaluation practice in two main categories (National vs Group-level) including, among others, the need for a national, strategic plan for evaluation as well as enhanced mentoring and training of the next generation of evaluators. Although based on the challenges and opportunities currently within the CTSA Consortium, takeaways from this study constitute important lessons with potential for application in other large evaluation consortia. To our knowledge, this is the first time 2021 survey findings are disseminated widely, to increase transparency of the CTSA evaluators' work and to motivate conversations within hub and beyond, as to how best to leverage existent evaluative capacity.

Distinguishing between translational science and translational research in CTSA pilot studies: A collaborative project across 12 CTSA hubs.

Introduction: The institutions (i.e., hubs) making up the National Institutes of Health (NIH)-funded network of Clinical and Translational Science Awards (CTSAs) share a mission to turn observations into interventions to improve public health. Recently, the focus of the CTSAs has turned increasingly from translational research (TR) to translational science (TS). The current NIH Funding Opportunity Announcement (PAR-21-293) for CTSAs stipulates that pilot studies funded through the CTSAs must be "focused on understanding a scientific or operational principle underlying a step of the translational process with the goal of developing generalizable solutions to accelerate translational research." This new directive places Pilot Program administrators in the position of arbiters with the task of distinguishing between TR and TS projects. The purpose of this study was to explore the utility of a set of TS principles set forth by NCATS for distinguishing between TR and TS. Methods: Twelve CTSA hubs collaborated to generate a list of Translational Science Principles questions. Twenty-nine Pilot Program administrators used these questions to evaluate 26 CTSA-funded pilot studies. Results: Factor analysis yielded three factors: Generalizability/Efficiency, Disruptive Innovation, and Team Science. The Generalizability/Efficiency factor explained the largest amount of variance in the questions and was significantly able to distinguish between projects that were verified as TS or TR (t = 6.92, p < .001) by an expert panel. Conclusions: The seven questions in this factor may be useful for informing deliberations regarding whether a study addresses a question that aligns with NCATS' vision of TS.

Learning health system benefits: Development and initial validation of a framework.

Introduction: Implementation of research findings in clinical practice often is not realized or only partially achieved, and if so, with a significant delay. Learning health systems (LHSs) hold promise to overcome this problem by embedding clinical research and evidence-based best practices into care delivery, enabling innovation and continuous improvement. Implementing an LHS is a complex process that requires participation and resources of a wide range of stakeholders, including healthcare leaders, clinical providers, patients and families, payers, and researchers. Engaging these stakeholders requires communicating clear, tangible value propositions. Existing models identify broad categories of benefits but do not explicate the full range of benefits or ways they can manifest in different organizations. Methods: To develop such a framework, a working group with representatives from six Clinical and Translational Science Award (CTSA) hubs reviewed existing literature on LHS characteristics, models, and goals; solicited expert input; and applied the framework to their local LHS experiences. Results: The Framework of LHS Benefits includes six categories of benefits (quality, safety, equity, patient satisfaction, reputation, and value) relevant for a range of stakeholders and defines key concepts within each benefit. Applying the framework to five LHS case examples indicated preliminary face validity across varied LHS approaches and revealed three dimensions in which the framework is relevant: defining goals of individual LHS projects, facilitating collaboration based on shared values, and establishing guiding tenets of an LHS program or mission. Conclusion: The framework can be used to communicate the value of an LHS to different stakeholders across varied contexts and purposes, and to identify future organizational priorities. Further validation will contribute to the framework's evolution and support its potential to inform the development of tools to evaluate LHS impact.

Engaging community in the translational process: Environmental scan of adaptive capacity and preparedness of Clinical and Translational Science Award Program hubs.

This paper is part of the Environmental Scan of Adaptive Capacity and Preparedness of Clinical and Translational Science Award (CTSA) hubs, illuminating challenges, practices, and lessons learned related to CTSA hubs' efforts of engaging community partners to reduce the spread of the virus, address barriers to COVID-19 testing, identify treatments to improve health outcomes, and advance community participation in research. CTSA researchers, staff, and community partners collaborated to develop evidence-based, inclusive, accessible, and culturally appropriate strategies and resources helping community members stay healthy, informed, and connected during the pandemic. CTSA institutions have used various mechanisms to advance co-learning and co-sharing of knowledge, resources, tools, and experiences between academic professionals, patients, community partners, and other stakeholders. Forward-looking and adaptive decision-making structures are those that prioritize sustained relationships, mutual trust and commitment, ongoing communication, proactive identification of community concerns and needs, shared goals and decision making, as well as ample appreciation of community members and their contributions to translational research. There is a strong need for further community-engaged research and workforce training on how to build our collective and individual adaptive capacity to sustain and improve processes and outcomes of engagement with and by communities-in all aspects of translational science.

Understanding and responding to COVID-19 in Wales: protocol for a privacy-protecting data platform for enhanced epidemiology and evaluation of interventions.

INTRODUCTION: The emergence of the novel respiratory SARS-CoV-2 and subsequent COVID-19 pandemic have required rapid assimilation of population-level data to understand and control the spread of infection in the general and vulnerable populations. Rapid analyses are needed to inform policy development and target interventions to at-risk groups to prevent serious health outcomes. We aim to provide an accessible research platform to determine demographic, socioeconomic and clinical risk factors for infection, morbidity and mortality of COVID-19, to measure the impact of COVID-19 on healthcare utilisation and long-term health, and to enable the evaluation of natural experiments of policy interventions. METHODS AND ANALYSIS: Two privacy-protecting population-level cohorts have been created and derived from multisourced demographic and healthcare data. The C20 cohort consists of 3.2 million people in Wales on the 1 January 2020 with follow-up until 31 May 2020. The complete cohort dataset will be updated monthly with some individual datasets available daily. The C16 cohort consists of 3 million people in Wales on the 1 January 2016 with follow-up to 31 December 2019. C16 is designed as a counterfactual cohort to provide contextual comparative population data on disease, health service utilisation and mortality. Study outcomes will: (a) characterise the epidemiology of COVID-19, (b) assess socioeconomic and demographic influences on infection and outcomes, (c) measure the impact of COVID-19 on short -term and longer-term population outcomes and (d) undertake studies on the transmission and spatial spread of infection. ETHICS AND DISSEMINATION: The Secure Anonymised Information Linkage-independent Information Governance Review Panel has approved this study. The study findings will be presented to policy groups, public meetings, national and international conferences, and published in peer-reviewed journals.

The Access Technology Program of the Indiana Clinical Translational Sciences Institute (CTSI): A model to facilitate access to cutting-edge technologies across a state.

INTRODUCTION: Access to cutting-edge technologies is essential for investigators to advance translational research. The Indiana Clinical and Translational Sciences Institute (CTSI) spans three major and preeminent universities, four large academic campuses across the state of Indiana, and is mandate to provide best practices to a whole state. METHODS: To address the need to facilitate the availability of innovative technologies to its investigators, the Indiana CTSI implemented the Access Technology Program (ATP). The activities of the ATP, or any program of the Indiana CTSI, are challenged to connect technologies and investigators on the multiple Indiana CTSI campuses by the geographical distances between campuses (1-4 hr driving time). RESULTS: Herein, we describe the initiatives developed by the ATP to increase the availability of state-of-the-art technologies to its investigators on all Indiana CTSI campuses, and the methods developed by the ATP to bridge the distance between campuses, technologies, and investigators for the advancement of clinical translational research. CONCLUSIONS: The methods and practices described in this publication may inform other approaches to enhance translational research, dissemination, and usage of innovative technologies by translational investigators, especially when distance or multi-campus cultural differences are factors to efficient application.

Classifying publications from the clinical and translational science award program along the translational research spectrum: a machine learning approach.

BACKGROUND: Translational research is a key area of focus of the National Institutes of Health (NIH), as demonstrated by the substantial investment in the Clinical and Translational Science Award (CTSA) program. The goal of the CTSA program is to accelerate the translation of discoveries from the bench to the bedside and into communities. Different classification systems have been used to capture the spectrum of basic to clinical to population health research, with substantial differences in the number of categories and their definitions. Evaluation of the effectiveness of the CTSA program and of translational research in general is hampered by the lack of rigor in these definitions and their application. This study adds rigor to the classification process by creating a checklist to evaluate publications across the translational spectrum and operationalizes these classifications by building machine learning-based text classifiers to categorize these publications. METHODS: Based on collaboratively developed definitions, we created a detailed checklist for categories along the translational spectrum from T0 to T4. We applied the checklist to CTSA-linked publications to construct a set of coded publications for use in training machine learning-based text classifiers to classify publications within these categories. The training sets combined T1/T2 and T3/T4 categories due to low frequency of these publication types compared to the frequency of T0 publications. We then compared classifier performance across different algorithms and feature sets and applied the classifiers to all publications in PubMed indexed to CTSA grants. To validate the algorithm, we manually classified the articles with the top 100 scores from each classifier. RESULTS: The definitions and checklist facilitated classification and resulted in good inter-rater reliability for coding publications for the training set. Very good performance was achieved for the classifiers as represented by the area under the receiver operating curves (AUC), with an AUC of 0.94 for the T0 classifier, 0.84 for T1/T2, and 0.92 for T3/T4. CONCLUSIONS: The combination of definitions agreed upon by five CTSA hubs, a checklist that facilitates more uniform definition interpretation, and algorithms that perform well in classifying publications along the translational spectrum provide a basis for establishing and applying uniform definitions of translational research categories. The classification algorithms allow publication analyses that would not be feasible with manual classification, such as assessing the distribution and trends of publications across the CTSA network and comparing the categories of publications and their citations to assess knowledge transfer across the translational research spectrum.

Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

Project development teams: a novel mechanism for accelerating translational research.

The trend in conducting successful biomedical research is shifting from individual academic labs to coordinated collaborative research teams. Teams of experienced investigators with a wide variety of expertise are now critical for developing and maintaining a successful, productive research program. However, assembling a team whose members have the right expertise requires a great deal of time and many resources. To assist investigators seeking such resources, the Indiana Clinical and Translational Sciences Institute (Indiana CTSI) created the Project Development Teams (PDTs) program to support translational research on and across the Indiana University-Purdue University Indianapolis, Indiana University, Purdue University, and University of Notre Dame campuses. PDTs are multidisciplinary committees of seasoned researchers who assist investigators, at any stage of research, in transforming ideas/hypotheses into well-designed translational research projects. The teams help investigators capitalize on Indiana CTSI resources by providing investigators with, as needed, mentoring and career development; protocol development; pilot funding; institutional review board, regulatory, and/or nursing support; intellectual property support; access to institutional technology; and assistance with biostatistics, bioethics, recruiting participants, data mining, engaging community health, and collaborating with other investigators.Indiana CTSI leaders have analyzed metrics, collected since the inception of the PDT program in 2008 from both investigators and team members, and found evidence strongly suggesting that the highly responsive teams have become an important one-stop venue for facilitating productive interactions between basic and clinical scientists across four campuses, have aided in advancing the careers of junior faculty, and have helped investigators successfully obtain external funds.

Use of social network analysis tools to validate a resources infrastructure for interinstitutional translational research: a case study.

QUESTION: How can knowledge management and innovative technology, cornerstones of library practice, be leveraged to validate the progress of Clinical and Translational Science Awards? SETTING: The Indiana Clinical and Translational Sciences Institute (Indiana CTSI) promotes interdisciplinary research across academic institutions. METHODS: Using social networking tools and knowledge management skills enabled the department of knowledge informatics and translation to create a visualization of utilization of resources across different Indiana CTSI programs and coauthorship and citation patterns. RESULTS: Contacts with different resources per investigator increased; every targeted program was shown to be linked to another. Analysis of publications established a baseline to further analyze the scientific contribution of Indiana CTSI projects. CONCLUSION: Knowledge management and social networking utilities validated the efficacy of the Indiana CTSI resources infrastructure and demonstrated visualization of collaboration. The bibliometric analysis of publications provides a basis for assessing longer-term contributions of support to scientific discovery and transdisciplinary science.