Benefits of a College STEM Faculty Development Initiative: Instructors Report Increased and Sustained Implementation of Research-Based Instructional Strategies.

The Summer Institutes on Scientific Teaching (SI) is a faculty development workshop in which science, technology, engineering, and mathematics (STEM) instructors, particularly from biology, are trained in the Scientific Teaching (ST) pedagogy. While participants have generally reported positive experiences, we aimed to assess how the SI affected participants' teaching practices. Building on a previously developed taxonomy of ST practices, we surveyed SI participants from the 2004-2014 SI classes regarding specific ST practices. Participants' self-reported use and implementation of ST practices increased immediately after SI attendance as well as over a longer time frame, suggesting that implementation persisted and even increased with time. However, instructors reported implementation gains for some practices more than others. The practices with the highest gains were engaging students in their own learning, using learning goals in course design, employing formative assessment, developing overarching course learning goals, representing science as a process, and facilitating group discussion activities. We propose that the ST practices showing the greatest gains may serve as beneficial focal points for professional development programs, while practices with smaller gains may require modified dissemination approaches or support structures.

Erratum: Opportunities for strengthening CTSA evaluation - CORRIGENDUM.

[This corrects the article DOI: 10.1017/cts.2019.387.].

Bridging the Gap Between Aging Research and Practice: A New Strategy for Enhancing the Consensus Workshop Model.

Objectives: The Cornell Research-to-Practice (RTP) Consensus Workshop Model is a strategy for bridging the gap between aging research and practice but lacks a technique for evaluating the relative importance of ideas. This project assessed the feasibility of adding a quantitative survey to the RTP model to address this gap. Method: Older adults with cancer (OACs), OAC caregivers, researchers, clinicians, and advocacy organization representatives participated in a RTP workshop on implementing psychological interventions for OACs. Following an in-person workshop, participants completed surveys assessing the relative importance of barriers and strategies for psychological intervention implementation. Results: Seventeen of 35 participants completed the survey, the majority of which were likely clinicians. Barriers and strategies to implementation rated as having the greatest impact were associated with the care team and institutional factors. Conclusion: Quantitative ratings add novel information to the RTP model that could potentially enhance the model's impact on aging research and practice.

A protocol for retrospective translational science case studies of health interventions.

The critical processes driving successful research translation remain understudied. We describe a mixed-method case study protocol for analyzing translational research that has led to the successful development and implementation of innovative health interventions. An overarching goal of these case studies is to describe systematically the chain of events between basic, fundamental scientific discoveries and the adoption of evidence-based health applications, including description of varied, long-term impacts. The case study approach isolates many of the key factors that enable the successful translation of research into practice and provides compelling evidence connecting the intervention to measurable changes in health and medical practice, public health outcomes, and other broader societal impacts. The goal of disseminating this protocol is to systematize a rigorous approach, which can enhance reproducibility, promote the development of a large collection of comparable studies, and enable cross-case analyses. This approach, an application of the "science of translational science," will lead to a better understanding of key research process markers, timelines, and potential points of leverage for intervention that may help facilitate decisions, processes, and policies to speed the sustainable translational process. Case studies are effective communication vehicles to demonstrate both accountability and the impacts of the public's investment in research.

Opportunities for strengthening CTSA evaluation.

The purpose of the article is to describe the progress of the Clinical and Translational Science Award (CTSA) Program to address the evaluation-related recommendations made by the 2013 Institute of Medicine's review of the CTSA Program and guidelines published in CTS Journal the same year (Trochim et al., Clinical and Translational Science 2013; 6(4): 303-309). We utilize data from a 2018 national survey of evaluators administered to all 64 CTSA hubs and a content analysis of the role of evaluation in the CTSA Program Funding Opportunity Announcements to document progress. We present four new opportunities for further strengthening CTSA evaluation efforts: (1) continue to build the collaborative evaluation infrastructure at local and national levels; (2) make better use of existing data; (3) strengthen and augment the common metrics initiative; and (4) pursue internal and external opportunities to evaluate the CTSA program at the national level. This article will be of significant interest to the funders of the CTSA Program and the multiple stakeholders in the larger consortium and will promote dialog from the broad range of CTSA stakeholders about further strengthening the CTSA Program's evaluation.

Advancing the Agenda in Pediatric Mental Health Education.

BACKGROUND AND OBJECTIVES: There is an urgent need to advance mental health (MH) education and/or training in pediatric residency programs, yet no consensus on how to achieve this. We created an operational framework from ideas provided by a diverse group of stakeholders on how to advance MH education. METHODS: Concept-mapping methodology was used, which involves brainstorming ideas by completing a focus prompt, sorting ideas into groups, and rating them for importance and feasibility. Multidimensional scaling and hierarchical cluster analysis grouped ideas into clusters. Average importance and feasibility were calculated for each statement and cluster and compared statistically in each cluster and between subgroups. RESULTS: Ninety-nine ideas were generated. Sorted ideas yielded a 7-cluster concept map: (1) modalities for MH training, (2) prioritization of MH, (3) systems-based practice, (4) self-awareness and/or relationship building, (5) training in clinical assessment of patients, (6) training in treatment, and (7) diagnosis-specific skills. Two hundred and sixteen participants rated ideas for importance and 209 for feasibility. Four clusters had a statistically significant difference between their importance and feasibility ratings (P < .001). Suburban and rural area respondents (versus urban) rated clusters higher in importance and feasibility (P < .004), trainees rated all clusters higher in feasibility than practicing clinicians, and MH professionals rated prioritization of MH higher in feasibility (3.42 vs 2.98; P < .001). CONCLUSIONS: This comprehensive set of ideas, especially those rated highly in both importance and feasibility, should inform curricular and policy initiatives. Differences between importance and feasibility may explain why there has been little progress in this field.

Setting strategy for system change: using concept mapping to prioritise national action for chronic disease prevention.

BACKGROUND: Chronic diseases are a serious and urgent problem, requiring at-scale, multi-component, multi-stakeholder action and cooperation. Despite numerous national frameworks and agenda-setting documents to coordinate prevention efforts, Australia, like many countries internationally, is yet to substantively impact the burden from chronic disease. Improved evidence on effective strategies for the prevention of chronic disease is required. This research sought to articulate a priority set of important and feasible action domains to inform future discussion and debate regarding priority areas for chronic disease prevention policy and strategy. METHODS: Using concept mapping, a mixed-methods approach to making use of the best available tacit knowledge of recognised, diverse and well-experienced actors, and national actions to improve the prevention of chronic disease in Australia were identified and then mapped. Participants (ranging from 58 to 78 in the various stages of the research) included a national sample of academics, policymakers and practitioners. Data collection involved the generation and sorting of statements by participants. A series of visual representations of the data were then developed. RESULTS: A total of 95 statements were distilled into 12 clusters for action, namely Inter-Sectoral Partnerships; Systems Perspective/Action; Governance; Roles and Responsibilities; Evidence, Feedback and Learning; Funding and Incentive; Creating Demand; Primary Prevention; Social Determinants and Equity; Healthy Environments; Food and Nutrition; and Regulation and Policy. Specific areas for more immediate national action included refocusing the health system to prevention over cure, raising the profile of public health with health decision-makers, funding policy- and practice-relevant research, improving communication about prevention, learning from both global best-practice and domestic successes and failures, increasing the focus on primary prevention, and developing a long-term prevention strategy with an explicit funding commitment. CONCLUSIONS: Preventing chronic diseases and their risk factors will require at-scale, multi-component, multi-stakeholder action and cooperation. The concept mapping procedures used in this research have enabled the synthesis of views across different stakeholders, bringing both divergent and convergent perspectives to light, and collectively creating signals for where to prioritise national action. Previous national strategies for chronic disease prevention have not collated the tacit knowledge of diverse actors in the prevention of chronic disease in this structured way.

Hindsight is 20/20: Reflections on the evolution of concept mapping.

This paper considers the origins and development of the concept mapping methodology, a summary of its growth, and its influence in a variety of fields. From initial discussions with graduate students, through the rise of the theory-driven approach to program evaluation and the development of a theoretical framework for conceptualization methodology, the paper highlights some of the key early efforts and pilot projects that culminated in a 1989 special issue on the method in Evaluation and Program Planning that brought the method to the attention of the field of evaluation. The paper details the thinking that led to the standard version of the method (the analytic sequence, "bridging" index, and pattern matching) and the development of the software for accomplishing it. A bibliometric analysis shows that the rate of citation continues to increase, where it has grown geographically and institutionally, that the method has been used in a wide variety of disciplines and specialties, and that the literature had an influence on the field. The article concludes with a critical appraisal of some of the key aspects of the approach that warrant further development.

Introduction to a special issue on concept mapping.

Concept mapping was developed in the 1980s as a unique integration of qualitative (group process, brainstorming, unstructured sorting, interpretation) and quantitative (multidimensional scaling, hierarchical cluster analysis) methods designed to enable a group of people to articulate and depict graphically a coherent conceptual framework or model of any topic or issue of interest. This introduction provides the basic definition and description of the methodology for the newcomer and describes the steps typically followed in its most standard canonical form (preparation, generation, structuring, representation, interpretation and utilization). It also introduces this special issue which reviews the history of the methodology, describes its use in a variety of contexts, shows the latest ways it can be integrated with other methodologies, considers methodological advances and developments, and sketches a vision of the future of the method's evolution.

Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

Evolutionary Evaluation: implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate.

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of "evidence-based programs". From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.

A randomized controlled trial of a community-based nutrition education program for low-income parents.

OBJECTIVE: Assess effectiveness of the Expanded Food and Nutrition Education Program on nutrition behaviors post-education and longitudinally. DESIGN: Switching replications randomized experimental design. Participants randomly assigned to immediate education (IE) or delayed education (DE). Participants in IE received intervention the first 8 weeks, and those in DE the second 8 weeks, with no intervention during alternate periods. Data were collected in 3 repeated measures. PARTICIPANTS: Parents (n = 168 randomized; n = 134 completed) of children in 2 Head Start and 6 low-income schools. INTERVENTION: Eight weekly workshops, based on Eating Right is Basic-Enhanced adapted to incorporate dialogue approach with experiential learning. MAIN OUTCOME MEASURES: Ten-item self-reported behavior checklist on nutrition, food resource management, food safety, and food security; responses on a 5-point scale reporting frequency of behavior. ANALYSIS: Chi-square, analysis of variance, and multiple regression. RESULTS: Groups were demographically similar. Both groups reported improved behaviors pre- to post-education (P < .05). There was no significant difference between groups at Time 1 (T1) or DE control period (T1 vs T2). Changed IE behavior was retained T2 to T3. A multiple regression model of overall change, controlling for T1 score and educator, showed significant improvement (n = 134, ß = 5.72, P < .001). CONCLUSIONS AND IMPLICATIONS: Positive outcomes were supported by this experimental study in a usual program context, with reported behavior changes retained at least 2 months.

Evaluation guidelines for the Clinical and Translational Science Awards (CTSAs).

The National Center for Advancing Translational Sciences (NCATS), a part of the National Institutes of Health, currently funds the Clinical and Translational Science Awards (CTSAs), a national consortium of 61 medical research institutions in 30 states and the District of Columbia. The program seeks to transform the way biomedical research is conducted, speed the translation of laboratory discoveries into treatments for patients, engage communities in clinical research efforts, and train a new generation of clinical and translational researchers. An endeavor as ambitious and complex as the CTSA program requires high-quality evaluations in order to show that the program is well implemented, efficiently managed, and demonstrably effective. In this paper, the Evaluation Key Function Committee of the CTSA Consortium presents an overall framework for evaluating the CTSA program and offers policies to guide the evaluation work. The guidelines set forth are designed to serve as a tool for education within the CTSA community by illuminating key issues and practices that should be considered during evaluation planning, implementation, and utilization. Additionally, these guidelines can provide a basis for ongoing discussions about how the principles articulated in this paper can most effectively be translated into operational reality.

Estimating return on investment in translational research: methods and protocols.

Assessing the value of clinical and translational research funding on accelerating the translation of scientific knowledge is a fundamental issue faced by the National Institutes of Health (NIH) and its Clinical and Translational Awards (CTSAs). To address this issue, the authors propose a model for measuring the return on investment (ROI) of one key CTSA program, the clinical research unit (CRU). By estimating the economic and social inputs and outputs of this program, this model produces multiple levels of ROI: investigator, program, and institutional estimates. A methodology, or evaluation protocol, is proposed to assess the value of this CTSA function, with specific objectives, methods, descriptions of the data to be collected, and how data are to be filtered, analyzed, and evaluated. This article provides an approach CTSAs could use to assess the economic and social returns on NIH and institutional investments in these critical activities.

Community-researcher partnerships at NIAID HIV/AIDS clinical trials sites: insights for evaluation and enhancement.

BACKGROUND: Community engagement has been a cornerstone of National Institute of Allergy and Infectious Diseases (NIAID)'s HIV/AIDS clinical trials programs since 1990. Stakeholders now consider this critical to success, hence the impetus to develop evaluation approaches. OBJECTIVES: The purpose was to assess the extent to which community advisory boards (CABs) at HIV/AIDS trials sites are being integrated into research activities. METHODS: CABs and research staff (RS) at NIAID research sites were surveyed for how each viewed (a) the frequency of activities indicative of community involvement, (b) the means for identifying, prioritizing, and supporting CAB needs, and (c) mission and operational challenges. RESULTS: Overall, CABs and RS share similar views about the frequency of community involvement activities. Cluster analysis reveals three groups of sites based on activity frequency ratings, including a group notable for CAB-RS discordance. CONCLUSIONS: Assessing differences between community and researcher perceptions about the frequency of and challenges posed by specific engagement activities may prove useful in developing evaluation tools for assessing community engagement in collaborative research settings.

A virtual national laboratory for reengineering clinical translational science.

Clinical research is burdened by inefficiencies and complexities, with a poor record of trial completion, none of which is desirable. The Clinical and Translational Science Award (CTSA) Consortium, including more than 60 clinical research institutions, supports a unified national effort to become, in effect, a virtual national laboratory designed to identify, implement, evaluate, and extend process improvements across all parts of clinical research, from conception to completion. If adequately supported by academic health centers, industry, and funding agencies, the Consortium could become a test bed for improvements that can dramatically reduce wasteful complexity, thus increasing the likelihood of clinical trial completion.

The end of the beginning: a commentary on 'Evaluation metrics for biostatistical and epidemiological collaborations'.

The paper 'Evaluation Metrics for Biostatistical and Epidemiological Collaborations' of Rubio et al. represents an important initial advance in the evaluation of biostatistics, epidemiology, and research design (BERD). The authors present a sensible three-domain model (collaboration with investigators, application of BERD-related methods, and discovery of new BERD methodologies), rightly acknowledge the importance of team science, and break new ground in illustrating that the Clinical Translational Science Awards can function as a kind of national laboratory for the development and exploration of measures and metrics. Building upon these gains, there are several future considerations worthy of subsequent serious attention: strengthening the connection between BERD evaluation and both the science of team science and the field of evaluation; facing the challenges of operationalization of the conceptual domains; augmenting the work of Rubio et al. with standard evaluative models; and anticipating the need for multiplistic mixed methods and experimental and quasi-experimental complements to the proposed BERD metrics. Several common pitfalls will also be important to avoid, including the tendency to conflate the meaning of 'metrics' and 'measures' and the potential for a premature rush to adopt national 'standards' before adequately pilot testing the initial set of methods they have worked so diligently to develop.

Evaluating translational research: a process marker model.

OBJECTIVE: We examine the concept of translational research from the perspective of evaluators charged with assessing translational efforts. One of the major tasks for evaluators involved in translational research is to help assess efforts that aim to reduce the time it takes to move research to practice and health impacts. Another is to assess efforts that are intended to increase the rate and volume of translation. METHODS: We offer an alternative to the dominant contemporary tendency to define translational research in terms of a series of discrete "phases." RESULTS: We contend that this phased approach has been confusing and that it is insufficient as a basis for evaluation. Instead, we argue for the identification of key operational and measurable markers along a generalized process pathway from research to practice. CONCLUSIONS: This model provides a foundation for the evaluation of interventions designed to improve translational research and the integration of these findings into a field of translational studies.

Evaluating research and impact: a bibliometric analysis of research by the NIH/NIAID HIV/AIDS clinical trials networks.

Evaluative bibliometrics uses advanced techniques to assess the impact of scholarly work in the context of other scientific work and usually compares the relative scientific contributions of research groups or institutions. Using publications from the National Institute of Allergy and Infectious Diseases (NIAID) HIV/AIDS extramural clinical trials networks, we assessed the presence, performance, and impact of papers published in 2006-2008. Through this approach, we sought to expand traditional bibliometric analyses beyond citation counts to include normative comparisons across journals and fields, visualization of co-authorship across the networks, and assess the inclusion of publications in reviews and syntheses. Specifically, we examined the research output of the networks in terms of the a) presence of papers in the scientific journal hierarchy ranked on the basis of journal influence measures, b) performance of publications on traditional bibliometric measures, and c) impact of publications in comparisons with similar publications worldwide, adjusted for journals and fields. We also examined collaboration and interdisciplinarity across the initiative, through network analysis and modeling of co-authorship patterns. Finally, we explored the uptake of network produced publications in research reviews and syntheses. Overall, the results suggest the networks are producing highly recognized work, engaging in extensive interdisciplinary collaborations, and having an impact across several areas of HIV-related science. The strengths and limitations of the approach for evaluation and monitoring research initiatives are discussed.