The end of the beginning: a commentary on 'Evaluation metrics for biostatistical and epidemiological collaborations'.

The paper 'Evaluation Metrics for Biostatistical and Epidemiological Collaborations' of Rubio et al. represents an important initial advance in the evaluation of biostatistics, epidemiology, and research design (BERD). The authors present a sensible three-domain model (collaboration with investigators, application of BERD-related methods, and discovery of new BERD methodologies), rightly acknowledge the importance of team science, and break new ground in illustrating that the Clinical Translational Science Awards can function as a kind of national laboratory for the development and exploration of measures and metrics. Building upon these gains, there are several future considerations worthy of subsequent serious attention: strengthening the connection between BERD evaluation and both the science of team science and the field of evaluation; facing the challenges of operationalization of the conceptual domains; augmenting the work of Rubio et al. with standard evaluative models; and anticipating the need for multiplistic mixed methods and experimental and quasi-experimental complements to the proposed BERD metrics. Several common pitfalls will also be important to avoid, including the tendency to conflate the meaning of 'metrics' and 'measures' and the potential for a premature rush to adopt national 'standards' before adequately pilot testing the initial set of methods they have worked so diligently to develop.

Erratum: Opportunities for strengthening CTSA evaluation - CORRIGENDUM.

[This corrects the article DOI: 10.1017/cts.2019.387.].

A protocol for retrospective translational science case studies of health interventions.

The critical processes driving successful research translation remain understudied. We describe a mixed-method case study protocol for analyzing translational research that has led to the successful development and implementation of innovative health interventions. An overarching goal of these case studies is to describe systematically the chain of events between basic, fundamental scientific discoveries and the adoption of evidence-based health applications, including description of varied, long-term impacts. The case study approach isolates many of the key factors that enable the successful translation of research into practice and provides compelling evidence connecting the intervention to measurable changes in health and medical practice, public health outcomes, and other broader societal impacts. The goal of disseminating this protocol is to systematize a rigorous approach, which can enhance reproducibility, promote the development of a large collection of comparable studies, and enable cross-case analyses. This approach, an application of the "science of translational science," will lead to a better understanding of key research process markers, timelines, and potential points of leverage for intervention that may help facilitate decisions, processes, and policies to speed the sustainable translational process. Case studies are effective communication vehicles to demonstrate both accountability and the impacts of the public's investment in research.

Bridging the Gap Between Aging Research and Practice: A New Strategy for Enhancing the Consensus Workshop Model.

Objectives: The Cornell Research-to-Practice (RTP) Consensus Workshop Model is a strategy for bridging the gap between aging research and practice but lacks a technique for evaluating the relative importance of ideas. This project assessed the feasibility of adding a quantitative survey to the RTP model to address this gap. Method: Older adults with cancer (OACs), OAC caregivers, researchers, clinicians, and advocacy organization representatives participated in a RTP workshop on implementing psychological interventions for OACs. Following an in-person workshop, participants completed surveys assessing the relative importance of barriers and strategies for psychological intervention implementation. Results: Seventeen of 35 participants completed the survey, the majority of which were likely clinicians. Barriers and strategies to implementation rated as having the greatest impact were associated with the care team and institutional factors. Conclusion: Quantitative ratings add novel information to the RTP model that could potentially enhance the model's impact on aging research and practice.

Benefits of a College STEM Faculty Development Initiative: Instructors Report Increased and Sustained Implementation of Research-Based Instructional Strategies.

The Summer Institutes on Scientific Teaching (SI) is a faculty development workshop in which science, technology, engineering, and mathematics (STEM) instructors, particularly from biology, are trained in the Scientific Teaching (ST) pedagogy. While participants have generally reported positive experiences, we aimed to assess how the SI affected participants' teaching practices. Building on a previously developed taxonomy of ST practices, we surveyed SI participants from the 2004-2014 SI classes regarding specific ST practices. Participants' self-reported use and implementation of ST practices increased immediately after SI attendance as well as over a longer time frame, suggesting that implementation persisted and even increased with time. However, instructors reported implementation gains for some practices more than others. The practices with the highest gains were engaging students in their own learning, using learning goals in course design, employing formative assessment, developing overarching course learning goals, representing science as a process, and facilitating group discussion activities. We propose that the ST practices showing the greatest gains may serve as beneficial focal points for professional development programs, while practices with smaller gains may require modified dissemination approaches or support structures.

Developing a conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical trials networks.

Globally, health research organizations are called upon to re-examine their policies and practices to more efficiently and effectively address current scientific and social needs, as well as increasing public demands for accountability.Through a case study approach, the authors examine an effort undertaken by the National Institute of Allergy & Infectious Diseases (part of the National Institutes of Health, Department of Health & Human Services, United States Government) to develop an evaluation system for its recently restructured HIV/AIDS clinical trials program. The challenges in designing, operationalizing, and managing global clinical trials programs are considered in the context of large scale scientific research initiatives.Through a process of extensive stakeholder input, a framework of success factors was developed that enables both a prospective view of the elements that must be addressed in an evaluation of this research and a current state assessment of the extent to which the goals of the restructuring are understood by stakeholders across the DAIDS clinical research networks.

Opportunities for strengthening CTSA evaluation.

The purpose of the article is to describe the progress of the Clinical and Translational Science Award (CTSA) Program to address the evaluation-related recommendations made by the 2013 Institute of Medicine's review of the CTSA Program and guidelines published in CTS Journal the same year (Trochim et al., Clinical and Translational Science 2013; 6(4): 303-309). We utilize data from a 2018 national survey of evaluators administered to all 64 CTSA hubs and a content analysis of the role of evaluation in the CTSA Program Funding Opportunity Announcements to document progress. We present four new opportunities for further strengthening CTSA evaluation efforts: (1) continue to build the collaborative evaluation infrastructure at local and national levels; (2) make better use of existing data; (3) strengthen and augment the common metrics initiative; and (4) pursue internal and external opportunities to evaluate the CTSA program at the national level. This article will be of significant interest to the funders of the CTSA Program and the multiple stakeholders in the larger consortium and will promote dialog from the broad range of CTSA stakeholders about further strengthening the CTSA Program's evaluation.

Systems thinking to improve the public's health.

Improving population health requires understanding and changing societal structures and functions, but countervailing forces sometimes undermine those changes, thus reflecting the adaptive complexity inherent in public health systems. The purpose of this paper is to propose systems thinking as a conceptual rubric for the practice of team science in public health, and transdisciplinary, translational research as a catalyst for promoting the functional efficiency of science. The paper lays a foundation for the conceptual understanding of systems thinking and transdisciplinary research, and will provide illustrative examples within and beyond public health. A set of recommendations for a systems-centric approach to translational science will be presented.

Measuring collaboration and transdisciplinary integration in team science.

PURPOSE: As the science of team science evolves, the development of measures that assess important processes related to working in transdisciplinary teams is critical. Therefore, the purpose of this paper is to present the psychometric properties of scales measuring collaborative processes and transdisciplinary integration. METHODS: Two hundred-sixteen researchers and research staff participating in the Transdisciplinary Tobacco Use Research Centers (TTURC) Initiative completed the TTURC researcher survey. Confirmatory-factor analyses were used to verify the hypothesized factor structures. Descriptive data pertinent to these scales and their associations with other constructs were included to further examine the properties of the scales. RESULTS: Overall, the hypothesized-factor structures, with some minor modifications, were validated. A total of four scales were developed, three to assess collaborative processes (satisfaction with the collaboration, impact of collaboration, trust and respect) and one to assess transdisciplinary integration. All scales were found to have adequate internal consistency (i.e., Cronbach alpha's were all >0.70); were correlated with intermediate markers of collaborations (e.g., the collaboration and transdisciplinary-integration scales were positively associated with the perception of a center's making good progress in creating new methods, new science and models, and new interventions); and showed some ability to detect group differences. CONCLUSIONS: This paper provides valid tools that can be utilized to examine the underlying processes of team science--an important step toward advancing the science of team science.

The critical role of conflict resolution in teams: a close look at the links between conflict type, conflict management strategies, and team outcomes.

This article explores the linkages between strategies for managing different types of conflict and group performance and satisfaction. Results from a qualitative study of 57 autonomous teams suggest that groups that improve or maintain top performance over time share 3 conflict resolution tendencies: (a) focusing on the content of interpersonal interactions rather than delivery style, (b) explicitly discussing reasons behind any decisions reached in accepting and distributing work assignments, and (c) assigning work to members who have the relevant task expertise rather than assigning by other common means such as volunteering, default, or convenience. The authors' results also suggest that teams that are successful over time are likely to be both proactive in anticipating the need for conflict resolution and pluralistic in developing conflict resolution strategies that apply to all group members.

Hindsight is 20/20: Reflections on the evolution of concept mapping.

This paper considers the origins and development of the concept mapping methodology, a summary of its growth, and its influence in a variety of fields. From initial discussions with graduate students, through the rise of the theory-driven approach to program evaluation and the development of a theoretical framework for conceptualization methodology, the paper highlights some of the key early efforts and pilot projects that culminated in a 1989 special issue on the method in Evaluation and Program Planning that brought the method to the attention of the field of evaluation. The paper details the thinking that led to the standard version of the method (the analytic sequence, "bridging" index, and pattern matching) and the development of the software for accomplishing it. A bibliometric analysis shows that the rate of citation continues to increase, where it has grown geographically and institutionally, that the method has been used in a wide variety of disciplines and specialties, and that the literature had an influence on the field. The article concludes with a critical appraisal of some of the key aspects of the approach that warrant further development.

Introduction to a special issue on concept mapping.

Concept mapping was developed in the 1980s as a unique integration of qualitative (group process, brainstorming, unstructured sorting, interpretation) and quantitative (multidimensional scaling, hierarchical cluster analysis) methods designed to enable a group of people to articulate and depict graphically a coherent conceptual framework or model of any topic or issue of interest. This introduction provides the basic definition and description of the methodology for the newcomer and describes the steps typically followed in its most standard canonical form (preparation, generation, structuring, representation, interpretation and utilization). It also introduces this special issue which reviews the history of the methodology, describes its use in a variety of contexts, shows the latest ways it can be integrated with other methodologies, considers methodological advances and developments, and sketches a vision of the future of the method's evolution.

Evaluating program effectiveness using the regression point displacement design.

Most health care initiatives are evaluated using observational study designs in lieu of randomized controlled trials (RCT) due primarily to resource limitations. However, although observational studies are less expensive to implement and evaluate, they are also more problematic in determining causality than the RCT. This trade off is most apparent in the initial planning stage of program development. An RCT is generally preferred though the cost of implementing a pilot program using the RCT might outstrip the potential benefit if the desired results are not obtained. This article describes a simple quasi-experimental model called the regression point displacement (RPD) design, which compares the prepost results of a single or multiple treatment groups to that of a control population. This design has shown great potential in evaluating health care pilot programs or demonstration projects-especially those that are community based-due to its relative ease of implementation and low cost of analysis.

An introduction to concept mapping as a participatory public health research method.

In this article, the authors introduce concept mapping as a useful participatory research method for public health researchers interested in generating hypotheses and developing theory. The authors first provide an overview of concept mapping, which combines qualitative approaches with quantitative analytical tools to produce visual displays of the relationship between ideas. Then, they present an illustrative research application of the method to the exploration of women's perceptions of the relationship between residential neighborhood factors and intimate partner violence experiences. They give attention to the data collection and analysis procedures and to demonstrating the intricacies of using concept mapping for public health research purposes. Finally, the article concludes with a discussion of the unique contributions and challenges associated with concept mapping.

Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

Complementary and conventional medicine: a concept map.

BACKGROUND: Despite the substantive literature from survey research that has accumulated on complementary and alternative medicine (CAM) in the United States and elsewhere, very little research has been done to assess conceptual domains that CAM and conventional providers would emphasize in CAM survey studies. The objective of this study is to describe and interpret the results of concept mapping with conventional and CAM practitioners from a variety of backgrounds on the topic of CAM. METHODS: Concept mapping, including free sorts, ratings, and multidimensional scaling was used to organize conceptual domains relevant to CAM into a visual "cluster map." The panel consisted of CAM providers, conventional providers, and university faculty, and was convened to help formulate conceptual domains to guide the development of a CAM survey for use with United States military veterans. RESULTS: Eight conceptual clusters were identified: 1) Self-assessment, Self-care, and Quality of Life; 2) Health Status, Health Behaviors; 3) Self-assessment of Health; 4) Practical/Economic/ Environmental Concerns; 5) Needs Assessment; 6) CAM vs. Conventional Medicine; 7) Knowledge of CAM; and 8) Experience with CAM. The clusters suggest panelists saw interactions between CAM and conventional medicine as a critical component of the current medical landscape. CONCLUSIONS: Concept mapping provided insight into how CAM and conventional providers view the domain of health care, and was shown to be a useful tool in the formulation of CAM-related conceptual domains.

Setting objectives for community and systems change: an application of concept mapping for planning a statewide health improvement initiative.

The Hawaii Department of Health (HDOH) used concept mapping techniques to engage local stakeholders and national subject area experts in defining the community and system factors that affect individuals' behaviors related to tobacco, nutrition, and physical activity. Over eight working days, project participants brainstormed 496 statements (edited to a final set of 90), which were then sorted and rated for their importance and feasibility. A sequence of multivariate statistical analyses, including multidimensional scaling and hierarchical cluster analysis, generated maps and figures that were then interpreted by project stakeholders. The results were immediately incorporated into an official plan, approved by the governor and state legislature, recommending how Hawaii's tobacco settlement resources could be used to create sustainable changes in population health. The results also provide empirical support for the premise that both community and systems factors ought to be considered when planning comprehensive health improvement initiatives.

Evolutionary Evaluation: implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate.

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of "evidence-based programs". From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.

Evaluation guidelines for the Clinical and Translational Science Awards (CTSAs).

The National Center for Advancing Translational Sciences (NCATS), a part of the National Institutes of Health, currently funds the Clinical and Translational Science Awards (CTSAs), a national consortium of 61 medical research institutions in 30 states and the District of Columbia. The program seeks to transform the way biomedical research is conducted, speed the translation of laboratory discoveries into treatments for patients, engage communities in clinical research efforts, and train a new generation of clinical and translational researchers. An endeavor as ambitious and complex as the CTSA program requires high-quality evaluations in order to show that the program is well implemented, efficiently managed, and demonstrably effective. In this paper, the Evaluation Key Function Committee of the CTSA Consortium presents an overall framework for evaluating the CTSA program and offers policies to guide the evaluation work. The guidelines set forth are designed to serve as a tool for education within the CTSA community by illuminating key issues and practices that should be considered during evaluation planning, implementation, and utilization. Additionally, these guidelines can provide a basis for ongoing discussions about how the principles articulated in this paper can most effectively be translated into operational reality.

Estimating return on investment in translational research: methods and protocols.

Assessing the value of clinical and translational research funding on accelerating the translation of scientific knowledge is a fundamental issue faced by the National Institutes of Health (NIH) and its Clinical and Translational Awards (CTSAs). To address this issue, the authors propose a model for measuring the return on investment (ROI) of one key CTSA program, the clinical research unit (CRU). By estimating the economic and social inputs and outputs of this program, this model produces multiple levels of ROI: investigator, program, and institutional estimates. A methodology, or evaluation protocol, is proposed to assess the value of this CTSA function, with specific objectives, methods, descriptions of the data to be collected, and how data are to be filtered, analyzed, and evaluated. This article provides an approach CTSAs could use to assess the economic and social returns on NIH and institutional investments in these critical activities.