Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study.

BACKGROUND: Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal. OBJECTIVE: This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators. METHODS: The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results. RESULTS: A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001). CONCLUSIONS: Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

Trends and Correlates of Hookah Use Among High School Students in North Carolina.

OBJECTIVES Although youth cigarette smoking has declined in the United States, use of alternative tobacco products, such as hookah, has increased. This study assesses changes in prevalence of use from 2011 to 2013, and examines factors associated with current hookah use among North Carolina high school students in 2013.METHODS Data came from the North Carolina Youth Tobacco Survey in 2011 (n = 4,791) and 2013 (n = 4,092). STATA (StataCorp LLC) logistic regression survey procedures account for the complex survey design and sampling weights.RESULTS Prevalence of reported current hookah use significantly increased from 3.6% (95% CI: 2.8-4.5) in 2011 to 6.1% (95% CI: 4.9-7.5) in 2013 while reported lifetime hookah use increased from 9.8% (95% CI: 8.0-12.0) in 2011 to 12.6% (95% CI: 11.0-14.4) in 2013. Correlates of current hookah use included having a weekly disposable income over $50 (adjusted odds ratio (AOR) = 2.05, 95% CI: 1.25-3.35), currently smoking cigarettes (AOR = 4.57, 95% CI: 1.80-11.62), and living with hookah users (AOR = 6.45, 95% CI: 3.21-12.93). Participant self-reports of "liking" or positively commenting on tobacco products on social media were associated with current hookah use (AOR = 1.83, 95% CI: 1.84-4.52). Frequent exposure to online tobacco advertisements (AOR = 1.61, 95% CI: 1.13-2.28) were also associated with current hookah use.CONCLUSIONS Comprehensive product specific communication and policy interventions are needed to educate youth about the dangers of hookah use and reduce social acceptability among youth. To decrease hookah use in North Carolina, policymakers should consider restoring funding for comprehensive tobacco prevention and control programs, and equalizing tobacco tax rates for all tobacco product types.

Electronic Cigarette Use Among High School Students and Its Association With Cigarette Use And Smoking Cessation, North Carolina Youth Tobacco Surveys, 2011 and 2013.

INTRODUCTION: Although adolescent cigarette use continues to decline in the United States, electronic cigarette (e­cigarette) use among adolescents has escalated rapidly. This study assessed trends and patterns of e­cigarette use and concurrent cigarette smoking and the relationships between e-cigarette use and smoking cessation intentions and behaviors among high school students in North Carolina. METHODS: Data came from high school students who completed the school-based, cross-sectional North Carolina Youth Tobacco Survey in 2011 (n = 4,791) and 2013 (n = 4,092). This study assessed changes in prevalence of e-cigarette and cigarette use from 2011 through 2013, and cessation-related factors associated with those students' current and past use of e­cigarettes in 2013. RESULTS: The prevalence of current e-cigarette use (use in the past 30 days) significantly increased from 1.7% (95% CI, 1.3%-2.2%) in 2011 to 7.7% (95% CI, 5.9%-10.0%) in 2013. Among dual users, current e-cigarette use was negatively associated with intention to quit cigarette smoking for good (relative risk ratio [RRR] = 0.51; 95% CI, 0.29-0.87) and with attempts to quit cigarette smoking in the past 12 months (RRR = 0.69; 95% CI, 0.49-0.97). Current e-cigarette smokers were less likely than those who only smoked cigarettes to have ever abstained from cigarette smoking for 6 months (RRR = 0.42; 95% CI, 0.21-0.82) or 1 year (RRR = 0.21; 95% CI, 0.09-0.51) and to have used any kind of aids for smoking cessation (RRR = 0.46; 95% CI, 0.29-0.74). CONCLUSION: Public health practitioners and cessation clinic service providers should educate adolescents about the risks of using any nicotine-containing products, including e-cigarettes, and provide adequate tobacco cessation resources and counseling to adolescent tobacco users.

The impact of smoke-free legislation on population health in North Carolina.

Since the first Surgeon General's report documented the health impact of smoking 50 years ago, North Carolina has made much progress in reducing tobacco use. This article focuses on tobacco-related policies and legislation that have contributed to this progress and discusses measures that could be taken to further reduce tobacco use.

Evaluating the impact of a CTSA program from 2008 to 2021 through bibliometrics, social network analysis, and altmetrics.

Introduction: We evaluate a CTSA program hub by applying bibliometrics, social network analysis (SNA), and altmetrics and examine the changes in research productivity, citation impact, research collaboration, and CTSA-supported research topics since our pilot study in 2017. Methods: The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2021. We applied measures and metrics from bibliometrics, SNA, and altmetrics to the dataset. In addition, we analyzed research topics and correlations between different metrics. Results: 1154 NC TraCS-supported publications generated over 53,560 citation counts by April 2021. The average cites per year and the relative citation ratio (RCR) mean of these publications improved from 33 and 2.26 in 2017 to 48 and 2.58 in 2021. The number of involved UNC units in the most published authors' collaboration network increased from 7 (2017) to 10 (2021). NC TraCS-supported co-authorship involved 61 NC organizations. PlumX metrics identified articles with the highest altmetrics scores. About 96% NC TraCS-supported publications have above the average SciVal Topic Prominence Percentile; the average approximate potential to translate of the included publication was 54.2%; and 177 publications addressed health disparity issues. Bibliometric measures (e.g., citation counts, RCR) and PlumX metrics (i.e., Citations, Captures, and Social-Media) are positively correlated (p < .05). Conclusion: Bibliometrics, SNA, and altmetrics offer distinctive but related perspectives to examine CTSA research performance and longitudinal growth, especially at the individual program hub level. These perspectives can help CTSAs build program foci.

Clarifying the concept of a learning health system for healthcare delivery organizations: Implications from a qualitative analysis of the scientific literature.

The "learning health system" (LHS) concept has been defined in broad terms, which makes it challenging for health system leaders to determine exactly what is required to transform their organization into an LHS. This study provides a conceptual map of the LHS landscape by identifying the activities, principles, tools, and conditions that LHS researchers have associated with the concept. Through a multi-step screening process, two researchers identified 79 publications from PubMed (published before January 2020) that contained information relevant to the question, "What work is required of a healthcare organization that is operating as an LHS?" Those publications were coded as to whether or not they referenced each of 94 LHS elements in the taxonomy developed by the study team. This taxonomy, named the Learning Health Systems Consolidated Framework (LHS-CF), organizes the elements into five "bodies of work" (organizational learning, translation of evidence into practice, building knowledge, analyzing clinical data, and engaging stakeholders) and four "enabling conditions" (workforce skilled for LHS work, data systems and informatics technology in place, organization invests resources in LHS work, and supportive organizational culture). We report the frequency that each of the 94 elements was referenced across the 79 publications. The four most referenced elements were: "organization builds knowledge or evidence," "quality improvement practices are standard practice," "patients and family members are actively engaged," and "organizational culture emphasizes and supports learning." By dissecting the LHS construct into its component elements, the LHS-CF taxonomy can serve as a useful tool for LHS researchers and practitioners in defining the aspects of LHS they are addressing. By assessing how often each element is referenced in the literature, the study provides guidance to health system leaders as to how their organization needs to evolve in order to become an LHS - while also recognizing that each organization should emphasize elements that are most aligned with their mission and goals.

Erratum: Opportunities for strengthening CTSA evaluation - CORRIGENDUM.

[This corrects the article DOI: 10.1017/cts.2019.387.].

Evaluating a CTSA-funded pilot grant program.

INTRODUCTION: Pilot programs are integral to catalyzing and accelerating research at Clinical and Translational Science Award (CTSA) hubs. However, little has been published about the structure and operationalization of pilot programs or how they impact the translational research enterprise at CTSAs. The North Carolina Translational and Clinical Science Institute (NC TraCS), the CTSA hub at the University of North Carolina at Chapel Hill (UNC-CH) conducted an evaluation case study to describe the pilot program structure, assess process outcomes, and provide a framework for other institutions to utilize for the evaluation of their respective pilot programs. METHODS: We describe the operationalization of our pilot program, the evaluation framework utilized to evaluate the program, and how we analyzed available data to understand how our pilot funding opportunities were utilized by investigators. We calculated application volumes and funding rates by investigator position title and pilot application type. We also reviewed feedback provided by pilot Principal Investigators (PIs) to understand how many pilot projects were completed, NC TraCS service utilization, and barriers to research. Limited data on publications and subsequent funding was also reviewed. RESULTS: Between 2009 and 2019 the NC TraCS Pilot Program received 2343 applications and funded 933 projects, ranging from $2000 to $100,000 in amount, with an overall funding rate of 39.8%. Utilization of NC TraCS services had positive impacts on both resubmission funding and project completion rates. CONCLUSION: This process evaluation indicates that the program is being operationalized in a way that successfully fulfills the program mission while meeting the needs of a diverse group of researchers.

Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study.

INTRODUCTION: To enhance the performance evaluation of Clinical and Translational Science Award (CTSA) hubs, we examined the utility of advanced bibliometric measures that go beyond simple publication counts to demonstrate the impact of translational research output. METHODS: The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2017. We adopted advanced bibliometric measures and a state-of-the-art bibliometric network analysis tool to assess research productivity, citation impact, the scope of research collaboration, and the clusters of research topics. RESULTS: Totally, 754 NC TraCS-supported publications generated over 24,000 citation counts by April 2017 with an average of 33 cites per article. NC TraCS-supported research papers received more than twice as many cites per year as the average National Institute of Health-funded research publications from the same field and time. We identified the top productive researchers and their networks within the CTSA hub. Findings demonstrated the impact of NC TraCS in facilitating interdisciplinary collaborations within the CTSA hub and across the CTSA consortium and connecting researchers with right peers and organizations. CONCLUSION: Both improved bibliometrics measures and bibliometric network analysis can bring new perspectives to CTSA evaluation via citation influence and the scope of research collaborations.

Opportunities for strengthening CTSA evaluation.

The purpose of the article is to describe the progress of the Clinical and Translational Science Award (CTSA) Program to address the evaluation-related recommendations made by the 2013 Institute of Medicine's review of the CTSA Program and guidelines published in CTS Journal the same year (Trochim et al., Clinical and Translational Science 2013; 6(4): 303-309). We utilize data from a 2018 national survey of evaluators administered to all 64 CTSA hubs and a content analysis of the role of evaluation in the CTSA Program Funding Opportunity Announcements to document progress. We present four new opportunities for further strengthening CTSA evaluation efforts: (1) continue to build the collaborative evaluation infrastructure at local and national levels; (2) make better use of existing data; (3) strengthen and augment the common metrics initiative; and (4) pursue internal and external opportunities to evaluate the CTSA program at the national level. This article will be of significant interest to the funders of the CTSA Program and the multiple stakeholders in the larger consortium and will promote dialog from the broad range of CTSA stakeholders about further strengthening the CTSA Program's evaluation.

Poly-Tobacco Use among High School Students.

Although cigarette use by adolescents is declining, emerging tobacco products are becoming increasingly popular and youth may use more than one type of tobacco product. The purposes of this study were: (1) to assess patterns of poly-tobacco use among a representative sample of high school students and (2) to determine how beliefs correlate with poly-tobacco use. Data came from the 2013 North Carolina Youth Tobacco Survey (n = 4092). SAS logistic regression survey procedures were used to account for the complex survey design and sampling weights. Among all high school students in NC in 2013, 29.7% reported current any tobacco use, with 19.1% reporting current poly-tobacco use, and 10.6% reporting current use of only one product. Among poly-tobacco users, 59.3% reported that one of the products they currently used was cigarettes. Positive tobacco product beliefs were found to be significantly associated with poly-tobacco use. Communication campaigns, policy efforts, and future research are needed for prevention, regulation, and control of poly-tobacco use among adolescents, which represents a significant public health problem.