Rigor and reproducibility instruction in academic medical libraries.

Background: Concerns over scientific reproducibility have grown in recent years, leading the National Institutes of Health (NIH) to require researchers to address these issues in research grant applications. Starting in 2020, training grants were required to provide a plan for educating trainees in rigor and reproducibility. Academic medical centers have responded with different solutions to fill this educational need. As experienced instructors with expertise in topics relating to reproducibility, librarians can play a prominent role in providing trainings, classes, and events to educate investigators and trainees, and bolstering reproducibility in their communities. Case Presentations: This special report summarizes efforts at five institutions to provide education in reproducibility to biomedical and life sciences researchers. Our goal is to expand awareness of the range of approaches in providing reproducibility services in libraries. Conclusions: Reproducibility education by medical librarians can take many forms. These specific programs in reproducibility education build upon libraries' existing collaborations, with funder mandates providing a major impetus. Collaborator needs shaped the exact type of educational or other reproducibility support and combined with each library's strengths to yield a diversity of offerings based on capacity and interest. As demand for and complexity of reproducibility education increases due to new institutional and funder mandates, reproducibility education will merit special attention.

Piloting a long-term evaluation of library data workshops.

BACKGROUND: Over four years of hosting library data workshops, we conducted post-workshop evaluation of attendees' satisfaction with the workshops but not longer-term follow-up. To best allocate library resources and most effectively serve the needs of our users, we sought to determine whether our data workshops were impactful and useful to our community. This paper describes a pilot project to evaluate the impact of data workshops at our academic health sciences library. CASE PRESENTATION: We surveyed individuals who signed up for data workshops between 2016 and 2019. Surveys included open-ended and multiple-choice questions, with the goal of having participants describe their motivations for taking the workshop(s) and how they ultimately used what they learned. An analysis of responses using the Applied Thematic Analysis model indicated that the workshops had an impact on the respondents, although the strength of our conclusions is limited by a relatively low response rate. CONCLUSIONS: Survey results indicated that our workshops impacted how researchers at our medical center collect and analyze data, supporting the conclusion that we should concentrate our educational efforts on providing skills-based workshops. The low response rate and time-consuming nature of the analysis point toward several improvements for future evaluation efforts, including better tracking of workshop attendees, a shorter survey with fewer open-ended questions, and survey implementation within one year of the workshop date.

A model for initiating research data management services at academic libraries.

BACKGROUND: Librarians developed a pilot program to provide training, resources, strategies, and support for medical libraries seeking to establish research data management (RDM) services. Participants were required to complete eight educational modules to provide the necessary background in RDM. Each participating institution was then required to use two of the following three elements: (1) a template and strategies for data interviews, (2) the Teaching Toolkit to teach an introductory RDM class, or (3) strategies for hosting a data class series. CASE PRESENTATION: Six libraries participated in the pilot, with between two and eight librarians participating from each institution. Librarians from each institution completed the online training modules. Each institution conducted between six and fifteen data interviews, which helped build connections with researchers, and taught between one and five introductory RDM classes. All classes received very positive evaluations from attendees. Two libraries conducted a data series, with one bringing in instructors from outside the library. CONCLUSION: The pilot program proved successful in helping participating librarians learn about and engage with their research communities, jump-start their teaching of RDM, and develop institutional partnerships around RDM services. The practical, hands-on approach of this pilot proved to be successful in helping libraries with different environments establish RDM services. The success of this pilot provides a proven path forward for libraries that are developing data services at their own institutions.

The relative citation ratio: what is it and why should medical librarians care?

Bibliometrics is becoming increasingly prominent in the world of medical libraries. The number of presentations related to research impact at the Medical Library Association (MLA) annual meeting has been increasing in past years. Medical centers have been using institutional dashboards to track clinical performance for over a decade, and more recently, these institutional dashboards have included measures of academic performance. This commentary reviews current practices and considers the role for a newer metric, the relative citation ratio.

Data Day to Day: building a community of expertise to address data skills gaps in an academic medical center.

BACKGROUND: The New York University Health Sciences Library data services team had developed educational material for research data management and data visualization and had been offering classes at the request of departments, research groups, and training programs, but many members of the medical center were unaware of these library data services. There were also indications of data skills gaps in these subject areas and other data-related topics. CASE PRESENTATION: The data services team enlisted instructors from across the medical center with data expertise to teach in a series of classes hosted by the library. We hosted eight classes branded as a series called "Data Day to Day." Seven instructors from four units in the medical center, including the library, taught the classes. A multipronged outreach approach resulted in high turnout. Evaluations indicated that attendees were very satisfied with the instruction, would use the skills learned, and were interested in future classes. CONCLUSIONS: Data Day to Day met previously unaddressed data skills gaps. Collaborating with outside instructors allowed the library to serve as a hub for a broad range of data instruction and to raise awareness of library services. We plan to offer the series three times in the coming year with an expanding roster of classes.

Improving data collection, documentation, and workflow in a dementia screening study.

BACKGROUND: A clinical study team performing three multicultural dementia screening studies identified the need to improve data management practices and facilitate data sharing. A collaboration was initiated with librarians as part of the National Library of Medicine (NLM) informationist supplement program. The librarians identified areas for improvement in the studies' data collection, entry, and processing workflows. CASE PRESENTATION: The librarians' role in this project was to meet needs expressed by the study team around improving data collection and processing workflows to increase study efficiency and ensure data quality. The librarians addressed the data collection, entry, and processing weaknesses through standardizing and renaming variables, creating an electronic data capture system using REDCap, and developing well-documented, reproducible data processing workflows. CONCLUSIONS: NLM informationist supplements provide librarians with valuable experience in collaborating with study teams to address their data needs. For this project, the librarians gained skills in project management, REDCap, and understanding of the challenges and specifics of a clinical research study. However, the time and effort required to provide targeted and intensive support for one study team was not scalable to the library's broader user community.

Classifying publications from the clinical and translational science award program along the translational research spectrum: a machine learning approach.

BACKGROUND: Translational research is a key area of focus of the National Institutes of Health (NIH), as demonstrated by the substantial investment in the Clinical and Translational Science Award (CTSA) program. The goal of the CTSA program is to accelerate the translation of discoveries from the bench to the bedside and into communities. Different classification systems have been used to capture the spectrum of basic to clinical to population health research, with substantial differences in the number of categories and their definitions. Evaluation of the effectiveness of the CTSA program and of translational research in general is hampered by the lack of rigor in these definitions and their application. This study adds rigor to the classification process by creating a checklist to evaluate publications across the translational spectrum and operationalizes these classifications by building machine learning-based text classifiers to categorize these publications. METHODS: Based on collaboratively developed definitions, we created a detailed checklist for categories along the translational spectrum from T0 to T4. We applied the checklist to CTSA-linked publications to construct a set of coded publications for use in training machine learning-based text classifiers to classify publications within these categories. The training sets combined T1/T2 and T3/T4 categories due to low frequency of these publication types compared to the frequency of T0 publications. We then compared classifier performance across different algorithms and feature sets and applied the classifiers to all publications in PubMed indexed to CTSA grants. To validate the algorithm, we manually classified the articles with the top 100 scores from each classifier. RESULTS: The definitions and checklist facilitated classification and resulted in good inter-rater reliability for coding publications for the training set. Very good performance was achieved for the classifiers as represented by the area under the receiver operating curves (AUC), with an AUC of 0.94 for the T0 classifier, 0.84 for T1/T2, and 0.92 for T3/T4. CONCLUSIONS: The combination of definitions agreed upon by five CTSA hubs, a checklist that facilitates more uniform definition interpretation, and algorithms that perform well in classifying publications along the translational spectrum provide a basis for establishing and applying uniform definitions of translational research categories. The classification algorithms allow publication analyses that would not be feasible with manual classification, such as assessing the distribution and trends of publications across the CTSA network and comparing the categories of publications and their citations to assess knowledge transfer across the translational research spectrum.

Starting the data conversation: informing data services at an academic health sciences library.

OBJECTIVE: The research obtained information to plan data-related products and services. METHODS: Biomedical researchers in an academic medical center were selected using purposive sampling and interviewed using open-ended questions based on a literature review. Interviews were conducted until saturation was achieved. RESULTS: Interview responses informed library planners about researchers' key data issues. CONCLUSIONS: This approach proved valuable for planning data management products and services and raising library visibility among clients in the research data realm.

Leveraging technology and staffing in developing a new liaison program.

With nearly all library resources and services delivered digitally, librarians working for the New York University Health Sciences Library struggled with maintaining awareness of changing user needs, understanding barriers faced in using library resources and services, and determining knowledge management challenges across the organization. A liaison program was created to provide opportunities for librarians to meaningfully engage with users. The program was directed toward a subset of high-priority user groups to provide focused engagement with these users. Responsibility for providing routine reference service was reduced for liaison librarians to provide maximum time to engage with their assigned user communities.