Manual versus machine: How accurately does the Medical Text Indexer (MTI) classify different document types into disease areas?

The Medical Subject Headings (MeSH) thesaurus is a controlled vocabulary developed by the U.S. National Library of Medicine (NLM) for classifying journal articles. It is increasingly used by researchers studying medical innovation to classify text into disease areas and other categories. Although this process was once manual, human indexers are now assisted by algorithms that automate some of the indexing process. NLM has made one of their algorithms, the Medical Text Indexer (MTI), available to researchers. MTI can be used to easily assign MeSH descriptors to arbitrary text, including from document types other than publications. However, the reliability of extending MTI to other document types has not been studied directly. To assess this, we collected text from grants, patents, and drug indications, and compared MTI's classification to expert manual classification of the same documents. We examined MTI's recall (how often correct terms were identified) and found that MTI identified 78% of expert-classified MeSH descriptors for grants, 78% for patents, and 86% for drug indications. This high recall could be driven merely by excess suggestions (at an extreme, all diseases being assigned to a piece of text); therefore, we also examined precision (how often identified terms were correct) and found that most MTI outputs were also identified by expert manual classification: precision was 53% for grant text, 73% for patent text, and 64% for drug indications. Additionally, we found that recall and precision could be improved by (i) utilizing ranking scores provided by MTI, (ii) excluding long documents, and (iii) aggregating to higher MeSH categories. For simply detecting the presence of any disease, MTI showed > 94% recall and > 87% precision. Our overall assessment is that MTI is a potentially useful tool for researchers wishing to classify texts from a variety of sources into disease areas.

Interactive Images in Library Instruction: A Case Study.

This paper discusses the design and implementation of the H5P image hotspots and describes additional use cases for library instruction in order to teach learners from across the country on how different spaces can be used to create private, accessible areas where patrons can conduct telehealth appointments. The Network of the National Library of Medicine created an online class using H5P image hotspots to allow learners to virtually explore a physical space. The class used Moodle, an open-source learning platform, and H5P image hotspots to demonstrate how libraries implemented telehealth programs.

Database resources of the National Center for Biotechnology Information.

The National Center for Biotechnology Information (NCBI) provides online information resources for biology, including the GenBank® nucleic acid sequence database and the PubMed® database of citations and abstracts published in life science journals. NCBI provides search and retrieval operations for most of these data from 35 distinct databases. The E-utilities serve as the programming interface for most of these databases. Resources receiving significant updates in the past year include PubMed, PMC, Bookshelf, SciENcv, the NIH Comparative Genomics Resource (CGR), NCBI Virus, SRA, RefSeq, foreign contamination screening tools, Taxonomy, iCn3D, ClinVar, GTR, MedGen, dbSNP, ALFA, ClinicalTrials.gov, Pathogen Detection, antimicrobial resistance resources, and PubChem. These resources can be accessed through the NCBI home page at https://www.ncbi.nlm.nih.gov.

Notes on the data quality of bibliographic records from the MEDLINE database.

The US National Library of Medicine has created and maintained the PubMed® database, a collection of over 33.8 million records that contain citations and abstracts from the biomedical and life sciences literature. This database is an important resource for researchers and information service providers alike. As part of our work related to the creation of an author graph for coronaviruses, we encountered several data quality issues with records from a curated subset of the PubMed database called MEDLINE. We provide a data quality assessment for records selected from the MEDLINE database and report on several issues ranging from parsing issues (e.g. character encodings and schema definition weaknesses) to low scores for identifiers against several data quality metrics (e.g. completeness, validity and uniqueness). Database URL  https://pubmed.ncbi.nlm.nih.gov.

Future advancement of health care through standardized nursing terminologies: reflections from a Friends of the National Library of Medicine workshop honoring Virginia K. Saba.

OBJECTIVE: To honor the legacy of nursing informatics pioneer and visionary, Dr. Virginia Saba, the Friends of the National Library of Medicine convened a group of international experts to reflect on Dr. Saba's contributions to nursing standardized nursing terminologies. PROCESS: Experts led a day-and-a-half virtual update on nursing's sustained and rigorous efforts to develop and use valid, reliable, and computable standardized nursing terminologies over the past 5 decades. Over the course of the workshop, policymakers, industry leaders, and scholars discussed the successful use of standardized nursing terminologies, the potential for expanded use of these vetted tools to advance healthcare, and future needs and opportunities. In this article, we elaborate on this vision and key recommendations for continued and expanded adoption and use of standardized nursing terminologies across settings and systems with the goal of generating new knowledge that improves health. CONCLUSION: Much of the promise that the original creators of standardized nursing terminologies envisioned has been achieved. Secondary analysis of clinical data using these terminologies has repeatedly demonstrated the value of nursing and nursing's data. With increased and widespread adoption, these achievements can be replicated across settings and systems.

Automated indexing using NLM's Medical Text Indexer (MTI) compared to human indexing in Medline: a pilot study.

Objective: In 2002, the National Library of Medicine (NLM) introduced semi-automated indexing of Medline using the Medical Text Indexer (MTI). In 2021, NLM announced that it would fully automate its indexing in Medline with an improved MTI by mid-2022. This pilot study examines indexing using a sample of records in Medline from 2000, and how an early, public version of MTI's outputs compares to records created by human indexers. Methods: This pilot study examines twenty Medline records from 2000, a year before the MTI was introduced as a MeSH term recommender. We identified twenty higher- and lower-impact biomedical journals based on Journal Impact Factor (JIF) and examined the indexing of papers by feeding their PubMed records into the Interactive MTI tool. Results: In the sample, we found key differences between automated and human-indexed Medline records: MTI assigned more terms and used them more accurately for citations in the higher JIF group, and MTI tended to rank the Male check tag more highly than the Female check tag and to omit Aged check tags. Sometimes MTI chose more specific terms than human indexers but was inconsistent in applying specificity principles. Conclusion: NLM's transition to fully automated indexing of the biomedical literature could introduce or perpetuate inconsistencies and biases in Medline. Librarians and searchers should assess changes to index terms, and their impact on PubMed's mapping features for a range of topics. Future research should evaluate automated indexing as it pertains to finding clinical information effectively, and in performing systematic searches.

Promoting rural residents' participation in clinical trials: clinical trials basics programming and training for rural public librarians.

Background: Having diverse representation in clinical trial participation is important. Historically, rural residents have been underrepresented in clinical trial research. Public librarians have an opportunity to promote clinical trial participation among rural residents by offering consumer health information services that help patrons to understand what clinical trials are and how they can find relevant clinical trials. Case Presentation: A consumer health library and a clinical trial center located at a large academic medical center collaborated to provide clinical trial information programming to rural public libraries. The group was awarded a Network of the National Library of Medicine (NNLM) Community Outreach Grant and was able to plan, develop, promote, and implement programs including training workshops, a speaker event, and a book discussion to rural public librarians. Discussion: Marketing the programs to rural public libraries was difficult and many barriers were encountered. Though registration and subsequent participation were low, participants expressed interest and gratitude for the programs. For any future programs targeting this population, further strategies will need to be implemented to ensure increased registrations and attendees.

PubMed's core clinical journals filter: redesigned for contemporary clinical impact and utility.

Objective: The Core Clinical Journals (CCJ) list, produced by the U.S. National Library of Medicine (NLM), has been used by clinicians and librarians for half a century for two main purposes: narrowing a literature search to clinically useful journals and identifying high priority titles for library collections. After documentation of low usage of the existing CCJ, a review was undertaken to assess current validity, followed by an update to current clinical needs. Methods: As the subject coverage of the 50-year-old list had never been evaluated, the CCJ committee began its innovative step-wise approach by analyzing the existing subject scope. To determine whether clinical subjects had changed over the last half-century, the committee collected data on journal usage in hospitals and medical facilities, adding journal usage from Morning Report blogs recording the journal article citations used by physicians and residents in response to clinical questions. Patient-driven high-frequency diagnoses and subjects added contextual data by depicting the clinical environment. Results: The analysis identified a total of 80 subjects and selected 241 journals for the updated Clinical Journals filter, based on actual clinical utility of each journal. Discussion: These data-driven methods created a different framework for evaluating the structure and content of this filter. It is the real-world evidence needed to highlight CCJ clinical impact and push clinically useful journals to first page results. Since the new process resulted in a new product, the name warrants a change from Core Clinical Journals (CCJ) to Clinically Useful Journals (CUJ). Therefore, the redesigned NLM Core Clinical Journals/AIM set from this point forward will be referred to as Clinically Useful Journals (CUJ). The evidence-based process used to reframe evaluation of the clinical impact and utility of biomedical journals is documented in this article.

Collaborative collection development: a MedPrint case report.

Background: In response to several of Texas' largest medical libraries being forced to discard all serial print holdings, the Texas A&M University System and University of Texas System's Joint Library Facility (JLF) staff worked to help provide a solution to save and store these resources. This process fire-started a comprehensive effort by JLF staff to contact the National Library of Medicine (NLM) and devise a blueprint that would be used to help save and preserve all serial medical resources listed in NLM's medical retention program. Case Presentation: In an unprecedented approach, the Texas A&M JLF staff launched efforts to collect and preserve the complete holdings range of all NLM MedPrint periodical runs. This case report details the planning and steps JLF staff took to accomplish this feat; highlights important matters of consideration for the medical community which heavily relies upon continuous access to MedPrint materials; and provides insight on the apparent preservation vulnerabilities these materials increasingly face in an environment where digitization may create a false sense of security. Discussion: By May 2021, JLF had collected complete title runs up to year 2000 for 202 of the 254 MedPrint titles, which consists of more than twelve thousand volumes. These efforts proved particularly beneficial in the wake of the COVID-19 pandemic, which forced NLM to halt ILL processing from their print collection. During this time, JLF was uniquely positioned to meet and respond to the historic high number of medical literature ILL requests it received during this time.

2022 Janet Doe Lecture, health science libraries in the emerging digital information era: charting the course.

The great challenge medical library professionals are facing is how we evolve and respond to the emerging digital era. If we successfully understand and adapt to the emerging digital information environment, medical librarians/Health Information Professionals (HIPs) can play an even greater role in the advance in the health care of our nation and its residents. The opportunities and challenges are at the level we successfully responded to in the late 1960's and the 1970's under the leadership of the National Library of Medicine with its MEDLARS/Medline programs and Medical Library Assistance Act which enabled medical libraries to enter what I have referred to as The Golden Age of Medical Libraries. In this presentation, I focused on the transition of the health-related print Knowledge-Based Information base to the emerging digital health-related ecosystem. I review how this transition is being driven by evolving information technology. The development of "data driven health care" built on this emerging information ecosystem is being led by the National Library of Medicine's 2017-2027 Strategic plan and the Medical Library Association's programs in support of developing medical librarian/HIP's training, skills, and services to support their users access and use of this rapidly expanding health information ecosystem. I then present a brief description of the digital health information ecosystem that is just starting to emerge and the emerging new roles and services HIPs and their libraries are developing to support effective institutional access and use.

Examination of reporting status of pediatric oncology trials within the national library of medicine's trial database.

Finding safer and more effective treatment options are critical in progressing the field of pediatric oncology. These treatment options are discovered through completion and publication of clinical trials. The primary objective of this study was to assess the overall study characteristics of pediatric oncology clinical trials initiated between 2008 and 2021. The secondary objective of our study was to assess rates of discontinuation and reporting of results as required by the Food and Drug Administration (FDA). After acquiring pediatric oncology clinical trials from ClinicalTrials.gov, a cross-sectional study was performed. Included trials have an intervention exclusive to pediatrics and were conducted between 2008 and 2021. The results measured were characteristics of the clinical trials and their rate of discontinuation. Of the 7,930 trials originally returned from the search, 349 trials met inclusion criteria. The majority of the trials were phase 1 and 2 pharmaceutical interventions studying brain and blood cancer. Our study found that 14.9% (52) of the pediatric oncology trials were discontinued. Given the breadth of study within pediatric oncology, our overarching assessment shows that drug trials geared toward treating cancers of the brain and blood were dominant in the field. It is crucial for the advancement of science that results of trials are known. This avoids duplication of studies and waste of funds. Of the trials that were completed, 40.3% (58) did not report results to ClinicalTrials.gov. The nonreporting of this data limits the information available delaying the advancement of treatment options.

Bookshelf: A Biomedical Database of Books and Documents.

Bookshelf is a database maintained by the National Center for Biotechnology Information (NCBI) at the National Library of Medicine that contains freely accessible online biomedical documents, including systematic reviews, technical reports, textbooks, and reference books. The database allows users to browse and search across all content and within individual books, and it is linked to other NCBI content. This article provides an overview of Bookshelf and demonstrates its usage in a sample search. The resources available in Bookshelf are useful for students, researchers, healthcare professionals, and librarians.

Chemical identification and indexing in full-text articles: an overview of the NLM-Chem track at BioCreative VII.

The BioCreative National Library of Medicine (NLM)-Chem track calls for a community effort to fine-tune automated recognition of chemical names in the biomedical literature. Chemicals are one of the most searched biomedical entities in PubMed, and-as highlighted during the coronavirus disease 2019 pandemic-their identification may significantly advance research in multiple biomedical subfields. While previous community challenges focused on identifying chemical names mentioned in titles and abstracts, the full text contains valuable additional detail. We, therefore, organized the BioCreative NLM-Chem track as a community effort to address automated chemical entity recognition in full-text articles. The track consisted of two tasks: (i) chemical identification and (ii) chemical indexing. The chemical identification task required predicting all chemicals mentioned in recently published full-text articles, both span [i.e. named entity recognition (NER)] and normalization (i.e. entity linking), using Medical Subject Headings (MeSH). The chemical indexing task required identifying which chemicals reflect topics for each article and should therefore appear in the listing of MeSH terms for the document in the MEDLINE article indexing. This manuscript summarizes the BioCreative NLM-Chem track and post-challenge experiments. We received a total of 85 submissions from 17 teams worldwide. The highest performance achieved for the chemical identification task was 0.8672 F-score (0.8759 precision and 0.8587 recall) for strict NER performance and 0.8136 F-score (0.8621 precision and 0.7702 recall) for strict normalization performance. The highest performance achieved for the chemical indexing task was 0.6073 F-score (0.7417 precision and 0.5141 recall). This community challenge demonstrated that (i) the current substantial achievements in deep learning technologies can be utilized to improve automated prediction accuracy further and (ii) the chemical indexing task is substantially more challenging. We look forward to further developing biomedical text-mining methods to respond to the rapid growth of biomedical literature. The NLM-Chem track dataset and other challenge materials are publicly available at https://ftp.ncbi.nlm.nih.gov/pub/lu/BC7-NLM-Chem-track/. Database URL https://ftp.ncbi.nlm.nih.gov/pub/lu/BC7-NLM-Chem-track/.

Database resources of the National Center for Biotechnology Information in 2023.

The National Center for Biotechnology Information (NCBI) provides online information resources for biology, including the GenBank® nucleic acid sequence database and the PubMed® database of citations and abstracts published in life science journals. NCBI provides search and retrieval operations for most of these data from 35 distinct databases. The E-utilities serve as the programming interface for most of these databases. New resources include the Comparative Genome Resource (CGR) and the BLAST ClusteredNR database. Resources receiving significant updates in the past year include PubMed, PMC, Bookshelf, IgBLAST, GDV, RefSeq, NCBI Virus, GenBank type assemblies, iCn3D, ClinVar, GTR, dbGaP, ALFA, ClinicalTrials.gov, Pathogen Detection, antimicrobial resistance resources, and PubChem. These resources can be accessed through the NCBI home page at https://www.ncbi.nlm.nih.gov.

Automatic Mapping of Terminology Items with Transformers.

Biomedical ontologies are a key component in many systems for the analysis of textual clinical data. They are employed to organize information about a certain domain relying on a hierarchy of different classes. Each class maps a concept to items in a terminology developed by domain experts. These mappings are then leveraged to organize the information extracted by Natural Language Processing (NLP) models to build knowledge graphs for inferences. The creation of these associations, however, requires extensive manual review. In this paper, we present an automated approach and repeatable framework to learn a mapping between ontology classes and terminology terms derived from vocabularies in the Unified Medical Language System (UMLS) metathesaurus. According to our evaluation, the proposed system achieves a performance close to humans and provides a substantial improvement over existing systems developed by the National Library of Medicine to assist researchers through this process.

Effects of Porting Essie Tokenization and Normalization to Solr.

Search for information is now an integral part of healthcare. Searches are enabled by search engines whose objective is to efficiently retrieve the relevant information for the user query. When it comes to retrieving biomedical text and literature, Essie search engine developed at the National Library of Medicine (NLM) performs exceptionally well. However, Essie is a software system developed for NLM that has ceased development and support. On the other hand, Solr is a popular opensource enterprise search engine used by many of the world's largest internet sites, offering continuous developments and improvements along with the state-of-the-art features. In this paper, we present our approach to porting the key features of Essie and developing custom components to be used in Solr. We demonstrate the effectiveness of the added components on three benchmark biomedical datasets. The custom components may aid the community in improving search methods for biomedical text retrieval.

Evaluation of Discrepancies Among National Library of Medicine (NLM) Value Set Authority Center (VSAC) ICD-10-CM Value Sets: Case Study for Diagnoses of Common Chronic Conditions, Implications, and Potential Solutions.

The National Library of Medicine (NLM)'s Value Set Authority Center (VSAC) is a crowd-sourced repository with a potential for substantial discrepancy among value sets for the same clinical concepts. To characterize this potential problem, we identified the most common chronic conditions affecting US adults and assessed for discrepancy among VSAC ICD-10-CM value sets for these conditions. An analysis of 32 value sets for 12 conditions identified that a median of 45% of codes for a given condition were potentially problematic (included in at least one, but not all, theoretically equivalent value sets). These problematic codes were used to document clinical care for potentially over 20 million patients in a data warehouse of approximately 150 million US adults. Users of VSAC diagnosis value sets should be cognizant of the prevalence of these discrepancies and take proactive steps to mitigate their impact. Further research is warranted to characterize and address this issue.

Expanding the Legacy: The National Library of Medicine Michael E. DeBakey Lecture in the History of Medicine.

In our 2021 article published in this journal, we described the development, historical significance, and impact of the National Library of Medicine (NLM) Michael E. DeBakey fellowship in the History of Medicine. This article focuses on a key part of the fellowship, the NLM Michael E. DeBakey Lecture in the History of Medicine, by explaining how this annual program advances historical scholarship and promotes awareness of DeBakey's legacy and his support of the world's largest biomedical library, whose collections are appreciated by researchers worldwide. The annual DeBakey Lecture provides a platform for a selected DeBakey fellow to share and expand on their fellowship research, connecting that research and the fellow's story with a global audience through a videocast, a permanently and freely available archived lecture, a research-based blog post, and an associated blog interview. The lectures have covered topics about DeBakey himself, his influence on the world, and new research that reflects his historical interests. The library's support of this impactful program, like the Michael E. DeBakey fellowship overall, testifies to its commitment to expanding the legacy of DeBakey hand in hand with its commitment to serving scientists and society in the 21st century.

NLM-Chem-BC7: manually annotated full-text resources for chemical entity annotation and indexing in biomedical articles.

The automatic recognition of chemical names and their corresponding database identifiers in biomedical text is an important first step for many downstream text-mining applications. The task is even more challenging when considering the identification of these entities in the article's full text and, furthermore, the identification of candidate substances for that article's metadata [Medical Subject Heading (MeSH) article indexing]. The National Library of Medicine (NLM)-Chem track at BioCreative VII aimed to foster the development of algorithms that can predict with high quality the chemical entities in the biomedical literature and further identify the chemical substances that are candidates for article indexing. As a result of this challenge, the NLM-Chem track produced two comprehensive, manually curated corpora annotated with chemical entities and indexed with chemical substances: the chemical identification corpus and the chemical indexing corpus. The NLM-Chem BioCreative VII (NLM-Chem-BC7) Chemical Identification corpus consists of 204 full-text PubMed Central (PMC) articles, fully annotated for chemical entities by 12 NLM indexers for both span (i.e. named entity recognition) and normalization (i.e. entity linking) using MeSH. This resource was used for the training and testing of the Chemical Identification task to evaluate the accuracy of algorithms in predicting chemicals mentioned in recently published full-text articles. The NLM-Chem-BC7 Chemical Indexing corpus consists of 1333 recently published PMC articles, equipped with chemical substance indexing by manual experts at the NLM. This resource was used for the evaluation of the Chemical Indexing task, which evaluated the accuracy of algorithms in predicting the chemicals that should be indexed, i.e. appear in the listing of MeSH terms for the document. This set was further enriched after the challenge in two ways: (i) 11 NLM indexers manually verified each of the candidate terms appearing in the prediction results of the challenge participants, but not in the MeSH indexing, and the chemical indexing terms appearing in the MeSH indexing list, but not in the prediction results, and (ii) the challenge organizers algorithmically merged the chemical entity annotations in the full text for all predicted chemical entities and used a statistical approach to keep those with the highest degree of confidence. As a result, the NLM-Chem-BC7 Chemical Indexing corpus is a gold-standard corpus for chemical indexing of journal articles and a silver-standard corpus for chemical entity identification in full-text journal articles. Together, these resources are currently the most comprehensive resources for chemical entity recognition, and we demonstrate improvements in the chemical entity recognition algorithms. We detail the characteristics of these novel resources and make them available for the community. Database URL: https://ftp.ncbi.nlm.nih.gov/pub/lu/NLM-Chem-BC7-corpus/.

Trends in Modernization at the Nation's Medical Library.

This manuscript describes the ongoing efforts to meet ever-changing patron needs by modernizing operations, infrastructure, workforce, and customer service at the National Library of Medicine (NLM). Charged with collecting, organizing, preserving, and disseminating biomedical and life sciences information to the public, the NLM constantly adapts to evolving technologies and scientific advancements in order to better fulfill its mission. Historical context and an overview of operational decision-making offer a window into the workings of our nation's medical library.