Towards a Reporting Guideline for Studies on Information Extraction from Clinical Texts.

BACKGROUND: The rapid technical progress in the domain of clinical Natural Language Processing and information extraction (IE) has resulted in challenges concerning the comparability and replicability of studies. AIM: This paper proposes a reporting guideline to standardize the description of methodologies and outcomes for studies involving IE from clinical texts. METHODS: The guideline is developed based on the experiences gained from data extraction for a previously conducted scoping review on IE from free-text radiology reports including 34 studies. RESULTS: The guideline comprises the five top-level categories information model, architecture, data, annotation, and outcomes. In total, we define 28 aspects to be reported on in IE studies related to these categories. CONCLUSIONS: The proposed guideline is expected to set a standard for reporting in studies describing IE from clinical text and promote uniformity across the research field. Expected future technological advancements may make regular updates of the guideline necessary. In future research, we plan to develop a taxonomy that clearly defines corresponding value sets as well as integrating both this guideline and the taxonomy by following a consensus-based methodology.

Transforming experimental radiology: Design and implementation of an innovative ePACS image storage system for AI imaging research environments.

INTRODUCTION AND PURPOSE: We present the needs, design, development, implementation, and accessibility of a crafted experimental PACS (ePACS) system to securely store images, ensuring efficiency and ease of use for AI processing, specifically tailored for research scenarios, including phantoms, animal and human studies and quality assurance (QA) exams. The ePACS system plays a crucial role in any medical imaging departments that handle non-care profile studies, such as protocol adjustments and dummy runs. By effectively segregating non-care profile studies from the healthcare assistance, the ePACS usefully prevents errors both in clinical practice and storage security. METHODS AND RESULTS: The developed ePACS system considers the best practices for management, maintenance, access, long-term storage and backups, regulatory audits, and economic aspects. Moreover, key aspects of the ePACS system include the design of data flows with a focus on incorporating data security and privacy, access control and levels based on user profiles, internal data management policies, standardized architecture, infrastructure and application monitorization and traceability, and periodic backup policies. A new tool called DicomStudiesQA has been developed to standardize the analysis of DICOM studies. The tool automatically identifies, extracts, and renames series using a consistent nomenclature. It also detects corrupted images and merges separated dynamic series that were initially split, allowing for streamlined post-processing. DISCUSSION AND CONCLUSIONS: The developed ePACS system encompasses a successful implementation, both in hospital and research environments, showcasing its transformative nature and the challenging yet crucial transfer of knowledge to industry. This underscores the practicality and real-world applicability of our innovative approach, highlighting the significant impact it has on the field of experimental radiology.

GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet.

BACKGROUND: Large language models (LLMs) that can efficiently screen and identify studies meeting specific criteria would streamline literature reviews. Additionally, those capable of extracting data from publications would enhance knowledge discovery by reducing the burden on human reviewers. METHODS: We created an automated pipeline utilizing OpenAI GPT-4 32 K API version "2023-05-15" to evaluate the accuracy of the LLM GPT-4 responses to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in PubMed. We presented the 60 papers to GPT-4 in four configurations: (1) all 60 questions simultaneously; (2) all 60 questions simultaneously with the instruction sheet; (3) each of the 60 questions individually; and (4) each of the 60 questions individually with the instruction sheet. RESULTS: GPT-4 achieved a mean accuracy of 86.9% - 24.0% higher than when the answers to papers were permuted. The overall recall and precision were 72.5% and 87.4%, respectively. The standard deviation of three replicates for the 60 questions ranged from 0 to 5.3% with a median of 1.2%. The instruction sheet did not significantly increase GPT-4's accuracy, recall, or precision. GPT-4 was more likely to provide false positive answers when the 60 questions were submitted individually compared to when they were submitted together. CONCLUSIONS: GPT-4 reproducibly answered 3600 questions about 60 papers on HIVDR with moderately high accuracy, recall, and precision. The instruction sheet's failure to improve these metrics suggests that more sophisticated approaches are necessary. Either enhanced prompt engineering or finetuning an open-source model could further improve an LLM's ability to answer questions about highly specialized HIVDR papers.

Efficiency and comparability of using new evidence platforms for updating recommendations: Experience with a type-2 diabetes guideline in Colombia.

Introduction: Updating recommendations for guidelines requires a comprehensive and efficient literature search. Although new information platforms are available for developing groups, their relative contributions to this purpose remain uncertain. Methods: As part of a review/update of eight selected evidence-based recommendationsfor type 2 diabetes, we evaluated the following five literature search approaches (targeting systematic reviews, using predetermined criteria): PubMed for MEDLINE, Epistemonikos database basic search, Epistemonikos database using a structured search strategy, Living overview of evidence (L.OVE) platform, and TRIP database. Three reviewers independently classified the retrieved references as definitely eligible, probably eligible, or not eligible. Those falling in the same "definitely" categories for all reviewers were labelled as "true" positives/negatives. The rest went to re-assessment and if found eligible/not eligible by consensus became "false" negatives/positives, respectively. We described the yield for each approach and computed "diagnostic accuracy" measures and agreement statistics. Results: Altogether, the five approaches identified 318 to 505 references for the eight recommendations, from which reviewers considered 4.2 to 9.4% eligible after the two rounds. While Pubmed outperformed the other approaches (diagnostic odds ratio 12.5 versus 2.6 to 5.3), no single search approach returned eligible references for all recommendations. Individually, searches found up to 40% of all eligible references (n = 71), and no combination of any three approaches could find over 80% of them. Kappa statistics for retrieval between searches were very poor (9 out of 10 paired comparisons did not surpass the chance-expected agreement). Conclusion: Among the information platforms assessed, PubMed appeared to be more efficient in updating this set of recommendations. However, the very poor agreement among search approaches in the reference yield demands that developing groups add information from several (probably more than three) sources for this purpose. Further research is needed to replicate our findings and enhance our understanding of how to efficiently update recommendations.

Introducción: La actualización de recomendaciones de las guías de práctica clínica requiere búsquedas bibliográficas exhaustivas y eficientes. Aunque están disponibles nuevas plataformas de información para grupos desarrolladores, su contribución a este propósito sigue siendo incierta. Métodos: Como parte de una revisión/actualización de 8 recomendaciones basadas en evidencia seleccionadas sobre diabetes tipo 2, evaluamos las siguientes cinco aproximaciones de búsqueda bibliográfica (dirigidas a revisiones sistemáticas, utilizando criterios predeterminados): PubMed para MEDLINE; Epistemonikos utilizando una búsqueda básica; Epistemonikos utilizando una estrategia de búsqueda estructurada; plataforma (L.OVE) y TRIP . Tres revisores clasificaron de forma independiente las referencias recuperadas como definitivamente o probablemente elegibles/no elegibles. Aquellas clasificadas en las mismas categorías "definitivas" para todos los revisores, se etiquetaron como "verdaderas" positivas/negativas. El resto se sometieron a una nueva evaluación y, si se consideraban por consenso elegibles/no elegibles, se convirtieron en "falsos" negativos/positivos, respectivamente. Describimos el rendimiento de cada aproximación, junto a sus medidas de "precisión diagnóstica" y las estadísticas de acuerdo. Resultados: En conjunto, las cinco aproximaciones identificaron 318-505 referencias para las 8 recomendaciones, de las cuales los revisores consideraron elegibles el 4,2 a 9,4% tras las dos rondas. Mientras que Pubmed superó a las otras aproximaciones (odds ratio de diagnóstico 12,5 versus 2,6 a 53), ninguna aproximación de búsqueda identificó por sí misma referencias elegibles para todas las recomendaciones. Individualmente, las búsquedas identificaron hasta el 40% de todas las referencias elegibles (n=71), y ninguna combinación de cualquiera de los tres enfoques pudo identificar más del 80% de ellas. Las estadísticas Kappa para la recuperación entre búsquedas fueron muy pobres (9 de cada 10 comparaciones pareadas no superaron el acuerdo esperado por azar). Conclusiones: Entre las plataformas de información evaluadas, Pubmed parece ser la más eficiente para actualizar este conjunto de recomendaciones. Sin embargo, la escasa concordancia en el rendimiento de las referencias exige que los grupos desarrolladores incorporen información de varias fuentes (probablemente más de tres) para este fin. Es necesario seguir investigando para replicar nuestros hallazgos y mejorar nuestra comprensión de cómo actualizar recomendaciones de forma eficiente.

Meeting a need: development and validation of PubMed search filters for immigrant populations.

Objective: There is a need for additional comprehensive and validated filters to find relevant references more efficiently in the growing body of research on immigrant populations. Our goal was to create reliable search filters that direct librarians and researchers to pertinent studies indexed in PubMed about health topics specific to immigrant populations. Methods: We applied a systematic and multi-step process that combined information from expert input, authoritative sources, automation, and manual review of sources. We established a focused scope and eligibility criteria, which we used to create the development and validation sets. We formed a term ranking system that resulted in the creation of two filters: an immigrant-specific and an immigrant-sensitive search filter. Results: When tested against the validation set, the specific filter sensitivity was 88.09%, specificity 97.26%, precision 97.88%, and the NNR 1.02. The sensitive filter sensitivity was 97.76%when tested against the development set. The sensitive filter had a sensitivity of 97.14%, specificity of 82.05%, precision of 88.59%, accuracy of 90.94%, and NNR [See Table 1] of 1.13 when tested against the validation set. Conclusion: We accomplished our goal of developing PubMed search filters to help researchers retrieve studies about immigrants. The specific and sensitive PubMed search filters give information professionals and researchers options to maximize the specificity and precision or increase the sensitivity of their search for relevant studies in PubMed. Both search filters generated strong performance measurements and can be used as-is, to capture a subset of immigrant-related literature, or adapted and revised to fit the unique research needs of specific project teams (e.g. remove US-centric language, add location-specific terminology, or expand the search strategy to include terms for the topic/s being investigated in the immigrant population identified by the filter). There is also a potential for teams to employ the search filter development process described here for their own topics and use.

MIMIC-IV on FHIR: converting a decade of in-patient data into an exchangeable, interoperable format.

OBJECTIVE: Convert the Medical Information Mart for Intensive Care (MIMIC)-IV database into Health Level 7 Fast Healthcare Interoperability Resources (FHIR). Additionally, generate and publish an openly available demo of the resources, and create a FHIR Implementation Guide to support and clarify the usage of MIMIC-IV on FHIR. MATERIALS AND METHODS: FHIR profiles and terminology system of MIMIC-IV were modeled from the base FHIR R4 resources. Data and terminology were reorganized from the relational structure into FHIR according to the profiles. Resources generated were validated for conformance with the FHIR profiles. Finally, FHIR resources were published as newline delimited JSON files and the profiles were packaged into an implementation guide. RESULTS: The modeling of MIMIC-IV in FHIR resulted in 25 profiles, 2 extensions, 35 ValueSets, and 34 CodeSystems. An implementation guide encompassing the FHIR modeling can be accessed at mimic.mit.edu/fhir/mimic. The generated demo dataset contained 100 patients and over 915 000 resources. The full dataset contained 315 000 patients covering approximately 5 840 000 resources. The final datasets in NDJSON format are accessible on PhysioNet. DISCUSSION: Our work highlights the challenges and benefits of generating a real-world FHIR store. The challenges arise from terminology mapping and profiling modeling decisions. The benefits come from the extensively validated openly accessible data created as a result of the modeling work. CONCLUSION: The newly created MIMIC-IV on FHIR provides one of the first accessible deidentified critical care FHIR datasets. The extensive real-world data found in MIMIC-IV on FHIR will be invaluable for research and the development of healthcare applications.

Molecular data storage with zero synthetic effort and simple read-out.

Compound mixtures represent an alternative, additional approach to DNA and synthetic sequence-defined macromolecules in the field of non-conventional molecular data storage, which may be useful depending on the target application. Here, we report a fast and efficient method for information storage in molecular mixtures by the direct use of commercially available chemicals and thus, zero synthetic steps need to be performed. As a proof of principle, a binary coding language is used for encoding words in ASCII or black and white pixels of a bitmap. This way, we stored a 25 × 25-pixel QR code (625 bits) and a picture of the same size. Decoding of the written information is achieved via spectroscopic (1H NMR) or chromatographic (gas chromatography) analysis. In addition, for a faster and automated read-out of the data, we developed a decoding software, which also orders the data sets according to an internal "ordering" standard. Molecular keys or anticounterfeiting are possible areas of application for information-containing compound mixtures.

Validation of EHR medication fill data obtained through electronic linkage with pharmacies.

BACKGROUND: Recent linkages between electronic health records (EHRs) and pharmacy data hold opportunity for up-to-date assessment of medication adherence at the point of care. OBJECTIVE: To validate linked EHR-pharmacy data, which can be used for point-of-care interventions for concordance with insurance claims data for patients in a large health care delivery system. METHODS: We performed a retrospective cohort study of adult patients with an active antihypertensive medication order and seen as outpatients between August 25, 2019, and August 31, 2019. Pharmacy fill information was obtained from the EHR via linkages with Surescripts pharmacy and pharmacy benefit manager data, as well as from insurance claims available at our institution. We matched antihypertensive medication fills observed in the linked EHR-pharmacy database with available fills in the insurance claims database and calculated the percentage of medication fills that were available in each database. We estimated medication adherence using proportion of days covered in the linked EHR-pharmacy database and in the insurance claims database. RESULTS: Of 26,679 patients with hypertension, 23,348 (87.5%) had at least 1 antihypertensive medication fill recorded in the linked EHR-pharmacy database. Of 1,501 patients matched with the insurance database and with a documented medication fill, a fill was present for 1,484 (98.9%) and 1,259 (83.9%) patients in the linked EHR-pharmacy and insurance databases, respectively. Of 12,109 medication fills recorded in the insurance data, we found an overlap of 11,060 (91.3%) fills with the linked EHR-pharmacy database. The linked EHR-pharmacy database also contained 18,232 of 19,281 (94.6%) medication fills present in either database. Measured medication adherence was higher for patients when based on linked EHR-pharmacy data compared with insurance claims data (42% vs 30%, P < 0.001). CONCLUSIONS: Linked EHR-pharmacy data captured medication fills for the vast majority of patients and resulted in higher estimates of adherence than insurance claims. Our results suggest that pharmacy fill data available in the EHR have sufficient reliability to be used for point-of-care assessment of medication adherence. DISCLOSURES: This study was supported by grant R01HL155149 from the National Heart, Lung, and Blood Institute. Allen Thorpe provided funding for the NYU Langone Health Learning Health System Program, which helped fund this project. The authors have nothing to disclose.

GRADE Notes 2: Criteria for searching non-randomized or indirect evidence should be defined early in the guideline production process.

OBJECTIVE: To discuss two alternative approaches for complementing the body of direct evidence from Randomized Controlled Trials (RCTs) when it is judged insufficient from a guideline panel making recommendations. The approaches included expanding the evidence's body to non-randomises studies on the population of interest or to RCTs on indirect populations. STUDY DESIGN AND SETTING: In this report, we adopt the perspective of an evidence review team developing guidelines following the GRADE approach. Our experience is based on the development of two evidence-based guidelines promoted by The Italian National Institute of Health (ISS) and focusing on diagnosis and treatment of Autism Spectrum Disorders (ASD) in children/adolescents and adults. RESULTS: We left panel members deciding case by case whether the direct evidence from RCTs was sufficient or not and indicating which alternative to implement. This strategy presented unanticipated challenges both from an organizational and methodological standpoint. CONCLUSION: We suggest an early-stage production of a research protocol to define the criteria for expanding the body of evidence. These criteria should be informed by considerations around the certainty in the evidence, the clinical applicability of the results, feasibility and conflict of interest.

The methodological quality of individual participant data meta-analysis on intervention effects: systematic review.

OBJECTIVE: To assess the methodological quality of individual participant data (IPD) meta-analysis and to identify areas for improvement. DESIGN: Systematic review. DATA SOURCES: Medline, Embase, and Cochrane Database of Systematic Reviews. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Systematic reviews with IPD meta-analyses of randomised controlled trials on intervention effects published in English. RESULTS: 323 IPD meta-analyses covering 21 clinical areas and published between 1991 and 2019 were included: 270 (84%) were non-Cochrane reviews and 269 (84%) were published in journals with a high impact factor (top quarter). The IPD meta-analyses showed low compliance in using a satisfactory technique to assess the risk of bias of the included randomised controlled trials (43%, 95% confidence interval 38% to 48%), accounting for risk of bias when interpreting results (40%, 34% to 45%), providing a list of excluded studies with justifications (32%, 27% to 37%), establishing an a priori protocol (31%, 26% to 36%), prespecifying methods for assessing both the overall effects (44%, 39% to 50%) and the participant-intervention interactions (31%, 26% to 36%), assessing and considering the potential of publication bias (31%, 26% to 36%), and conducting a comprehensive literature search (19%, 15% to 23%). Up to 126 (39%) IPD meta-analyses failed to obtain IPD from 90% or more of eligible participants or trials, among which only 60 (48%) provided reasons and 21 (17%) undertook certain strategies to account for the unavailable IPD. CONCLUSIONS: The methodological quality of IPD meta-analyses is unsatisfactory. Future IPD meta-analyses need to establish an a priori protocol with prespecified data syntheses plan, comprehensively search the literature, critically appraise included randomised controlled trials with appropriate technique, account for risk of bias during data analyses and interpretation, and account for unavailable IPD.

A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools.

OBJECTIVE: We compared the process of developing searches with and without using text-mining tools (TMTs) for evidence synthesis products. STUDY DESIGN: This descriptive comparative analysis included seven systematic reviews, classified as simple or complex. Two librarians created MEDLINE strategies for each review, using either usual practice (UP) or TMTs. For each search we calculated sensitivity, number-needed-to-read (NNR) and time spent developing the search strategy. RESULTS: We found UP searches were more sensitive (UP 92% (95% CI, 85-99); TMT 84.9% (95% CI, 74.4-95.4)), with lower NNR (UP 83 (SD 34); TMT 90 (SD 68)). UP librarians spent an average of 12 h (SD 8) developing search strategies, compared to TMT librarians' 5 hours (SD 2). CONCLUSION: Across all reviews, TMT searches were less sensitive than UP searches, but confidence intervals overlapped. For simple SR topics, TMT searches were faster and slightly less sensitive than UP. For complex SR topics, TMT searches were faster and less sensitive than UP searches but identified unique eligible citations not found by the UP searches.

Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews.

BACKGROUND AND OBJECTIVE: To examine whether the use of natural language processing (NLP) technology is effective in assisting rapid title and abstract screening when updating a systematic review. STUDY DESIGN: Using the searched literature from a published systematic review, we trained and tested an NLP model that enables rapid title and abstract screening when updating a systematic review. The model was a light gradient boosting machine (LightGBM), an ensemble learning classifier which integrates four pretrained Bidirectional Encoder Representations from Transformers (BERT) models. We divided the searched citations into two sets (ie, training and test sets). The model was trained using the training set and assessed for screening performance using the test set. The searched citations, whose eligibility was determined by two independent reviewers, were treated as the reference standard. RESULTS: The test set included 947 citations; our model included 340 citations, excluded 607 citations, and achieved 96% sensitivity, and 78% specificity. If the classifier assessment in the case study was accepted, reviewers would lose 8 of 180 eligible citations (4%), none of which were ultimately included in the systematic review after full-text consideration, while decreasing the workload by 64.1%. CONCLUSION: NLP technology using the ensemble learning method may effectively assist in rapid literature screening when updating systematic reviews.

Assessing the roles and challenges of librarians in dental systematic and scoping reviews.

OBJECTIVE: The objective of this study was to determine the scope of experience, roles, and challenges that librarians face in participating in dental and oral health systematic and scoping reviews to inform outreach efforts to researchers and identify areas for librarian professional development. METHODS: The authors developed a twenty-three-item survey based on the findings of two recent articles about health sciences librarians' roles and challenges in conducting systematic and scoping reviews. The survey was distributed via electronic mailing lists to librarians who were likely to have participated in conducting dental systematic and scoping reviews. RESULTS: While survey respondents reported participating in many dental reviews, they participated more commonly in systematic reviews than in scoping reviews. Also, they worked less commonly on dental and oral health reviews than on non-dental reviews. Librarian roles in dental reviews tended to follow traditional librarian roles: all respondents had participated in planning and information retrieval stages, whereas fewer respondents had participated in screening and assessing articles. The most frequently reported challenges involved the lead reviewer or review team rather than the librarians themselves, with time- and methodology-related challenges being most common. CONCLUSIONS: Although librarians might not be highly involved in dental and oral health systematic and scoping reviews, more librarian participation in these reviews, either as methodologists or information experts, may improve their reviews' overall quality.

Causality dilemma: creating a twenty-first century university archive.

For its fifteenth anniversary, the Jay Sexter Library at Touro University Nevada (TUN) sought ways to capture its institutional history by founding an archive. Among many challenges, the library struggled to convince the administration of the importance of an archive. To generate interest in TUN's history, a task force comprising library, executive administration, and advancement staff hosted and recorded a panel event with some of the university's original faculty, staff, and administration. By having this event, new TUN employees were able to experience the shared knowledge of TUN's early days, and the library was able to create and preserve its own institutional history.

Medical Library Association Diversity and Inclusion Task Force Report.

The Medical Library Association (MLA) appointed a Diversity and Inclusion Task Force (DITF) in 2017. Sandra G. Franklin, AHIP, FMLA, chaired the task force and guided initiatives. From 2017 to 2020, the task force completed a review of MLA defining documents-including the mission, vision, values, and code of ethics-resulting in language updates to these documents. As MLA transitioned through the communities process, the DITF contributed to the transition. Other recommended essential changes to MLA profiles to promote awareness included updating pronouns to promote gender inclusivity and suggestions for the Annual Meeting Innovation Task Force. DITF members actively brought diversity and inclusion programming and engagement to MLA members at annual meetings. The task force held a fish bowl conversation, an open forum, and a Diversity Dialogues roundtable discussion; provided interactive discussion boards; and designed an MLA diversity button. Beyond MLA annual meetings, the task force hosted two critical librarianship meetings and a Twitter chat to engage MLA members with diversity and inclusion topics. Task force members promoted diversity and inclusion beyond their task force appointments with presentations at chapter meetings and other non-DITF MLA annual meeting programming. A notable task force accomplishment included completing a survey of MLA members to gather baseline demographic characteristics, including never before collected data about disability, socioeconomics, and caregiver status. This report provides an overview of DITF activities from 2017 to 2020.