A 12-gene pharmacogenetic panel to prevent adverse drug reactions: an open-label, multicentre, controlled, cluster-randomised crossover implementation study.

BACKGROUND: The benefit of pharmacogenetic testing before starting drug therapy has been well documented for several single gene-drug combinations. However, the clinical utility of a pre-emptive genotyping strategy using a pharmacogenetic panel has not been rigorously assessed. METHODS: We conducted an open-label, multicentre, controlled, cluster-randomised, crossover implementation study of a 12-gene pharmacogenetic panel in 18 hospitals, nine community health centres, and 28 community pharmacies in seven European countries (Austria, Greece, Italy, the Netherlands, Slovenia, Spain, and the UK). Patients aged 18 years or older receiving a first prescription for a drug clinically recommended in the guidelines of the Dutch Pharmacogenetics Working Group (ie, the index drug) as part of routine care were eligible for inclusion. Exclusion criteria included previous genetic testing for a gene relevant to the index drug, a planned duration of treatment of less than 7 consecutive days, and severe renal or liver insufficiency. All patients gave written informed consent before taking part in the study. Participants were genotyped for 50 germline variants in 12 genes, and those with an actionable variant (ie, a drug-gene interaction test result for which the Dutch Pharmacogenetics Working Group [DPWG] recommended a change to standard-of-care drug treatment) were treated according to DPWG recommendations. Patients in the control group received standard treatment. To prepare clinicians for pre-emptive pharmacogenetic testing, local teams were educated during a site-initiation visit and online educational material was made available. The primary outcome was the occurrence of clinically relevant adverse drug reactions within the 12-week follow-up period. Analyses were irrespective of patient adherence to the DPWG guidelines. The primary analysis was done using a gatekeeping analysis, in which outcomes in people with an actionable drug-gene interaction in the study group versus the control group were compared, and only if the difference was statistically significant was an analysis done that included all of the patients in the study. Outcomes were compared between the study and control groups, both for patients with an actionable drug-gene interaction test result (ie, a result for which the DPWG recommended a change to standard-of-care drug treatment) and for all patients who received at least one dose of index drug. The safety analysis included all participants who received at least one dose of a study drug. This study is registered with ClinicalTrials.gov, NCT03093818 and is closed to new participants. FINDINGS: Between March 7, 2017, and June 30, 2020, 41 696 patients were assessed for eligibility and 6944 (51·4 % female, 48·6% male; 97·7% self-reported European, Mediterranean, or Middle Eastern ethnicity) were enrolled and assigned to receive genotype-guided drug treatment (n=3342) or standard care (n=3602). 99 patients (52 [1·6%] of the study group and 47 [1·3%] of the control group) withdrew consent after group assignment. 652 participants (367 [11·0%] in the study group and 285 [7·9%] in the control group) were lost to follow-up. In patients with an actionable test result for the index drug (n=1558), a clinically relevant adverse drug reaction occurred in 152 (21·0%) of 725 patients in the study group and 231 (27·7%) of 833 patients in the control group (odds ratio [OR] 0·70 [95% CI 0·54-0·91]; p=0·0075), whereas for all patients, the incidence was 628 (21·5%) of 2923 patients in the study group and 934 (28·6%) of 3270 patients in the control group (OR 0·70 [95% CI 0·61-0·79]; p <0·0001). INTERPRETATION: Genotype-guided treatment using a 12-gene pharmacogenetic panel significantly reduced the incidence of clinically relevant adverse drug reactions and was feasible across diverse European health-care system organisations and settings. Large-scale implementation could help to make drug therapy increasingly safe. FUNDING: European Union Horizon 2020.

LinkExplorer: predicting, explaining and exploring links in large biomedical knowledge graphs.

SUMMARY: Machine learning algorithms for link prediction can be valuable tools for hypothesis generation. However, many current algorithms are black boxes or lack good user interfaces that could facilitate insight into why predictions are made. We present LinkExplorer, a software suite for predicting, explaining and exploring links in large biomedical knowledge graphs. LinkExplorer integrates our novel, rule-based link prediction engine SAFRAN, which was recently shown to outcompete other explainable algorithms and established black-box algorithms. Here, we demonstrate highly competitive evaluation results of our algorithm on multiple large biomedical knowledge graphs, and release a web interface that allows for interactive and intuitive exploration of predicted links and their explanations. AVAILABILITY AND IMPLEMENTATION: A publicly hosted instance, source code and further documentation can be found at https://github.com/OpenBioLink/Explorer. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals.

Publicly accessible benchmarks that allow for assessing and comparing model performances are important drivers of progress in artificial intelligence (AI). While recent advances in AI capabilities hold the potential to transform medical practice by assisting and augmenting the cognitive processes of healthcare professionals, the coverage of clinically relevant tasks by AI benchmarks is largely unclear. Furthermore, there is a lack of systematized meta-information that allows clinical AI researchers to quickly determine accessibility, scope, content and other characteristics of datasets and benchmark datasets relevant to the clinical domain. To address these issues, we curated and released a comprehensive catalogue of datasets and benchmarks pertaining to the broad domain of clinical and biomedical natural language processing (NLP), based on a systematic review of literature and. A total of 450 NLP datasets were manually systematized and annotated with rich metadata, such as targeted tasks, clinical applicability, data types, performance metrics, accessibility and licensing information, and availability of data splits. We then compared tasks covered by AI benchmark datasets with relevant tasks that medical practitioners reported as highly desirable targets for automation in a previous empirical study. Our analysis indicates that AI benchmarks of direct clinical relevance are scarce and fail to cover most work activities that clinicians want to see addressed. In particular, tasks associated with routine documentation and patient data administration workflows are not represented despite significant associated workloads. Thus, currently available AI benchmarks are improperly aligned with desired targets for AI automation in clinical settings, and novel benchmarks should be created to fill these gaps.

Improving the robustness and accuracy of biomedical language models through adversarial training.

Deep transformer neural network models have improved the predictive accuracy of intelligent text processing systems in the biomedical domain. They have obtained state-of-the-art performance scores on a wide variety of biomedical and clinical Natural Language Processing (NLP) benchmarks. However, the robustness and reliability of these models has been less explored so far. Neural NLP models can be easily fooled by adversarial samples, i.e. minor changes to input that preserve the meaning and understandability of the text but force the NLP system to make erroneous decisions. This raises serious concerns about the security and trust-worthiness of biomedical NLP systems, especially when they are intended to be deployed in real-world use cases. We investigated the robustness of several transformer neural language models, i.e. BioBERT, SciBERT, BioMed-RoBERTa, and Bio-ClinicalBERT, on a wide range of biomedical and clinical text processing tasks. We implemented various adversarial attack methods to test the NLP systems in different attack scenarios. Experimental results showed that the biomedical NLP models are sensitive to adversarial samples; their performance dropped in average by 21 and 18.9 absolute percent on character-level and word-level adversarial noise, respectively, on Micro-F1, Pearson Correlation, and Accuracy measures. Conducting extensive adversarial training experiments, we fine-tuned the NLP models on a mixture of clean samples and adversarial inputs. Results showed that adversarial training is an effective defense mechanism against adversarial noise; the models' robustness improved in average by 11.3 absolute percent. In addition, the models' performance on clean data increased in average by 2.4 absolute percent, demonstrating that adversarial training can boost generalization abilities of biomedical NLP systems. This study takes an important step towards revealing vulnerabilities of deep neural language models in biomedical NLP applications. It also provides practical and effective strategies to develop secure, trust-worthy, and accurate intelligent text processing systems in the biomedical domain.

OpenBioLink: a benchmarking framework for large-scale biomedical link prediction.

SUMMARY: Recently, novel machine-learning algorithms have shown potential for predicting undiscovered links in biomedical knowledge networks. However, dedicated benchmarks for measuring algorithmic progress have not yet emerged. With OpenBioLink, we introduce a large-scale, high-quality and highly challenging biomedical link prediction benchmark to transparently and reproducibly evaluate such algorithms. Furthermore, we present preliminary baseline evaluation results. AVAILABILITY AND IMPLEMENTATION: Source code and data are openly available at https://github.com/OpenBioLink/OpenBioLink. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

The roles of predictors in cardiovascular risk models - a question of modeling culture?

BACKGROUND: While machine learning (ML) algorithms may predict cardiovascular outcomes more accurately than statistical models, their result is usually not representable by a transparent formula. Hence, it is often unclear how specific values of predictors lead to the predictions. We aimed to demonstrate with graphical tools how predictor-risk relations in cardiovascular risk prediction models fitted by ML algorithms and by statistical approaches may differ, and how sample size affects the stability of the estimated relations. METHODS: We reanalyzed data from a large registry of 1.5 million participants in a national health screening program. Three data analysts developed analytical strategies to predict cardiovascular events within 1 year from health screening. This was done for the full data set and with gradually reduced sample sizes, and each data analyst followed their favorite modeling approach. Predictor-risk relations were visualized by partial dependence and individual conditional expectation plots. RESULTS: When comparing the modeling algorithms, we found some similarities between these visualizations but also occasional divergence. The smaller the sample size, the more the predictor-risk relation depended on the modeling algorithm used, and also sampling variability played an increased role. Predictive performance was similar if the models were derived on the full data set, whereas smaller sample sizes favored simpler models. CONCLUSION: Predictor-risk relations from ML models may differ from those obtained by statistical models, even with large sample sizes. Hence, predictors may assume different roles in risk prediction models. As long as sample size is sufficient, predictive accuracy is not largely affected by the choice of algorithm.

Generating evidence for precision medicine: considerations made by the Ubiquitous Pharmacogenomics Consortium when designing and operationalizing the PREPARE study.

OBJECTIVES: Pharmacogenetic panel-based testing represents a new model for precision medicine. A sufficiently powered prospective study assessing the (cost-)effectiveness of a panel-based pharmacogenomics approach to guide pharmacotherapy is lacking. Therefore, the Ubiquitous Pharmacogenomics Consortium initiated the PREemptive Pharmacogenomic testing for prevention of Adverse drug Reactions (PREPARE) study. Here, we provide an overview of considerations made to mitigate multiple methodological challenges that emerged during the design. METHODS: An evaluation of considerations made when designing the PREPARE study across six domains: study aims and design, primary endpoint definition and collection of adverse drug events, inclusion and exclusion criteria, target population, pharmacogenomics intervention strategy, and statistical analyses. RESULTS: Challenges and respective solutions included: (1) defining and operationalizing a composite primary endpoint enabling measurement of the anticipated effect, by including only severe, causal, and drug genotype-associated adverse drug reactions; (2) avoiding overrepresentation of frequently prescribed drugs within the patient sample while maintaining external validity, by capping drugs of enrolment; (3) designing the pharmacogenomics intervention strategy to be applicable across ethnicities and healthcare settings; and (4) designing a statistical analysis plan to avoid dilution of effect by initially excluding patients without a gene-drug interaction in a gatekeeping analysis. CONCLUSION: Our design considerations will enable quantification of the collective clinical utility of a panel of pharmacogenomics-markers within one trial as a proof-of-concept for pharmacogenomics-guided pharmacotherapy across multiple actionable gene-drug interactions. These considerations may prove useful to other investigators aiming to generate evidence for precision medicine.

Summarization of biomedical articles using domain-specific word embeddings and graph ranking.

Text summarization tools can help biomedical researchers and clinicians reduce the time and effort needed for acquiring important information from numerous documents. It has been shown that the input text can be modeled as a graph, and important sentences can be selected by identifying central nodes within the graph. However, the effective representation of documents, quantifying the relatedness of sentences, and selecting the most informative sentences are main challenges that need to be addressed in graph-based summarization. In this paper, we address these challenges in the context of biomedical text summarization. We evaluate the efficacy of a graph-based summarizer using different types of context-free and contextualized embeddings. The word representations are produced by pre-training neural language models on large corpora of biomedical texts. The summarizer models the input text as a graph in which the strength of relations between sentences is measured using the domain specific vector representations. We also assess the usefulness of different graph ranking techniques in the sentence selection step of our summarization method. Using the common Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics, we evaluate the performance of our summarizer against various comparison methods. The results show that when the summarizer utilizes proper combinations of context-free and contextualized embeddings, along with an effective ranking method, it can outperform the other methods. We demonstrate that the best settings of our graph-based summarizer can efficiently improve the informative content of summaries and decrease the redundancy.

Neural sentence embedding models for semantic similarity estimation in the biomedical domain.

BACKGROUND: Neural network based embedding models are receiving significant attention in the field of natural language processing due to their capability to effectively capture semantic information representing words, sentences or even larger text elements in low-dimensional vector space. While current state-of-the-art models for assessing the semantic similarity of textual statements from biomedical publications depend on the availability of laboriously curated ontologies, unsupervised neural embedding models only require large text corpora as input and do not need manual curation. In this study, we investigated the efficacy of current state-of-the-art neural sentence embedding models for semantic similarity estimation of sentences from biomedical literature. We trained different neural embedding models on 1.7 million articles from the PubMed Open Access dataset, and evaluated them based on a biomedical benchmark set containing 100 sentence pairs annotated by human experts and a smaller contradiction subset derived from the original benchmark set. RESULTS: Experimental results showed that, with a Pearson correlation of 0.819, our best unsupervised model based on the Paragraph Vector Distributed Memory algorithm outperforms previous state-of-the-art results achieved on the BIOSSES biomedical benchmark set. Moreover, our proposed supervised model that combines different string-based similarity metrics with a neural embedding model surpasses previous ontology-dependent supervised state-of-the-art approaches in terms of Pearson's r (r = 0.871) on the biomedical benchmark set. In contrast to the promising results for the original benchmark, we found our best models' performance on the smaller contradiction subset to be poor. CONCLUSIONS: In this study, we have highlighted the value of neural network-based models for semantic similarity estimation in the biomedical domain by showing that they can keep up with and even surpass previous state-of-the-art approaches for semantic similarity estimation that depend on the availability of laboriously curated ontologies, when evaluated on a biomedical benchmark set. Capturing contradictions and negations in biomedical sentences, however, emerged as an essential area for further work.

Fast and scalable neural embedding models for biomedical sentence classification.

BACKGROUND: Biomedical literature is expanding rapidly, and tools that help locate information of interest are needed. To this end, a multitude of different approaches for classifying sentences in biomedical publications according to their coarse semantic and rhetoric categories (e.g., Background, Methods, Results, Conclusions) have been devised, with recent state-of-the-art results reported for a complex deep learning model. Recent evidence showed that shallow and wide neural models such as fastText can provide results that are competitive or superior to complex deep learning models while requiring drastically lower training times and having better scalability. We analyze the efficacy of the fastText model in the classification of biomedical sentences in the PubMed 200k RCT benchmark, and introduce a simple pre-processing step that enables the application of fastText on sentence sequences. Furthermore, we explore the utility of two unsupervised pre-training approaches in scenarios where labeled training data are limited. RESULTS: Our fastText-based methodology yields a state-of-the-art F1 score of.917 on the PubMed 200k benchmark when sentence ordering is taken into account, with a training time of only 73 s on standard hardware. Applying fastText on single sentences, without taking sentence ordering into account, yielded an F1 score of.852 (training time 13 s). Unsupervised pre-training of N-gram vectors greatly improved the results for small training set sizes, with an increase of F1 score of.21 to.74 when trained on only 1000 randomly picked sentences without taking sentence ordering into account. CONCLUSIONS: Because of it's ease of use and performance, fastText should be among the first choices of tools when tackling biomedical text classification problems with large corpora. Unsupervised pre-training of N-gram vectors on domain-specific corpora also makes it possible to apply fastText when labeled training data are limited.

Toward a complete dataset of drug-drug interaction information from publicly available sources.

Although potential drug-drug interactions (PDDIs) are a significant source of preventable drug-related harm, there is currently no single complete source of PDDI information. In the current study, all publically available sources of PDDI information that could be identified using a comprehensive and broad search were combined into a single dataset. The combined dataset merged fourteen different sources including 5 clinically-oriented information sources, 4 Natural Language Processing (NLP) Corpora, and 5 Bioinformatics/Pharmacovigilance information sources. As a comprehensive PDDI source, the merged dataset might benefit the pharmacovigilance text mining community by making it possible to compare the representativeness of NLP corpora for PDDI text extraction tasks, and specifying elements that can be useful for future PDDI extraction purposes. An analysis of the overlap between and across the data sources showed that there was little overlap. Even comprehensive PDDI lists such as DrugBank, KEGG, and the NDF-RT had less than 50% overlap with each other. Moreover, all of the comprehensive lists had incomplete coverage of two data sources that focus on PDDIs of interest in most clinical settings. Based on this information, we think that systems that provide access to the comprehensive lists, such as APIs into RxNorm, should be careful to inform users that the lists may be incomplete with respect to PDDIs that drug experts suggest clinicians be aware of. In spite of the low degree of overlap, several dozen cases were identified where PDDI information provided in drug product labeling might be augmented by the merged dataset. Moreover, the combined dataset was also shown to improve the performance of an existing PDDI NLP pipeline and a recently published PDDI pharmacovigilance protocol. Future work will focus on improvement of the methods for mapping between PDDI information sources, identifying methods to improve the use of the merged dataset in PDDI NLP algorithms, integrating high-quality PDDI information from the merged dataset into Wikidata, and making the combined dataset accessible as Semantic Web Linked Data.

Utilizing the Wikidata system to improve the quality of medical content in Wikipedia in diverse languages: a pilot study.

BACKGROUND: Wikipedia is an important source of medical information for both patients and medical professionals. Given its wide reach, improving the quality, completeness, and accessibility of medical information on Wikipedia could have a positive impact on global health. OBJECTIVE: We created a prototypical implementation of an automated system for keeping drug-drug interaction (DDI) information in Wikipedia up to date with current evidence about clinically significant drug interactions. Our work is based on Wikidata, a novel, graph-based database backend of Wikipedia currently in development. METHODS: We set up an automated process for integrating data from the Office of the National Coordinator for Health Information Technology (ONC) high priority DDI list into Wikidata. We set up exemplary implementations demonstrating how the DDI data we introduced into Wikidata could be displayed in Wikipedia articles in diverse languages. Finally, we conducted a pilot analysis to explore if adding the ONC high priority data would substantially enhance the information currently available on Wikipedia. RESULTS: We derived 1150 unique interactions from the ONC high priority list. Integration of the potential DDI data from Wikidata into Wikipedia articles proved to be straightforward and yielded useful results. We found that even though the majority of current English Wikipedia articles about pharmaceuticals contained sections detailing contraindications, only a small fraction of articles explicitly mentioned interaction partners from the ONC high priority list. For 91.30% (1050/1150) of the interaction pairs we tested, none of the 2 articles corresponding to the interacting substances explicitly mentioned the interaction partner. For 7.21% (83/1150) of the pairs, only 1 of the 2 associated Wikipedia articles mentioned the interaction partner; for only 1.48% (17/1150) of the pairs, both articles contained explicit mentions of the interaction partner. CONCLUSIONS: Our prototype demonstrated that automated updating of medical content in Wikipedia through Wikidata is a viable option, albeit further refinements and community-wide consensus building are required before integration into public Wikipedia is possible. A long-term endeavor to improve the medical information in Wikipedia through structured data representation and automated workflows might lead to a significant improvement of the quality of medical information in one of the world's most popular Web resources.

Pharmacogenomic knowledge representation, reasoning and genome-based clinical decision support based on OWL 2 DL ontologies.

BACKGROUND: Every year, hundreds of thousands of patients experience treatment failure or adverse drug reactions (ADRs), many of which could be prevented by pharmacogenomic testing. However, the primary knowledge needed for clinical pharmacogenomics is currently dispersed over disparate data structures and captured in unstructured or semi-structured formalizations. This is a source of potential ambiguity and complexity, making it difficult to create reliable information technology systems for enabling clinical pharmacogenomics. METHODS: We developed Web Ontology Language (OWL) ontologies and automated reasoning methodologies to meet the following goals: 1) provide a simple and concise formalism for representing pharmacogenomic knowledge, 2) finde errors and insufficient definitions in pharmacogenomic knowledge bases, 3) automatically assign alleles and phenotypes to patients, 4) match patients to clinically appropriate pharmacogenomic guidelines and clinical decision support messages and 5) facilitate the detection of inconsistencies and overlaps between pharmacogenomic treatment guidelines from different sources. We evaluated different reasoning systems and test our approach with a large collection of publicly available genetic profiles. RESULTS: Our methodology proved to be a novel and useful choice for representing, analyzing and using pharmacogenomic data. The Genomic Clinical Decision Support (Genomic CDS) ontology represents 336 SNPs with 707 variants; 665 haplotypes related to 43 genes; 22 rules related to drug-response phenotypes; and 308 clinical decision support rules. OWL reasoning identified CDS rules with overlapping target populations but differing treatment recommendations. Only a modest number of clinical decision support rules were triggered for a collection of 943 public genetic profiles. We found significant performance differences across available OWL reasoners. CONCLUSIONS: The ontology-based framework we developed can be used to represent, organize and reason over the growing wealth of pharmacogenomic knowledge, as well as to identify errors, inconsistencies and insufficient definitions in source data sets or individual patient data. Our study highlights both advantages and potential practical issues with such an ontology-based approach.

A comparison of chain-of-thought reasoning strategies across datasets and models.

Emergent chain-of-thought (CoT) reasoning capabilities promise to improve the performance and explainability of large language models (LLMs). However, uncertainties remain about how reasoning strategies formulated for previous model generations generalize to new model generations and different datasets. In this small-scale study, we compare different reasoning strategies induced by zero-shot prompting across six recently released LLMs (davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl and Cohere command-xlarge). We test them on six question-answering datasets that require real-world knowledge application and logical verbal reasoning, including datasets from scientific and medical domains. Our findings demonstrate that while some variations in effectiveness occur, gains from CoT reasoning strategies remain robust across different models and datasets. GPT-4 benefits the most from current state-of-the-art reasoning strategies and performs best by applying a prompt previously discovered through automated discovery.

GPT-4 as a biomedical simulator.

BACKGROUND: Computational simulation of biological processes can be a valuable tool for accelerating biomedical research, but usually requires extensive domain knowledge and manual adaptation. Large language models (LLMs) such as GPT-4 have proven surprisingly successful for a wide range of tasks. This study provides proof-of-concept for the use of GPT-4 as a versatile simulator of biological systems. METHODS: We introduce SimulateGPT, a proof-of-concept for knowledge-driven simulation across levels of biological organization through structured prompting of GPT-4. We benchmarked our approach against direct GPT-4 inference in blinded qualitative evaluations by domain experts in four scenarios and in two quantitative scenarios with experimental ground truth. The qualitative scenarios included mouse experiments with known outcomes and treatment decision support in sepsis. The quantitative scenarios included prediction of gene essentiality in cancer cells and progression-free survival in cancer patients. RESULTS: In qualitative experiments, biomedical scientists rated SimulateGPT's predictions favorably over direct GPT-4 inference. In quantitative experiments, SimulateGPT substantially improved classification accuracy for predicting the essentiality of individual genes and increased correlation coefficients and precision in the regression task of predicting progression-free survival. CONCLUSION: This proof-of-concept study suggests that LLMs may enable a new class of biomedical simulators. Such text-based simulations appear well suited for modeling and understanding complex living systems that are difficult to describe with physics-based first-principles simulations, but for which extensive knowledge is available as written text. Finally, we propose several directions for further development of LLM-based biomedical simulators, including augmentation through web search retrieval, integrated mathematical modeling, and fine-tuning on experimental data.

Utilization and perceived problems of online medical resources and search tools among different groups of European physicians.

BACKGROUND: There is a large body of research suggesting that medical professionals have unmet information needs during their daily routines. OBJECTIVE: To investigate which online resources and tools different groups of European physicians use to gather medical information and to identify barriers that prevent the successful retrieval of medical information from the Internet. METHODS: A detailed Web-based questionnaire was sent out to approximately 15,000 physicians across Europe and disseminated through partner websites. 500 European physicians of different levels of academic qualification and medical specialization were included in the analysis. Self-reported frequency of use of different types of online resources, perceived importance of search tools, and perceived search barriers were measured. Comparisons were made across different levels of qualification (qualified physicians vs physicians in training, medical specialists without professorships vs medical professors) and specialization (general practitioners vs specialists). RESULTS: Most participants were Internet-savvy, came from Austria (43%, 190/440) and Switzerland (31%, 137/440), were above 50 years old (56%, 239/430), stated high levels of medical work experience, had regular patient contact and were employed in nonacademic health care settings (41%, 177/432). All groups reported frequent use of general search engines and cited "restricted accessibility to good quality information" as a dominant barrier to finding medical information on the Internet. Physicians in training reported the most frequent use of Wikipedia (56%, 31/55). Specialists were more likely than general practitioners to use medical research databases (68%, 185/274 vs 27%, 24/88; χ²2=44.905, P<.001). General practitioners were more likely than specialists to report "lack of time" as a barrier towards finding information on the Internet (59%, 50/85 vs 43%, 111/260; χ²1=7.231, P=.007) and to restrict their search by language (48%, 43/89 vs 35%, 97/278; χ²1=5.148, P=.023). They frequently consult general health websites (36%, 31/87 vs 19%, 51/269; χ²2=12.813, P=.002) and online physician network communities (17%, 15/86, χ²2=9.841 vs 6%, 17/270, P<.001). CONCLUSIONS: The reported inaccessibility of relevant, trustworthy resources on the Internet and frequent reliance on general search engines and social media among physicians require further attention. Possible solutions may be increased governmental support for the development and popularization of user-tailored medical search tools and open access to high-quality content for physicians. The potential role of collaborative tools in providing the psychological support and affirmation normally given by medical colleagues needs further consideration. Tools that speed up quality evaluation and aid selection of relevant search results need to be identified. In order to develop an adequate search tool, a differentiated approach considering the differing needs of physician subgroups may be beneficial.

ThoughtSource: A central hub for large language model reasoning data.

Large language models (LLMs) such as GPT-4 have recently demonstrated impressive results across a wide range of tasks. LLMs are still limited, however, in that they frequently fail at complex reasoning, their reasoning processes are opaque, they are prone to 'hallucinate' facts, and there are concerns about their underlying biases. Letting models verbalize reasoning steps as natural language, a technique known as chain-of-thought prompting, has recently been proposed as a way to address some of these issues. Here we present ThoughtSource, a meta-dataset and software library for chain-of-thought (CoT) reasoning. The goal of ThoughtSource is to improve future artificial intelligence systems by facilitating qualitative understanding of CoTs, enabling empirical evaluations, and providing training data. This first release of ThoughtSource integrates seven scientific/medical, three general-domain and five math word question answering datasets.

The Arden Syntax standard for clinical decision support: experiences and directions.

Arden Syntax is a widely recognized standard for representing clinical and scientific knowledge in an executable format. It has a history that reaches back until 1989 and is currently maintained by the Health Level 7 (HL7) organization. We created a production-ready development environment, compiler, rule engine and application server for Arden Syntax. Over the course of several years, we have applied this Arden - Syntax - based CDS system in a wide variety of clinical problem domains, such as hepatitis serology interpretation, monitoring of nosocomial infections or the prediction of metastatic events in melanoma patients. We found the Arden Syntax standard to be very suitable for the practical implementation of CDS systems. Among the advantages of Arden Syntax are its status as an actively developed HL7 standard, the readability of the syntax, and various syntactic features such as flexible list handling. A major challenge we encountered was the technical integration of our CDS systems in existing, heterogeneous health information systems. To address this issue, we are currently working on incorporating the HL7 standard GELLO, which provides a standardized interface and query language for accessing data in health information systems. We hope that these planned extensions of the Arden Syntax might eventually help in realizing the vision of a global, interoperable and shared library of clinical decision support knowledge.

A data-driven living review for pharmacogenomic decision support in cancer treatment.

With drastically decreasing costs of genetic sequencing, it has become feasible to use individual genetic markers to optimize treatment selection in cancer therapy. However, it is still difficult for medical practitioners to integrate these new kinds of data into clinical routine, since available information is growing rapidly. We demonstrate how a blend of manual curation and automated data extraction and evidence synthesis can be used to generate a 'living review', a summarization of current evidence on cancer classification, corresponding genetic markers, genetic tests and treatment options that can be used by clinicians to refine treatment choices. In contrast to a classical review, this automated 'living review' offers the opportunity of automatically updating core content when available data changes, making it easier to keep an overview of the best current evidence. We discuss some of the findings we made while creating a prototype of a 'living review' for colorectal cancer pharmacotherapy.

A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks.

Research in artificial intelligence (AI) is addressing a growing number of tasks through a rapidly growing number of models and methodologies. This makes it difficult to keep track of where novel AI methods are successfully - or still unsuccessfully - applied, how progress is measured, how different advances might synergize with each other, and how future research should be prioritized. To help address these issues, we created the Intelligence Task Ontology and Knowledge Graph (ITO), a comprehensive, richly structured and manually curated resource on artificial intelligence tasks, benchmark results and performance metrics. The current version of ITO contains 685,560 edges, 1,100 classes representing AI processes and 1,995 properties representing performance metrics. The primary goal of ITO is to enable analyses of the global landscape of AI tasks and capabilities. ITO is based on technologies that allow for easy integration and enrichment with external data, automated inference and continuous, collaborative expert curation of underlying ontological models. We make the ITO dataset and a collection of Jupyter notebooks utilizing ITO openly available.