Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation.

OBJECTIVE: The paper introduces a framework for the evaluation of the encoding of factual scientific knowledge, designed to streamline the manual evaluation process typically conducted by domain experts. Inferring over and extracting information from Large Language Models (LLMs) trained on a large corpus of scientific literature can potentially define a step change in biomedical discovery, reducing the barriers for accessing and integrating existing medical evidence. This work explores the potential of LLMs for dialoguing with biomedical background knowledge, using the context of antibiotic discovery. METHODS: The framework involves three evaluation steps, each assessing different aspects sequentially: fluency, prompt alignment, semantic coherence, factual knowledge, and specificity of the generated responses. By splitting these tasks between non-experts and experts, the framework reduces the effort required from the latter. The work provides a systematic assessment on the ability of eleven state-of-the-art LLMs, including ChatGPT, GPT-4 and Llama 2, in two prompting-based tasks: chemical compound definition generation and chemical compound-fungus relation determination. RESULTS: Although recent models have improved in fluency, factual accuracy is still low and models are biased towards over-represented entities. The ability of LLMs to serve as biomedical knowledge bases is questioned, and the need for additional systematic evaluation frameworks is highlighted. CONCLUSION: While LLMs are currently not fit for purpose to be used as biomedical factual knowledge bases in a zero-shot setting, there is a promising emerging property in the direction of factuality as the models become domain specialised, scale up in size and level of human feedback.

Epidemiological and clinical analysis of polish short-term and long-term travelers returning from tropical countries.

BACKGROUND: There is a lack of information about health problems after returning from tropical countries among travelers from Poland. The aim was to create characteristics of diseases imported to Poland from tropical regions and to determine the changes in the travel trends. METHOD: This retrospective study is based on medical records of 2391 Polish patients >18 years old, hospitalized between 2006 and 2016 after returning from tropical areas. The analysis covered purpose, duration and travel destination, and health problems related to the travel. 1098 patients (short travel, <1 month, n = 345 vs long travel, >6 months, n = 753) were selected for further analysis. RESULTS: The most frequently visited region was Sub-Saharan Africa. Tourists dominated among short-term and missionaries among long-term travelers. The most popular health problems in both groups were digestive system disease and febrile diseases. Diarrhoea of undetermined aetiology, dengue fever, malaria, fever of unknown aetiology and infectious mononucleosis were more likely to occur among short-term travelers whereas blastocystosis, giardiasis, schistosomiasis among long-term travelers. In the group of long-term travelers 363/753 (47,8%) were diagnosed with an infectious or parasitic disease in relation to a trip to a country with a hot climate. CONCLUSIONS: Tropical diseases occur among Polish travelers so they should be taken into account in the context of prophylaxis when preparing for travel and in the diagnosis of diseases that occur after returning from a tropical zone. This first analysis of disease incidence among Polish travelers indicates a strong need for more research in this area.

A systematic review of biologically-informed deep learning models for cancer: fundamental trends for encoding and interpreting oncology data.

BACKGROUND: There is an increasing interest in the use of Deep Learning (DL) based methods as a supporting analytical framework in oncology. However, most direct applications of DL will deliver models with limited transparency and explainability, which constrain their deployment in biomedical settings. METHODS: This systematic review discusses DL models used to support inference in cancer biology with a particular emphasis on multi-omics analysis. It focuses on how existing models address the need for better dialogue with prior knowledge, biological plausibility and interpretability, fundamental properties in the biomedical domain. For this, we retrieved and analyzed 42 studies focusing on emerging architectural and methodological advances, the encoding of biological domain knowledge and the integration of explainability methods. RESULTS: We discuss the recent evolutionary arch of DL models in the direction of integrating prior biological relational and network knowledge to support better generalisation (e.g. pathways or Protein-Protein-Interaction networks) and interpretability. This represents a fundamental functional shift towards models which can integrate mechanistic and statistical inference aspects. We introduce a concept of bio-centric interpretability and according to its taxonomy, we discuss representational methodologies for the integration of domain prior knowledge in such models. CONCLUSIONS: The paper provides a critical outlook into contemporary methods for explainability and interpretability used in DL for cancer. The analysis points in the direction of a convergence between encoding prior knowledge and improved interpretability. We introduce bio-centric interpretability which is an important step towards formalisation of biological interpretability of DL models and developing methods that are less problem- or application-specific.

Meta-analysis informed machine learning: Supporting cytokine storm detection during CAR-T cell Therapy.

Cytokine release syndrome (CRS), also known as cytokine storm, is one of the most consequential adverse effects of chimeric antigen receptor therapies that have shown otherwise promising results in cancer treatment. When emerging, CRS could be identified by the analysis of specific cytokine and chemokine profiles that tend to exhibit similarities across patients. In this paper, we exploit these similarities using machine learning algorithms and set out to pioneer a meta-review informed method for the identification of CRS based on specific cytokine peak concentrations and evidence from previous clinical studies. To this end we also address a widespread challenge of the applicability of machine learning in general: reduced training data availability. We do so by augmenting available (but often insufficient) patient cytokine concentrations with statistical knowledge extracted from domain literature. We argue that such methods could support clinicians in analyzing suspect cytokine profiles by matching them against the said CRS knowledge from past clinical studies, with the ultimate aim of swift CRS diagnosis. We evaluate our proposed methods under several design choices, achieving performance of more than 90% in terms of CRS identification accuracy, and showing that many of our choices outperform a purely data-driven alternative. During evaluation with real-world CRS clinical data, we emphasize the potential of our proposed method of producing interpretable results, in addition to being effective in identifying the onset of cytokine storm.

An International Comparison of Presentation, Outcomes and CORONET Predictive Score Performance in Patients with Cancer Presenting with COVID-19 across Different Pandemic Waves.

Patients with cancer have been shown to have increased risk of COVID-19 severity. We previously built and validated the COVID-19 Risk in Oncology Evaluation Tool (CORONET) to predict the likely severity of COVID-19 in patients with active cancer who present to hospital. We assessed the differences in presentation and outcomes of patients with cancer and COVID-19, depending on the wave of the pandemic. We examined differences in features at presentation and outcomes in patients worldwide, depending on the waves of the pandemic: wave 1 D614G (n = 1430), wave 2 Alpha (n = 475), and wave 4 Omicron variant (n = 63, UK and Spain only). The performance of CORONET was evaluated on 258, 48, and 54 patients for each wave, respectively. We found that mortality rates were reduced in subsequent waves. The majority of patients were vaccinated in wave 4, and 94% were treated with steroids if they required oxygen. The stages of cancer and the median ages of patients significantly differed, but features associated with worse COVID-19 outcomes remained predictive and did not differ between waves. The CORONET tool performed well in all waves, with scores in an area under the curve (AUC) of >0.72. We concluded that patients with cancer who present to hospital with COVID-19 have similar features of severity, which remain discriminatory despite differences in variants and vaccination status. Survival improved following the first wave of the pandemic, which may be associated with vaccination and the increased steroid use in those patients requiring oxygen. The CORONET model demonstrated good performance, independent of the SARS-CoV-2 variants.

Establishment of CORONET, COVID-19 Risk in Oncology Evaluation Tool, to Identify Patients With Cancer at Low Versus High Risk of Severe Complications of COVID-19 Disease On Presentation to Hospital.

PURPOSE: Patients with cancer are at increased risk of severe COVID-19 disease, but have heterogeneous presentations and outcomes. Decision-making tools for hospital admission, severity prediction, and increased monitoring for early intervention are critical. We sought to identify features of COVID-19 disease in patients with cancer predicting severe disease and build a decision support online tool, COVID-19 Risk in Oncology Evaluation Tool (CORONET). METHODS: Patients with active cancer (stage I-IV) and laboratory-confirmed COVID-19 disease presenting to hospitals worldwide were included. Discharge (within 24 hours), admission (≥ 24 hours inpatient), oxygen (O2) requirement, and death were combined in a 0-3 point severity scale. Association of features with outcomes were investigated using Lasso regression and Random Forest combined with Shapley Additive Explanations. The CORONET model was then examined in the entire cohort to build an online CORONET decision support tool. Admission and severe disease thresholds were established through pragmatically defined cost functions. Finally, the CORONET model was validated on an external cohort. RESULTS: The model development data set comprised 920 patients, with median age 70 (range 5-99) years, 56% males, 44% females, and 81% solid versus 19% hematologic cancers. In derivation, Random Forest demonstrated superior performance over Lasso with lower mean squared error (0.801 v 0.807) and was selected for development. During validation (n = 282 patients), the performance of CORONET varied depending on the country cohort. CORONET cutoffs for admission and mortality of 1.0 and 2.3 were established. The CORONET decision support tool recommended admission for 95% of patients eventually requiring oxygen and 97% of those who died (94% and 98% in validation, respectively). The specificity for mortality prediction was 92% and 83% in derivation and validation, respectively. Shapley Additive Explanations revealed that National Early Warning Score 2, C-reactive protein, and albumin were the most important features contributing to COVID-19 severity prediction in patients with cancer at time of hospital presentation. CONCLUSION: CORONET, a decision support tool validated in health care systems worldwide, can aid admission decisions and predict COVID-19 severity in patients with cancer.

Biomarker identification using dynamic time warping analysis: a longitudinal cohort study of patients with COVID-19 in a UK tertiary hospital.

OBJECTIVES: COVID-19 is a heterogeneous disease, and many reports have described variations in demographic, biochemical and clinical features at presentation influencing overall hospital mortality. However, there is little information regarding longitudinal changes in laboratory prognostic variables in relation to disease progression in hospitalised patients with COVID-19. DESIGN AND SETTING: This retrospective observational report describes disease progression from symptom onset, to admission to hospital, clinical response and discharge/death among patients with COVID-19 at a tertiary centre in South East England. PARTICIPANTS: Six hundred and fifty-one patients treated for SARS-CoV-2 between March and September 2020 were included in this analysis. Ethical approval was obtained from the HRA Specific Review Board (REC 20/HRA/2986) for waiver of informed consent. RESULTS: The majority of patients presented within 1 week of symptom onset. The lowest risk patients had low mortality (1/45, 2%), and most were discharged within 1 week after admission (30/45, 67%). The highest risk patients, as determined by the 4C mortality score predictor, had high mortality (27/29, 93%), with most dying within 1 week after admission (22/29, 76%). Consistent with previous reports, most patients presented with high levels of C reactive protein (CRP) (67% of patients >50 mg/L), D-dimer (98%>upper limit of normal (ULN)), ferritin (65%>ULN), lactate dehydrogenase (90%>ULN) and low lymphocyte counts (81%

Wave comparisons of clinical characteristics and outcomes of COVID-19 admissions - Exploring the impact of treatment and strain dynamics.

OBJECTIVES: Dexamethasone has now been incorporated into the standard of care for COVID-19 hospital patients. However, larger intensive care unit studies have failed to show discernible improvements in mortality in the recent wave. We aimed to investigate the impacts of these factors on disease outcomes in a UK hospital study. METHODS: This retrospective observational study reports patient characteristics, interventions and outcomes in COVID-19 patients from a UK teaching hospital; cohort 1, pre 16th June-2020 (pre-dexamethasone); cohort 2, 17th June to 30th November-2020 (post-dexamethasone, pre-VOC 202,012/01 as dominant strain); cohort 3, 1st December-2020 to 3rd March-2021 (during establishment of VOC202012/01 as the dominant strain). RESULTS: Dexamethasone treatment was more common in cohorts 2 and 3 (42.7% and 51.6%) compared with cohort 1 (2.5%). After adjusting for risk, odds of death within 28 days were 2-fold lower in cohort 2 vs 1 (OR:0.47,[0.27,0.79],p = 0.006). Mortality was higher cohort 3 vs 2 (20% vs 14%); but not significantly different to cohort 1 (OR: 0.86,[0.64, 1.15],p = 0.308). CONCLUSIONS: The real world finding of lower mortality following dexamethasone supports the published trial evidence and highlights ongoing need for research with introduction of new treatments and ongoing concern over new COVID-19 variants.