Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19.

From early in the coronavirus disease 2019 (COVID-19) pandemic, there was interest in using machine learning methods to predict COVID-19 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing-RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS-CoV-2 infection status and extensive study participant meta-data. This allowed us to rigorously assess state-of-the-art machine learning techniques to predict SARS-CoV-2 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.

Interoperability of statistical models in pandemic preparedness: principles and reality.

We present interoperability as a guiding framework for statistical modelling to assist policy makers asking multiple questions using diverse datasets in the face of an evolving pandemic response. Interoperability provides an important set of principles for future pandemic preparedness, through the joint design and deployment of adaptable systems of statistical models for disease surveillance using probabilistic reasoning. We illustrate this through case studies for inferring and characterising spatial-temporal prevalence and reproduction numbers of SARS-CoV-2 infections in England.

A large-scale and PCR-referenced vocal audio dataset for COVID-19.

The UK COVID-19 Vocal Audio Dataset is designed for the training and evaluation of machine learning models that classify SARS-CoV-2 infection status or associated respiratory symptoms using vocal audio. The UK Health Security Agency recruited voluntary participants through the national Test and Trace programme and the REACT-1 survey in England from March 2021 to March 2022, during dominant transmission of the Alpha and Delta SARS-CoV-2 variants and some Omicron variant sublineages. Audio recordings of volitional coughs, exhalations, and speech were collected in the 'Speak up and help beat coronavirus' digital survey alongside demographic, symptom and self-reported respiratory condition data. Digital survey submissions were linked to SARS-CoV-2 test results. The UK COVID-19 Vocal Audio Dataset represents the largest collection of SARS-CoV-2 PCR-referenced audio recordings to date. PCR results were linked to 70,565 of 72,999 participants and 24,105 of 25,706 positive cases. Respiratory symptoms were reported by 45.6% of participants. This dataset has additional potential uses for bioacoustics research, with 11.3% participants self-reporting asthma, and 27.2% with linked influenza PCR test results.

Wastewater-based surveillance models for COVID-19: A focused review on spatio-temporal models.

The evident shedding of the SARS-CoV-2 RNA particles from infected individuals into the wastewater opened up a tantalizing array of possibilities for prediction of COVID-19 prevalence prior to symptomatic case identification through community testing. Many countries have therefore explored the use of wastewater metrics as a surveillance tool, replacing traditional direct measurement of prevalence with cost-effective approaches based on SARS-CoV-2 RNA concentrations in wastewater samples. Two important aspects in building prediction models are: time over which the prediction occurs and space for which the predicted case numbers is shown. In this review, our main focus was on finding mathematical models which take into the account both the time-varying and spatial nature of wastewater-based metrics into account. We used six main characteristics as our assessment criteria: i) modelling approach; ii) temporal coverage; iii) spatial coverage; iv) sample size; v) wastewater sampling method; and vi) covariates included in the modelling. The majority of studies in the early phases of the pandemic recognized the temporal association of SARS-CoV-2 RNA concentration level in wastewater with the number of COVID-19 cases, ignoring their spatial context. We examined 15 studies up to April 2023, focusing on models considering both temporal and spatial aspects of wastewater metrics. Most early studies correlated temporal SARS-CoV-2 RNA levels with COVID-19 cases but overlooked spatial factors. Linear regression and SEIR models were commonly used (n = 10, 66.6 % of studies), along with machine learning (n = 1, 6.6 %) and Bayesian approaches (n = 1, 6.6 %) in some cases. Three studies employed spatio-temporal modelling approach (n = 3, 20.0 %). We conclude that the development, validation and calibration of further spatio-temporally explicit models should be done in parallel with the advancement of wastewater metrics before the potential of wastewater as a surveillance tool can be fully realised.

A spatio-temporal framework for modelling wastewater concentration during the COVID-19 pandemic.

The potential utility of wastewater-based epidemiology as an early warning tool has been explored widely across the globe during the current COVID-19 pandemic. Methods to detect the presence of SARS-CoV-2 RNA in wastewater were developed early in the pandemic, and extensive work has been conducted to evaluate the relationship between viral concentration and COVID-19 case numbers at the catchment areas of sewage treatment works (STWs) over time. However, no attempt has been made to develop a model that predicts wastewater concentration at fine spatio-temporal resolutions covering an entire country, a necessary step towards using wastewater monitoring for the early detection of local outbreaks. We consider weekly averages of flow-normalised viral concentration, reported as the number of SARS-CoV-2N1 gene copies per litre (gc/L) of wastewater available at 303 STWs over the period between 1 June 2021 and 30 March 2022. We specify a spatially continuous statistical model that quantifies the relationship between weekly viral concentration and a collection of covariates covering socio-demographics, land cover and virus associated genomic characteristics at STW catchment areas while accounting for spatial and temporal correlation. We evaluate the model's predictive performance at the catchment level through 10-fold cross-validation. We predict the weekly viral concentration at the population-weighted centroid of the 32,844 lower super output areas (LSOAs) in England, then aggregate these LSOA predictions to the Lower Tier Local Authority level (LTLA), a geography that is more relevant to public health policy-making. We also use the model outputs to quantify the probability of local changes of direction (increases or decreases) in viral concentration over short periods (e.g. two consecutive weeks). The proposed statistical framework can predict SARS-CoV-2 viral concentration in wastewater at high spatio-temporal resolution across England. Additionally, the probabilistic quantification of local changes can be used as an early warning tool for public health surveillance.

Bayesian imputation of COVID-19 positive test counts for nowcasting under reporting lag.

Obtaining up to date information on the number of UK COVID-19 regional infections is hampered by the reporting lag in positive test results for people with COVID-19 symptoms. In the UK, for 'Pillar 2' swab tests for those showing symptoms, it can take up to five days for results to be collated. We make use of the stability of the under reporting process over time to motivate a statistical temporal model that infers the final total count given the partial count information as it arrives. We adopt a Bayesian approach that provides for subjective priors on parameters and a hierarchical structure for an underlying latent intensity process for the infection counts. This results in a smoothed time-series representation nowcasting the expected number of daily counts of positive tests with uncertainty bands that can be used to aid decision making. Inference is performed using sequential Monte Carlo.

Improving local prevalence estimates of SARS-CoV-2 infections using a causal debiasing framework.

Global and national surveillance of SARS-CoV-2 epidemiology is mostly based on targeted schemes focused on testing individuals with symptoms. These tested groups are often unrepresentative of the wider population and exhibit test positivity rates that are biased upwards compared with the true population prevalence. Such data are routinely used to infer infection prevalence and the effective reproduction number, Rt, which affects public health policy. Here, we describe a causal framework that provides debiased fine-scale spatiotemporal estimates by combining targeted test counts with data from a randomized surveillance study in the United Kingdom called REACT. Our probabilistic model includes a bias parameter that captures the increased probability of an infected individual being tested, relative to a non-infected individual, and transforms observed test counts to debiased estimates of the true underlying local prevalence and Rt. We validated our approach on held-out REACT data over a 7-month period. Furthermore, our local estimates of Rt are indicative of 1-week- and 2-week-ahead changes in SARS-CoV-2-positive case numbers. We also observed increases in estimated local prevalence and Rt that reflect the spread of the Alpha and Delta variants. Our results illustrate how randomized surveys can augment targeted testing to improve statistical accuracy in monitoring the spread of emerging and ongoing infectious disease.

Time varying association between deprivation, ethnicity and SARS-CoV-2 infections in England: A population-based ecological study.

BACKGROUND: Ethnically diverse and socio-economically deprived communities have been differentially affected by the COVID-19 pandemic in the UK. METHOD: Using a multilevel regression model we assessed the time-varying association between SARS-CoV-2 infections and areal level deprivation and ethnicity from 1st of June 2020 to the 19th of September 2021. We separately considered weekly test positivity rate and estimated debiased prevalence at the Lower Tier Local Authority (LTLA) level, adjusting for confounders and spatio-temporal correlation structure. FINDINGS: Comparing the least deprived and predominantly White areas with most deprived and predominantly non-White areas over the whole study period, the weekly positivity rate increases from 2·977% (95% CrI 2.913%-3.029%) to 3·347% (95% CrI 3.300%-3.402%). Similarly, prevalence increases from 0·369% (95% CrI 0.361%-0.375%) to 0·405% (95% CrI 0.399%-0.412%). Deprivation has a stronger effect until October 2020, while the effect of ethnicity becomes more pronounced at the peak of the second wave and then again in May-June 2021. In the second wave of the pandemic, LTLAs with large South Asian populations were the most affected, whereas areas with large Black populations did not show increased values for either outcome during the entire period under analysis. INTERPRETATION: Deprivation and proportion of non-White populations are both associated with an increased COVID-19 burden in terms of disease spread and monitoring, but the strength of association varies over the course of the pandemic and for different ethnic subgroups. The consistency of results across the two outcomes suggests that deprivation and ethnicity have a differential impact on disease exposure or susceptibility rather than testing access and habits. FUNDINGS: EPSRC, MRC, The Alan Turing Institute, NIH, UKHSA, DHSC.

Time varying association between deprivation, ethnicity and SARS-CoV-2 infections in England: a space-time study.

BACKGROUND: Ethnically diverse and socio-economically deprived communities have been differentially affected by the COVID-19 pandemic in the UK. METHOD: Using a multilevel regression model we assess the time-varying association between SARS-CoV-2 infections and areal level deprivation and ethnicity. We separately consider weekly test positivity rate (number of positive tests over the total number of tests) and estimated unbiased prevalence (proportion of individuals in the population who would test positive) at the Lower Tier Local Authority (LTLA) level. The model also adjusts for age, urbanicity, vaccine uptake and spatio-temporal correlation structure. FINDINGS: Comparing the least deprived and predominantly White areas with most deprived and predominantly non-White areas over the whole study period, the weekly positivity rate increases by 13% from 297% to 335%. Similarly, prevalence increases by 10% from 037% to 041%. Deprivation has a stronger effect until October 2020, while the effect of ethnicity becomes slightly more pronounced at the peak of the second wave and then again in May-June 2021. Not all BAME groups were equally affected: in the second wave of the pandemic, LTLAs with large South Asian populations were the most affected, whereas areas with large Black populations did not show increased values for either outcome during the entire period under analysis. INTERPRETATION: At the area level, IMD and BAME% are both associated with an increased COVID-19 burden in terms of prevalence (disease spread) and test positivity (disease monitoring), and the strength of association varies over the course of the pandemic. The consistency of results across the two outcome measures suggests that community level characteristics such as deprivation and ethnicity have a differential impact on disease exposure or susceptibility rather than testing access and habits. FUNDINGS: EPSRC, MRC, The Alan Turing Institute, NIH, UKHSA, DHSC, NIHR.

Metacognitive judgements of perceptual-motor steering performance.

Control of skilled actions requires rapid information sampling and processing, which may largely be carried out subconsciously. However, individuals often need to make conscious strategic decisions that ideally would be based upon accurate knowledge of performance. Here, we determined the extent to which individuals have explicit awareness of their steering performance (conceptualised as "metacognition"). Participants steered in a virtual environment along a bending road while attempting to keep within a central demarcated target zone. Task demands were altered by manipulating locomotor speed (fast/slow) and the target zone (narrow/wide). All participants received continuous visual feedback about position in zone, and one sub-group was given additional auditory warnings when exiting/entering the zone. At the end of each trial, participants made a metacognitive evaluation: the proportion of the trial they believed was spent in the zone. Overall, although evaluations broadly shifted in line with task demands, participants showed limited calibration to performance. Regression analysis showed that evaluations were influenced by two components: (a) direct monitoring of performance and (b) indirect task heuristics estimating performance based on salient cues (e.g., speed). Evaluations often weighted indirect task heuristics inappropriately, but the additional auditory feedback improved evaluations seemingly by reducing this weighting. These results have important implications for all motor tasks where conscious cognitive control can be used to influence action selection.

Investigating the Role of Assessment Method on Reports of Déjà Vu and Tip-of-the-Tongue States during Standard Recognition Tests.

Déjà vu and tip-of-the-tongue (TOT) are retrieval-related subjective experiences whose study relies on participant self-report. In four experiments (ns = 224, 273, 123 and 154), we explored the effect of questioning method on reported occurrence of déjà vu and TOT in experimental settings. All participants carried out a continuous recognition task, which was not expected to induce déjà vu or TOT, but were asked about their experiences of these subjective states. When presented with contemporary definitions, between 32% and 58% of participants nonetheless reported experiencing déjà vu or TOT. Changing the definition of déjà vu or asking participants to bring to mind a real-life instance of déjà vu or TOT before completing the recognition task had no impact on reporting rates. However, there was an indication that changing the method of requesting subjective reports impacted reporting of both experiences. More specifically, moving from the commonly used retrospective questioning (e.g. "Have you experienced déjà vu?") to free report instructions (e.g. "Indicate whenever you experience déjà vu.") reduced the total number of reported déjà vu and TOT occurrences. We suggest that research on subjective experiences should move toward free report assessments. Such a shift would potentially reduce the presence of false alarms in experimental work, thereby reducing the overestimation of subjective experiences prevalent in this area of research.

Negative affect does not impact semantic retrieval failure monitoring.

This study investigated the effect of the emotional nature of to-be-retrieved material on semantic retrieval monitoring. Across 2 groups, participants were either asked whether they have experienced a tip-of-the-tongue (TOT) state or to make a feeling-of-knowing (FOK) judgment. We examined the overall reporting rate as well as subjective (not accompanied by partial information recall) TOT and FOK reporting, comparing whether these differed between emotional (negatively valenced and arousing) and neutral items. The results demonstrated that emotion does not impact semantic TOT and FOK reports, a conclusion supported by Bayesian analysis of the results. The outcomes extend other findings in the metamemory literature, and are discussed with a focus on future research avenues concerning interactions between emotion and metamemory.