Ensuring accuracy in the development and application of nucleic acid amplification tests (NAATs) for infectious disease.

Diagnostic tests were heralded as crucial during the Coronavirus disease (COVID-19) pandemic with most of the key methods using bioanalytical approaches that detected larger molecules (RNA, protein antigens or antibodies) rather than conventional clinical biochemical techniques. Nucleic Acid Amplification Tests (NAATs), like the Polymerase Chain Reaction (PCR), and other molecular methods, like sequencing (that often work in combination with NAATs), were essential to the diagnosis and management during COVID-19. This was exemplified both early in the pandemic but also later on, following the emergence of new genetic SARS-CoV-2 variants. The 100 day mission to respond to future pandemic threats highlights the need for effective diagnostics, therapeutics and vaccines. Of the three, diagnostics represents the first opportunity to manage infectious diseases while also being the most poorly supported in terms of the infrastructure needed to demonstrate effectiveness. Where performance targets exist, they are not well served by consensus on how to demonstrate they are being met; this includes analytical factors such as limit of detection (LOD) false positive results as well as how to approach clinical evaluation. The selection of gold standards or use of epidemiological factors such as predictive value, reference ranges or clinical thresholds are seldom correctly considered. The attention placed on molecular diagnostic tests during COVID-19 illustrates important considerations and assumptions on the use of these methods for infectious disease diagnosis and beyond. In this manuscript, we discuss state-of-the-art approaches to diagnostic evaluation and explore how they may be better tailored to diagnostic techniques like NAATs to maximise the impact of these highly versatile bioanalytical tools, both generally and during future outbreaks.

Examining the effect of evaluation sample size on the sensitivity and specificity of COVID-19 diagnostic tests in practice: a simulation study.

BACKGROUND: In response to the global COVID-19 pandemic, many in vitro diagnostic (IVD) tests for SARS-CoV-2 have been developed. Given the urgent clinical demand, researchers must balance the desire for precise estimates of sensitivity and specificity against the need for rapid implementation. To complement estimates of precision used for sample size calculations, we aimed to estimate the probability that an IVD will fail to perform to expected standards after implementation, following clinical studies with varying sample sizes. METHODS: We assumed that clinical validation study estimates met the 'desirable' performance (sensitivity 97%, specificity 99%) in the target product profile (TPP) published by the Medicines and Healthcare products Regulatory Agency (MHRA). To estimate the real-world impact of imprecision imposed by sample size we used Bayesian posterior calculations along with Monte Carlo simulations with 10,000 independent iterations of 5,000 participants. We varied the prevalence between 1 and 15% and the sample size between 30 and 2,000. For each sample size, we estimated the probability that diagnostic accuracy would fail to meet the TPP criteria after implementation. RESULTS: For a validation study that demonstrates 'desirable' sensitivity within a sample of 30 participants who test positive for COVID-19 using the reference standard, the probability that real-world performance will fail to meet the 'desirable' criteria is 10.7-13.5%, depending on prevalence. Theoretically, demonstrating the 'desirable' performance in 90 positive participants would reduce that probability to below 5%. A marked reduction in the probability of failure to hit 'desirable' specificity occurred between samples of 100 (19.1-21.5%) and 160 (4.3-4.8%) negative participants. There was little further improvement above sample sizes of 160 negative participants. CONCLUSION: Based on imprecision alone, small evaluation studies can lead to the acceptance of diagnostic tests which are likely to fail to meet performance targets when deployed. There is diminished return on uncertainty surrounding an accuracy estimate above a total sample size of 250 (90 positive and 160 negative).

Assessment of Automated Flow Cytometry Data Analysis Tools within Cell and Gene Therapy Manufacturing.

Flow cytometry is widely used within the manufacturing of cell and gene therapies to measure and characterise cells. Conventional manual data analysis relies heavily on operator judgement, presenting a major source of variation that can adversely impact the quality and predictive potential of therapies given to patients. Computational tools have the capacity to minimise operator variation and bias in flow cytometry data analysis; however, in many cases, confidence in these technologies has yet to be fully established mirrored by aspects of regulatory concern. Here, we employed synthetic flow cytometry datasets containing controlled population characteristics of separation, and normal/skew distributions to investigate the accuracy and reproducibility of six cell population identification tools, each of which implement different unsupervised clustering algorithms: Flock2, flowMeans, FlowSOM, PhenoGraph, SPADE3 and SWIFT (density-based, k-means, self-organising map, k-nearest neighbour, deterministic k-means, and model-based clustering, respectively). We found that outputs from software analysing the same reference synthetic dataset vary considerably and accuracy deteriorates as the cluster separation index falls below zero. Consequently, as clusters begin to merge, the flowMeans and Flock2 software platforms struggle to identify target clusters more than other platforms. Moreover, the presence of skewed cell populations resulted in poor performance from SWIFT, though FlowSOM, PhenoGraph and SPADE3 were relatively unaffected in comparison. These findings illustrate how novel flow cytometry synthetic datasets can be utilised to validate a range of automated cell identification methods, leading to enhanced confidence in the data quality of automated cell characterisations and enumerations.

Systematic Design, Generation, and Application of Synthetic Datasets for Flow Cytometry.

Application of synthetic datasets in training and validation of analysis tools has led to improvements in many decision-making tasks in a range of domains from computer vision to digital pathology. Synthetic datasets overcome the constraints of real-world datasets, namely difficulties in collection and labeling, expense, time, and privacy concerns. In flow cytometry, real cell-based datasets are limited by properties such as size, number of parameters, distance between cell populations, and distributions and are often focused on a narrow range of disease or cell types. Researchers in some cases have designed these desired properties into synthetic datasets; however, operators have implemented them in inconsistent approaches, and there is a scarcity of publicly available, high-quality synthetic datasets. In this research, we propose a method to systematically design and generate flow cytometry synthetic datasets with highly controlled characteristics. We demonstrate the generation of two-cluster synthetic datasets with specific degrees of separation between cell populations, and of non-normal distributions with increasing levels of skewness and orientations of skew pairs. We apply our synthetic datasets to test the performance of a popular automated cell populations identification software, SPADE3, and define the region where the software performance decreases as the clusters get closer together. Application of the synthetic skewed dataset suggests the software is capable of processing non-normal data. We calculate the classification accuracy of SPADE3 with robustness not achievable with real-world datasets. Our approach aims to advance research toward generation of high-quality synthetic flow cytometry datasets and to increase their awareness among the community. The synthetic datasets can be used in benchmarking studies that critically evaluate cell population identification tools and help illustrate potential digital platform inconsistencies. These datasets have the potential to improve cell characterization workflows that integrate automated analysis in clinical diagnostics and cell therapy manufacturing.

The Dangers of Using Cq to Quantify Nucleic Acid in Biological Samples: A Lesson From COVID-19.

BACKGROUND: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) RNA quantities, measured by reverse transcription quantitative PCR (RT-qPCR), have been proposed to stratify clinical risk or determine analytical performance targets. We investigated reproducibility and how setting diagnostic cutoffs altered the clinical sensitivity of coronavirus disease 2019 (COVID-19) testing. METHODS: Quantitative SARS-CoV-2 RNA distributions [quantification cycle (Cq) and copies/mL] from more than 6000 patients from 3 clinical laboratories in United Kingdom, Belgium, and the Republic of Korea were analyzed. Impact of Cq cutoffs on clinical sensitivity was assessed. The June/July 2020 INSTAND external quality assessment scheme SARS-CoV-2 materials were used to estimate laboratory reported copies/mL and to estimate the variation in copies/mL for a given Cq. RESULTS: When the WHO-suggested Cq cutoff of 25 was applied, the clinical sensitivity dropped to about 16%. Clinical sensitivity also dropped to about 27% when a simulated limit of detection of 106 copies/mL was applied. The interlaboratory variation for a given Cq value was >1000 fold in copies/mL (99% CI). CONCLUSION: While RT-qPCR has been instrumental in the response to COVID-19, we recommend Cq (cycle threshold or crossing point) values not be used to set clinical cutoffs or diagnostic performance targets due to poor interlaboratory reproducibility; calibrated copy-based units (used elsewhere in virology) offer more reproducible alternatives. We also report a phenomenon where diagnostic performance may change relative to the effective reproduction number. Our findings indicate that the disparities between patient populations across time are an important consideration when evaluating or deploying diagnostic tests. This is especially relevant to the emergency situation of an evolving pandemic.

Assessment of Protocol Impact on Subjectivity Uncertainty When Analyzing Peripheral Blood Mononuclear Cell Flow Cytometry Data Files.

Measured variability of product within Cell and Gene Therapy (CGT) manufacturing arises from numerous sources across pre-analytical to post-analytical phases of testing. Operators are a function of the manufacturing process and are an important source of variability as a result of personal differences impacted by numerous factors. This research uses measurement uncertainty in comparison to Coefficient of Variation to quantify variation of participants when they complete Flow Cytometry data analysis through a 5-step gating sequence. Two study stages captured participants applying gates using their own judgement, and then following a diagrammatical protocol, respectively. Measurement uncertainty was quantified for each participant (and analysis phase) by following Guide to the Expression of Uncertainty in Measurement protocols, combining their standard deviations in quadrature from each gating step in the respective protocols. When participants followed a diagrammatical protocol, variation between participants reduced by 57%, increasing confidence in a more uniform reported cell count percentage. Measurement uncertainty provided greater resolution to the analysis processes, identifying that most variability contributed in the Flow Cytometry gating process is from the very first gate, where isolating target cells from dead or dying cells is required. This work has demonstrated the potential for greater usage of measurement uncertainty within CGT manufacturing scenarios, due to the resolution it provides for root cause analysis and continuous improvement.

Current trends in flow cytometry automated data analysis software.

Automated flow cytometry (FC) data analysis tools for cell population identification and characterization are increasingly being used in academic, biotechnology, pharmaceutical, and clinical laboratories. The development of these computational methods is designed to overcome reproducibility and process bottleneck issues in manual gating, however, the take-up of these tools remains (anecdotally) low. Here, we performed a comprehensive literature survey of state-of-the-art computational tools typically published by research, clinical, and biomanufacturing laboratories for automated FC data analysis and identified popular tools based on literature citation counts. Dimensionality reduction methods ranked highly, such as generic t-distributed stochastic neighbor embedding (t-SNE) and its initial Matlab-based implementation for cytometry data viSNE. Software with graphical user interfaces also ranked highly, including PhenoGraph, SPADE1, FlowSOM, and Citrus, with unsupervised learning methods outnumbering supervised learning methods, and algorithm type popularity spread across K-Means, hierarchical, density-based, model-based, and other classes of clustering algorithms. Additionally, to illustrate the actual use typically within clinical spaces alongside frequent citations, a survey issued by UK NEQAS Leucocyte Immunophenotyping to identify software usage trends among clinical laboratories was completed. The survey revealed 53% of laboratories have not yet taken up automated cell population identification methods, though among those that have, Infinicyt software is the most frequently identified. Survey respondents considered data output quality to be the most important factor when using automated FC data analysis software, followed by software speed and level of technical support. This review found differences in software usage between biomedical institutions, with tools for discovery, data exploration, and visualization more popular in academia, whereas automated tools for specialized targeted analysis that apply supervised learning methods were more used in clinical settings.

Quantifying Operator Subjectivity within Flow Cytometry Data Analysis as a Source of Measurement Uncertainty and the Impact of Experience on Results.

Flow cytometry is a complex measurement characterization technique, utilized within the manufacture, measurement, and release of cell and gene therapy products for rapid, high-content, and multiplexed discriminatory cell analysis. A number of factors influence the variability in the measurement reported including, but not limited to, biological variation, reagent variation, laser and optical configurations, and data analysis methods. This research focused on understanding the contribution of manual operator variability within the data analysis phase. Thirty-eight participants completed a questionnaire, providing information about experience and motivational factors, before completing a simple gating study. The results were analyzed using gauge repeatability and reproducibility techniques to quantify participant uncertainty. The various stages of the gating sequence were combined through summation in quadrature and expanded to give each participant a representative uncertainty value. Of the participants surveyed, 85% preferred manual gating to automated data analysis, with the primary reasons being legacy ("it's always been done that way") and accuracy, not in the metrological sense but in the clear definition of the correct target population. The median expanded uncertainty was calculated as 3.6% for the population studied, with no significant difference among more or less experienced users. Operator subjectivity can be quantified to include within measurement uncertainty budgets, required for various standards and qualifications. An emphasis on biomanufacturing measurement terminology is needed to help understand future and potential solutions, possibly looking at translational clinical models to engage and enhance better training and protocols within industrial and research settings.

Science-based assessment of source materials for cell-based medicines: report of a stakeholders workshop.

Human pluripotent stem cells (hPSCs) have the potential to transform medicine. However, hurdles remain to ensure safety for such cellular products. Science-based understanding of the requirements for source materials is required as are appropriate materials. Leaders in hPSC biology, clinical translation, biomanufacturing and regulatory issues were brought together to define requirements for source materials for the production of hPSC-derived therapies and to identify other key issues for the safety of cell therapy products. While the focus of this meeting was on hPSC-derived cell therapies, many of the issues are generic to all cell-based medicines. The intent of this report is to summarize the key issues discussed and record the consensus reached on each of these by the expert delegates.

Fractionation of cadmium in tobacco and cigarette smoke condensate using XANES and sequential leaching with ICP-MS/MS.

Fractionation data for cadmium in tobacco products, as obtained by sequential leaching of cadmium species with ICP-MS/MS analysis, and separately by X-ray absorption near edge structure (XANES) are presented here for the first time. The total amount of cadmium found in 3R4F cigarette cut tobacco was 1526 ± 42 µg kg-1, of which 5% was found in the smoke under ISO smoking conditions. XANES analysis showed that Cd in tobacco, cigarette smoke and ash was present in the + 2 oxidation state. Examination of the gas-particle partitioning of smoke cadmium suggests that Cd in mainstream smoke is best viewed as semi-volatile, existing in both particulate and gas phases. Sequential extraction of trapped tobacco smoke was carried out to get a deeper insight into the chemistry of cigarette smoke cadmium compounds. Consecutive extractions with ultrapure water, dilute (1%) nitric acid and 10% nitric acid led to extraction of a total amount of Cd which agreed with that obtained after microwave digestion of the whole sample, suggesting that cadmium was quantitatively leachable into aqueous/acidic solutions. Most Cd (~ 90% of the total Cd in the smoke condensate) was extracted into dilute nitric acid (likely as CdO, Cd(OH)2 and CdCO3) with a minor percentage (3%) extracted into water (likely as CdCl2) and in 10% nitric acid (likely as CdS). Extraction of trapped mainstream smoke with pentane, followed by ICP-MS/MS analysis, to examine the possible presence of organocadmium in 3R4F tobacco smoke, did not show the presence of organocadmium compounds above the method LOQ (2 µg kg-1), possibly due to their reactivity under the experimental conditions. The high selectivity with sufficient sensitivity achieved by ICP-MS/MS was invaluable to quantify Cd (at low µg kg-1levels) simultaneously with sulphur and chlorine in the tobacco smoke fractions of complex matrix. The cadmium chemistry in the smoke, identified in this study, is consistent with both relatively high lung absorption and DNA binding; both potentially important factors for disease progression in smokers. Graphical Abstract This paper provides quantitative fractionation data for cadmium in tobacco and smoke by using sequential leaching with ICPMS and XANES.

Comparability: manufacturing, characterization and controls, report of a UK Regenerative Medicine Platform Pluripotent Stem Cell Platform Workshop, Trinity Hall, Cambridge, 14-15 September 2015.

This paper summarizes the proceedings of a workshop held at Trinity Hall, Cambridge to discuss comparability and includes additional information and references to related information added subsequently to the workshop. Comparability is the need to demonstrate equivalence of product after a process change; a recent publication states that this 'may be difficult for cell-based medicinal products'. Therefore a well-managed change process is required which needs access to good science and regulatory advice and developers are encouraged to seek help early. The workshop shared current thinking and best practice and allowed the definition of key research questions. The intent of this report is to summarize the key issues and the consensus reached on each of these by the expert delegates.

LGC: clinical about clinical measurement.

LGC is the UK's designated National Measurement Institute for chemical and bioanalytical measurement, and through this role improves the quality and international acceptance of measurements performed within the UK. This research spotlight, highlighting measurement 'across the scale', from elemental analysis and small molecules, through to proteins, DNA and RNA and on to whole cells and tissues, demonstrates how LGC is supporting the clinical sector by ensuring sound measurement practice that underpins clinical efficacy, quality assurance and patient safety.