Empirical methods for the validation of time-to-event mathematical models taking into account uncertainty and variability: application to EGFR + lung adenocarcinoma.

BACKGROUND: Over the past several decades, metrics have been defined to assess the quality of various types of models and to compare their performance depending on their capacity to explain the variance found in real-life data. However, available validation methods are mostly designed for statistical regressions rather than for mechanistic models. To our knowledge, in the latter case, there are no consensus standards, for instance for the validation of predictions against real-world data given the variability and uncertainty of the data. In this work, we focus on the prediction of time-to-event curves using as an application example a mechanistic model of non-small cell lung cancer. We designed four empirical methods to assess both model performance and reliability of predictions: two methods based on bootstrapped versions of parametric statistical tests: log-rank and combined weighted log-ranks (MaxCombo); and two methods based on bootstrapped prediction intervals, referred to here as raw coverage and the juncture metric. We also introduced the notion of observation time uncertainty to take into consideration the real life delay between the moment when an event happens, and the moment when it is observed and reported. RESULTS: We highlight the advantages and disadvantages of these methods according to their application context. We have shown that the context of use of the model has an impact on the model validation process. Thanks to the use of several validation metrics we have highlighted the limit of the model to predict the evolution of the disease in the whole population of mutations at the same time, and that it was more efficient with specific predictions in the target mutation populations. The choice and use of a single metric could have led to an erroneous validation of the model and its context of use. CONCLUSIONS: With this work, we stress the importance of making judicious choices for a metric, and how using a combination of metrics could be more relevant, with the objective of validating a given model and its predictions within a specific context of use. We also show how the reliability of the results depends both on the metric and on the statistical comparisons, and that the conditions of application and the type of available information need to be taken into account to choose the best validation strategy.

Knowledge-based mechanistic modeling accurately predicts disease progression with gefitinib in EGFR-mutant lung adenocarcinoma.

Lung adenocarcinoma (LUAD) is associated with a low survival rate at advanced stages. Although the development of targeted therapies has improved outcomes in LUAD patients with identified and specific genetic alterations, such as activating mutations on the epidermal growth factor receptor gene (EGFR), the emergence of tumor resistance eventually occurs in all patients and this is driving the development of new therapies. In this paper, we present the In Silico EGFR-mutant LUAD (ISELA) model that links LUAD patients' individual characteristics, including tumor genetic heterogeneity, to tumor size evolution and tumor progression over time under first generation EGFR tyrosine kinase inhibitor gefitinib. This translational mechanistic model gathers extensive knowledge on LUAD and was calibrated on multiple scales, including in vitro, human tumor xenograft mouse and human, reproducing more than 90% of the experimental data identified. Moreover, with 98.5% coverage and 99.4% negative logrank tests, the model accurately reproduced the time to progression from the Lux-Lung 7 clinical trial, which was unused in calibration, thus supporting the model high predictive value. This knowledge-based mechanistic model could be a valuable tool in the development of new therapies targeting EGFR-mutant LUAD as a foundation for the generation of synthetic control arms.

Modeling the disruption of respiratory disease clinical trials by non-pharmaceutical COVID-19 interventions.

Respiratory disease trials are profoundly affected by non-pharmaceutical interventions (NPIs) against COVID-19 because they perturb existing regular patterns of all seasonal viral epidemics. To address trial design with such uncertainty, we developed an epidemiological model of respiratory tract infection (RTI) coupled to a mechanistic description of viral RTI episodes. We explored the impact of reduced viral transmission (mimicking NPIs) using a virtual population and in silico trials for the bacterial lysate OM-85 as prophylaxis for RTI. Ratio-based efficacy metrics are only impacted under strict lockdown whereas absolute benefit already is with intermediate NPIs (eg. mask-wearing). Consequently, despite NPI, trials may meet their relative efficacy endpoints (provided recruitment hurdles can be overcome) but are difficult to assess with respect to clinical relevance. These results advocate to report a variety of metrics for benefit assessment, to use adaptive trial design and adapted statistical analyses. They also question eligibility criteria misaligned with the actual disease burden.

Solving the Evidence Interpretability Crisis in Health Technology Assessment: A Role for Mechanistic Models?

Health technology assessment (HTA) aims to be a systematic, transparent, unbiased synthesis of clinical efficacy, safety, and value of medical products (MPs) to help policymakers, payers, clinicians, and industry to make informed decisions. The evidence available for HTA has gaps-impeding timely prediction of the individual long-term effect in real clinical practice. Also, appraisal of an MP needs cross-stakeholder communication and engagement. Both aspects may benefit from extended use of modeling and simulation. Modeling is used in HTA for data-synthesis and health-economic projections. In parallel, regulatory consideration of model informed drug development (MIDD) has brought attention to mechanistic modeling techniques that could in fact be relevant for HTA. The ability to extrapolate and generate personalized predictions renders the mechanistic MIDD approaches suitable to support translation between clinical trial data into real-world evidence. In this perspective, we therefore discuss concrete examples of how mechanistic models could address HTA-related questions. We shed light on different stakeholder's contributions and needs in the appraisal phase and suggest how mechanistic modeling strategies and reporting can contribute to this effort. There are still barriers dissecting the HTA space and the clinical development space with regard to modeling: lack of an adapted model validation framework for decision-making process, inconsistent and unclear support by stakeholders, limited generalizable use cases, and absence of appropriate incentives. To address this challenge, we suggest to intensify the collaboration between competent authorities, drug developers and modelers with the aim to implement mechanistic models central in the evidence generation, synthesis, and appraisal of HTA so that the totality of mechanistic and clinical evidence can be leveraged by all relevant stakeholders.

Comparison of an effect-model-law-based method versus traditional clinical practice guidelines for optimal treatment decision-making: application to statin treatment in the French population.

Healthcare authorities make difficult decisions about how to spend limited budgets for interventions that guarantee the best cost-efficacy ratio. We propose a novel approach for treatment decision-making, OMES-in French: Objectif thérapeutique Modèle Effet Seuil (in English: Therapeutic Objective-Threshold-Effect Model; TOTEM). This approach takes into consideration results from clinical trials, adjusted for the patients' characteristics in treatment decision-making. We compared OMES with the French clinical practice guidelines (CPGs) for the management of dyslipidemia with statin in a computer-generated realistic virtual population, representing the adult French population, in terms of the number of all-cause deaths avoided (number of avoided events: NAEs) under treatment and the individual absolute benefit. The total budget was fixed at the annual amount reimbursed by the French social security for statins. With the CPGs, the NAEs was 292 for an annual cost of 122.54 M€ compared with 443 with OMES. For a fixed NAEs, OMES reduced costs by 50% (60.53 M€ yr(-1)). The results demonstrate that OMES is at least as good as, and even better than, the standard CPGs when applied to the same population. Hence the OMES approach is a practical, useful alternative which will help to overcome the limitations of treatment decision-making based uniquely on CPGs.

Effect model law: an approach for the implementation of personalized medicine.

The effect model law states that a natural relationship exists between the frequency (observation) or the probability (prediction) of a morbid event without any treatment and the frequency or probability of the same event with a treatment. This relationship is called the effect model. It applies to a single individual, individuals within a population, or groups. In the latter case, frequencies or probabilities are averages of the group. The relationship is specific to a therapy, a disease or an event, and a period of observation. If one single disease is expressed through several distinct events, a treatment will be characterized by as many effect models. Empirical evidence, simulations with models of diseases and therapies and virtual populations, as well as theoretical derivation support the existence of the law. The effect model could be estimated through statistical fitting or mathematical modelling. It enables the prediction of the (absolute) benefit of a treatment for a given patient. It thus constitutes the theoretical basis for the design of practical tools for personalized medicine.

The global risk approach should be better applied in French hypertensive patients: a comparison between simulation and observation studies.

BACKGROUND: The prediction of the public health impact of a preventive strategy provides valuable support for decision-making. International guidelines for hypertension management have introduced the level of absolute cardiovascular risk in the definition of the treatment target population. The public health impact of implementing such a recommendation has not been measured. METHODOLOGY/PRINCIPAL FINDINGS: We assessed the efficiency of three treatment scenarios according to historical and current versions of practice guidelines on a Realistic Virtual Population representative of the French population aged from 35 to 64 years: 1) BP≥160/95 mm Hg; 2) BP≥140/90 mm Hg and 3) BP≥140/90 mm Hg plus increased CVD risk. We compared the eligibility following the ESC guidelines with the recently observed proportion of treated amongst hypertensive individuals reported by the Etude Nationale Nutrition Santé survey. Lowering the threshold to define hypertension multiplied by 2.5 the number of eligible individuals. Applying the cardiovascular risk rule reduced this number significantly: less than 1/4 of hypertensive women under 55 years and less than 1/3 of hypertensive men below 45 years of age. This was the most efficient strategy. Compared to the simulated guidelines application, men of all ages were undertreated (between 32 and 60%), as were women over 55 years (70%). By contrast, younger women were over-treated (over 200%). CONCLUSION: The global CVD risk approach to decide for treatment is more efficient than the simple blood pressure level. However, lack of screening rather than guideline application seems to explain the low prescription rates among hypertensive individuals in France. Multidimensional analyses required to obtain these results are possible only through databases at the individual level: realistic virtual populations should become the gold standard for assessing the impact of public health policies at the national level.

Towards personalized medicine: exploring the consequences of the effect model-based approach.

Although personalized medicine has been a subject of research and debate in recent years, it has been underused in medical practice, except in some cancers. We believe that the main reason for the gap between the potential of personalized medicine and its use in daily medical practice can be explained by the lack of an appropriate tool to facilitate the use of biomarker values in a doctor's decision-making process. We propose that the effect model could form the basis of such a tool.