Covasim: An agent-based model of COVID-19 dynamics and interventions.

The COVID-19 pandemic has created an urgent need for models that can project epidemic trends, explore intervention scenarios, and estimate resource needs. Here we describe the methodology of Covasim (COVID-19 Agent-based Simulator), an open-source model developed to help address these questions. Covasim includes country-specific demographic information on age structure and population size; realistic transmission networks in different social layers, including households, schools, workplaces, long-term care facilities, and communities; age-specific disease outcomes; and intrahost viral dynamics, including viral-load-based transmissibility. Covasim also supports an extensive set of interventions, including non-pharmaceutical interventions, such as physical distancing and protective equipment; pharmaceutical interventions, including vaccination; and testing interventions, such as symptomatic and asymptomatic testing, isolation, contact tracing, and quarantine. These interventions can incorporate the effects of delays, loss-to-follow-up, micro-targeting, and other factors. Implemented in pure Python, Covasim has been designed with equal emphasis on performance, ease of use, and flexibility: realistic and highly customized scenarios can be run on a standard laptop in under a minute. In collaboration with local health agencies and policymakers, Covasim has already been applied to examine epidemic dynamics and inform policy decisions in more than a dozen countries in Africa, Asia-Pacific, Europe, and North America.

Characterization of a Real-World Response Variable and Comparison with RECIST-Based Response Rates from Clinical Trials in Advanced NSCLC.

INTRODUCTION: Effectiveness metrics for real-word research, analogous to clinical trial ones, are needed. This study aimed to develop a real-world response (rwR) variable applicable to solid tumors and to evaluate its clinical relevance and meaningfulness. METHODS: This retrospective study used patient cohorts with advanced non-small cell lung cancer from a nationwide, de-identified electronic health record (EHR)-derived database. Disease burden information abstracted manually was classified into response categories anchored to discrete therapy lines (per patient-line). In part 1, we quantified the feasibility and reliability of data capture, and estimated the association between rwR status and real-world progression-free survival (rwPFS) and real-world overall survival (rwOS). In part 2, we investigated the correlation between published clinical trial overall response rates (ORRs) and real-world response rates (rwRRs) from corresponding real-world patient cohorts. RESULTS: In part 1, 85.4% of patients (N = 3248) had at least one radiographic assessment documented. Median abstraction time per patient-line was 15.0 min (IQR 7.8-28.1). Inter-abstractor agreement on presence/absence of at least one assessment was 0.94 (95% CI 0.92-0.96; n = 503 patient-lines abstracted in duplicate); inter-abstractor agreement on best confirmed response category was 0.82 (95% CI 0.78-0.86; n = 384 with at least one captured assessment). Confirmed responders at a 3-month landmark showed significantly lower risk of death and progression in rwOS and rwPFS analyses across all line settings. In part 2, rwRRs (from 12 rw cohorts) showed a high correlation with trial ORRs (Spearman's ρ = 0.99). CONCLUSIONS: We developed a rwR variable generated from clinician assessments documented in EHRs following radiographic evaluations. This variable provides clinically meaningful information and may provide a real-world measure of treatment effectiveness.

Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment.

BACKGROUND: Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals. OBJECTIVE: This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience. METHODS: After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon. RESULTS: The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI. CONCLUSIONS: Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.

Clinical exome sequencing vs. usual care for hereditary colorectal cancer diagnosis: A pilot comparative effectiveness study.

BACKGROUND: Clinical exome sequencing (CES) provides the advantage of assessing genetic variation across the human exome compared to a traditional stepwise diagnostic approach or multi-gene panels. Comparative effectiveness research methods offer an approach to better understand the patient-centered and economic outcomes of CES. PURPOSE: To evaluate CES compared to usual care (UC) in the diagnostic work-up of inherited colorectal cancer/polyposis (CRCP) in a randomized controlled trial (RCT). METHODS: The primary outcome was clinical sensitivity for the diagnosis of inherited CRCP; secondary outcomes included psychosocial outcomes, family communication, and healthcare resource utilization. Participants were surveyed 2 and 4 weeks after results return and at 3-month intervals up to 1 year. RESULTS: Evolving outcome measures and standard of care presented critical challenges. The majority of participants in the UC arm received multi-gene panels [94.73%]. Rates of genetic findings supporting the diagnosis of hereditary CRCP were 7.5% [7/93] vs. 5.4% [5/93] in the CES and UC arms, respectively (P = 0.28). Differences in privacy concerns after receiving CRCP results were identified (0.88 in UC vs 0.38 in CES, P = 0.05); however, healthcare resource utilization, family communication and psychosocial outcomes were similar between the two arms. More participants with positive results (17.7%) intended to change their life insurance 1  month after the first return visit compared to participants returned a variant of uncertain significance (9.1%) or negative result (4.8%) (P = 0.09). CONCLUSION: Our results suggest that CES provides similar clinical benefits to multi-gene panels in the diagnosis of hereditary CRCP.

Do cancer treatments have option value? Real-world evidence from metastatic melanoma.

A change in the expectations about future treatments may change the option value of a current treatment, thereby affecting its utilization. We conducted an interrupted time series analysis using a large administrative claims database to test whether the utilization of existing cancer treatments changed after the disclosures of the then-investigational drug ipilimumab's Phase II and Phase III results among metastatic melanoma patients from 2008 to 2011. We used a multinomial logistic regression to analyze the temporal probability of receiving antineoplastic systemic therapy, surgical resection of metastasis, or both, relative to no treatment, in the first 3 months following the first metastasis diagnosis. One thousand eight hundred forty-six metastatic melanoma patients were included. After adjusting for clinical and sociodemographic variables and the underlying time trend, the disclosure of ipilimumab's Phase II result was associated with a nearly twofold immediate increase in the probability of receiving surgical resection of metastasis relative to no treatment, which was significant at 5% level. No significant effect was observed for the time trend. No significant effects were found for the announcement of the Phase III result. Our findings in metastatic melanoma provide the first empirical evidence of the impact of option value in cancer treatment decision making.

The Feelings About genomiC Testing Results (FACToR) Questionnaire: Development and Preliminary Validation.

The purpose of this study was to develop a brief instrument, the Feelings About genomiC Testing Results (FACToR), to measure the psychosocial impact of returning genomic findings to patients in research and clinical practice. To create the FACToR, we modified and augmented the Multidimensional Impact of Cancer Risk Assessment (MICRA) questionnaire based on findings from a literature review, two focus groups (N = 12), and cognitive interviews (N = 6). We evaluated data from 122 participants referred for evaluation for inherited colorectal cancer or polyposis from the New EXome Technology in (NEXT) Medicine Study, an RCT of exome sequencing versus usual care. We assessed floor and ceiling effects of each item, conducted principal component analysis to identify subscales, and evaluated each subscale's internal consistency, test-retest reliability, and construct validity. After excluding items that were ambiguous or demonstrated floor or ceiling effects, 12 items forming four distinct subscales were retained for further analysis: negative emotions, positive feelings, uncertainty, and privacy concerns. All four showed good internal consistency (0.66-0.78) and test-retest reliability (0.65-0.91). The positive feelings and the uncertainty subscales demonstrated known-group validity. The 12-item FACToR with four subscales shows promising psychometric properties on preliminary evaluation in a limited sample and needs to be evaluated in other populations.

Integrating value of research into NCI Clinical Trials Cooperative Group research review and prioritization: A pilot study.

BACKGROUND: The Institute of Medicine has called for approaches to help maximize the return on investments (ROI) in cancer clinical trials. Value of Research (VOR) is a health economics technique that estimates ROI and can inform research prioritization. Our objective was to evaluate the impact of using VOR analyses on the clinical trial proposal review process within the SWOG cancer clinical trials consortium. METHODS: We used a previously developed minimal modeling approach to calculate VOR estimates for 9 phase II/III SWOG proposals between February 2015 and December 2016. Estimates were presented to executive committee (EC) members (N = 12) who determine which studies are sent to the National Cancer Institute for funding consideration. EC members scored proposals from 1 (best) to 5 based on scientific merit and potential impact before and after receiving VOR estimates. EC members were surveyed to assess research priorities, proposal evaluation process satisfaction, and the VOR process. RESULTS: Value of Research estimates ranged from -$2.1B to $16.46B per proposal. Following review of VOR results, the EC changed their score for eight of nine proposals. Proposal rankings were different in pre- vs postscores (P value: 0.03). Respondents had mixed views of the ultimate utility of VOR for their decisions with most supporting (42%) or neutral (41%) to the idea of adding VOR to the evaluation process. CONCLUSIONS: The findings from this pilot study indicate use of VOR analyses may be a useful adjunct to inform proposal reviews within NCI Cooperative Clinical Trials groups.

Steady Increase In Prices For Oral Anticancer Drugs After Market Launch Suggests A Lack Of Competitive Pressure.

The cost of treating cancer has risen to unprecedented heights, putting tremendous financial pressure on patients, payers, and society. Previous studies have documented the rising prices of cancer drugs at launch, but less critical attention has been paid to the cost of these drugs after launch. We used pharmacy claims for commercially insured individuals to examine trends in postlaunch prices over time for orally administered anticancer drugs recently approved by the Food and Drug Administration (FDA). In the period 2007-13, inflation-adjusted per patient monthly drug prices increased 5 percent each year. Certain market changes also played a role, with prices rising an additional 10 percent with each supplemental indication approved by the FDA and declining 2 percent with the FDA's approval of a competitor drug. Our findings suggest that there is currently little competitive pressure in the oral anticancer drug market. Policy makers who wish to reduce the costs of anticancer drugs should consider implementing policies that affect prices not only at launch but also later.

Breast MRI in the Diagnostic and Preoperative Workup Among Medicare Beneficiaries With Breast Cancer.

PURPOSE: We compared the frequency and sequence of breast imaging and biopsy use for the diagnostic and preoperative workup of breast cancer according to breast magnetic resonance imaging (MRI) use among older women. MATERIALS AND METHODS: Using SEER-Medicare data from 2004 to 2010, we identified women with and without breast MRI as part of their diagnostic and preoperative breast cancer workup and measured the number and sequence of breast imaging and biopsy events per woman. RESULTS: A total of 10,766 (20%) women had an MRI in the diagnostic/preoperative period, 32,178 (60%) had mammogram and ultrasound, and 10,669 (20%) had mammography alone. MRI use increased across study years, tripling from 2005 to 2009 (9%-29%). Women with MRI had higher rates of breast imaging and biopsy compared with those with mammogram and ultrasound or those with mammography alone (5.8 vs. 4.1 vs. 2.8, respectively). There were 4254 unique sequences of breast events; the dominant patterns for women with MRI were an MRI occurring at the end of the care pathway. Among women receiving an MRI postdiagnosis, 26% had a subsequent biopsy compared with 51% receiving a subsequent biopsy in the subgroup without MRI. CONCLUSIONS: Older women who receive breast MRI undergo additional breast imaging and biopsy events. There is much variability in the diagnostic/preoperative work-up in older women, demonstrating the opportunity to increase standardization to optimize care for all women.

Development and Evaluation of an Approach to Using Value of Information Analyses for Real-Time Prioritization Decisions Within SWOG, a Large Cancer Clinical Trials Cooperative Group.

OBJECTIVE: Value of information (VOI) analyses can align research with areas with the greatest potential impact on patient outcome, but questions remain concerning the feasibility and acceptability of these approaches to inform prioritization decisions. Our objective was to develop a process for calculating VOI in "real time" to inform trial funding decisions within SWOG, a large cancer clinical trials group. METHODS: We developed an efficient and scalable VOI modeling approach using a selected sample of 9 randomized phase II/III trial proposals from the Breast, Gastrointestinal, and Genitourinary Disease Committees reviewed by SWOG's leadership between 2008 and 2013. There was bidirectional communication between SWOG investigators and the research team throughout the modeling development. Partial expected value of sample information for the treatment effect evaluated by the proposed trial's primary endpoint was calculated using Monte Carlo simulation. RESULTS: We derived prior uncertainty in the treatment effect estimate from the sample size calculations. Our process was feasible for 8 of 9 trial proposals and efficient: the time required of 1 researcher was <1 week per proposal. We accommodated stakeholder input primarily by deconstructing VOI metrics into expected health benefits and incremental healthcare costs and assuming treatment decisions within our simulations were based on health benefits. Following customization, feedback from over 200 SWOG members was positive regarding the overall VOI framework, specific retrospective results, and potential for VOI analyses to inform future trial proposal evaluations. CONCLUSIONS: We developed an efficient and customized process to calculate the expected VOI of cancer clinical trials that is feasible for use in decision making and acceptable to investigators. Prospective use and evaluation of this approach is currently underway within SWOG.

Predicting Low Accrual in the National Cancer Institute's Cooperative Group Clinical Trials.

BACKGROUND: The extent to which trial-level factors differentially influence accrual to trials has not been comprehensively studied. Our objective was to evaluate the empirical relationship and predictive properties of putative risk factors for low accrual in the National Cancer Institute's (NCI's) Cooperative Group Program, now the National Clinical Trials Network (NCTN). METHODS: Data from 787 phase II/III adult NCTN-sponsored trials launched between 2000 and 2011 were used to develop a logistic regression model to predict low accrual, defined as trials that closed with or were accruing at less than 50% of target; 46 trials opened between 2012 and 2013 were used for prospective validation. Candidate predictors were identified from a literature review and expert interviews; final predictors were selected using stepwise regression. Model performance was evaluated by calibration and discrimination via the area under the curve (AUC). All statistical tests were two-sided. RESULTS: Eighteen percent (n = 145) of NCTN-sponsored trials closed with low accrual or were accruing at less than 50% of target three years or more after initiation. A multivariable model of twelve trial-level risk factors had good calibration and discrimination for predicting trials with low accrual (AUC in trials launched 2000-2011 = 0.739, 95% confidence interval [CI] = 0.696 to 0.783]; 2012-2013: AUC = 0.732, 95% CI = 0.547 to 0.917). Results were robust to different definitions of low accrual and predictor selection strategies. CONCLUSIONS: We identified multiple characteristics of NCTN-sponsored trials associated with low accrual, several of which have not been previously empirically described, and developed a prediction model that can provide a useful estimate of accrual risk based on these factors. Future work should assess the role of such prediction tools in trial design and prioritization decisions.

Next-Generation Sequencing Panels for the Diagnosis of Colorectal Cancer and Polyposis Syndromes: A Cost-Effectiveness Analysis.

PURPOSE: To evaluate the cost effectiveness of next-generation sequencing (NGS) panels for the diagnosis of colorectal cancer and polyposis (CRCP) syndromes in patients referred to cancer genetics clinics. PATIENTS AND METHODS: We developed a decision model to evaluate NGS panel testing compared with current standard of care in patients referred to a cancer genetics clinic. We obtained data on the prevalence of genetic variants from a large academic laboratory and calculated the costs and health benefits of identifying relatives with a pathogenic variant, in life-years and quality-adjusted life-years (QALYs). We classified the CRCP syndromes according to their type of inheritance and penetrance of colorectal cancer. One-way and probabilistic sensitivity analyses were conducted to assess uncertainty. RESULTS: Evaluation with an NGS panel that included Lynch syndrome genes and other genes associated with highly penetrant CRCP syndromes led to an average increase of 0.151 year of life, 0.128 QALY, and $4,650 per patient, resulting in an incremental cost-effectiveness ratio of $36,500 per QALY compared with standard care and a 99% probability that this panel was cost effective at a threshold of $100,000 per QALY. When compared with this panel, the addition of genes with low colorectal cancer penetrance resulted in an incremental cost-effectiveness ratio of $77,300 per QALY. CONCLUSION: The use of an NGS panel that includes genes associated with highly penetrant CRCP syndromes in addition to Lynch syndrome genes as a first-line test is likely to provide meaningful clinical benefits in a cost-effective manner at a $100,000 per QALY threshold.

The cost-effectiveness of returning incidental findings from next-generation genomic sequencing.

PURPOSE: The American College of Medical Genetics and Genomics (ACMG) recommended that clinical laboratories performing next-generation sequencing analyze and return pathogenic variants for 56 specific genes it considered medically actionable. Our objective was to evaluate the clinical and economic impact of returning these results. METHODS: We developed a decision-analytic policy model to project the quality-adjusted life-years and lifetime costs associated with returning ACMG-recommended incidental findings in three hypothetical cohorts of 10,000 patients. RESULTS: Returning incidental findings to cardiomyopathy patients, colorectal cancer patients, or healthy individuals would increase costs by $896,000, $2.9 million, and $3.9 million, respectively, and would increase quality-adjusted life-years by 20, 25.4, and 67 years, respectively, for incremental cost-effectiveness ratios of $44,800, $115,020, and $58,600, respectively. In probabilistic analyses, returning incidental findings cost less than $100,000/quality-adjusted life-year gained in 85, 28, and 91%, respectively, of simulations. Assuming next-generation sequencing costs $500, the incremental cost-effectiveness ratio for primary screening of healthy individuals was $133,400 (<$100,000/quality-adjusted life-year gained in 10% of simulations). Results were sensitive to the cohort age and assumptions about gene penetrance. CONCLUSION: Returning incidental findings is likely cost-effective for certain patient populations. Screening of generally healthy individuals is likely not cost-effective based on current data, unless next-generation sequencing costs less than $500.

Comparative effectiveness of next generation genomic sequencing for disease diagnosis: design of a randomized controlled trial in patients with colorectal cancer/polyposis syndromes.

Whole exome and whole genome sequencing are applications of next generation sequencing transforming clinical care, but there is little evidence whether these tests improve patient outcomes or if they are cost effective compared to current standard of care. These gaps in knowledge can be addressed by comparative effectiveness and patient-centered outcomes research. We designed a randomized controlled trial that incorporates these research methods to evaluate whole exome sequencing compared to usual care in patients being evaluated for hereditary colorectal cancer and polyposis syndromes. Approximately 220 patients will be randomized and followed for 12 months after return of genomic findings. Patients will receive findings associated with colorectal cancer in a first return of results visit, and findings not associated with colorectal cancer (incidental findings) during a second return of results visit. The primary outcome is efficacy to detect mutations associated with these syndromes; secondary outcomes include psychosocial impact, cost-effectiveness and comparative costs. The secondary outcomes will be obtained via surveys before and after each return visit. The expected challenges in conducting this randomized controlled trial include the relatively low prevalence of genetic disease, difficult interpretation of some genetic variants, and uncertainty about which incidental findings should be returned to patients. The approaches utilized in this study may help guide other investigators in clinical genomics to identify useful outcome measures and strategies to address comparative effectiveness questions about the clinical implementation of genomic sequencing in clinical care.