Evaluating risk factor assumptions: a simulation-based approach.

BACKGROUND: Microsimulation models are an important tool for estimating the comparative effectiveness of interventions through prediction of individual-level disease outcomes for a hypothetical population. To estimate the effectiveness of interventions targeted toward high risk groups, the mechanism by which risk factors influence the natural history of disease must be specified. We propose a method for evaluating these risk factor assumptions as part of model-building. METHODS: We used simulation studies to examine the impact of risk factor assumptions on the relative rate (RR) of colorectal cancer (CRC) incidence and mortality for a cohort with a risk factor compared to a cohort without the risk factor using an extension of the CRC-SPIN model for colorectal cancer. We also compared the impact of changing age at initiation of screening colonoscopy for different risk mechanisms. RESULTS: Across CRC-specific risk factor mechanisms, the RR of CRC incidence and mortality decreased (towards one) with increasing age. The rate of change in RRs across age groups depended on both the risk factor mechanism and the strength of the risk factor effect. Increased non-CRC mortality attenuated the effect of CRC-specific risk factors on the RR of CRC when both were present. For each risk factor mechanism, earlier initiation of screening resulted in more life years gained, though the magnitude of life years gained varied across risk mechanisms. CONCLUSIONS: Simulation studies can provide insight into both the effect of risk factor assumptions on model predictions and the type of data needed to calibrate risk factor models.

A systematic comparison of microsimulation models of colorectal cancer: the role of assumptions about adenoma progression.

BACKGROUND: As the complexity of microsimulation models increases, concerns about model transparency are heightened. METHODS: The authors conducted model "experiments" to explore the impact of variations in "deep" model parameters using 3 colorectal cancer (CRC) models. All natural history models were calibrated to match observed data on adenoma prevalence and cancer incidence but varied in their underlying specification of the adenocarcinoma process. The authors projected CRC incidence among individuals with an underlying adenoma or preclinical cancer v. those without any underlying condition and examined the impact of removing adenomas. They calculated the percentage of simulated CRC cases arising from adenomas that developed within 10 or 20 years prior to cancer diagnosis and estimated dwell time-defined as the time from the development of an adenoma to symptom-detected cancer in the absence of screening among individuals with a CRC diagnosis. RESULTS: The 20-year CRC incidence among 55-year-old individuals with an adenoma or preclinical cancer was 7 to 75 times greater than in the condition-free group. The removal of all adenomas among the subgroup with an underlying adenoma or cancer resulted in a reduction of 30% to 89% in cumulative incidence. Among CRCs diagnosed at age 65 years, the proportion arising from adenomas formed within 10 years ranged between 4% and 67%. The mean dwell time varied from 10.6 to 25.8 years. CONCLUSIONS: Models that all match observed data on adenoma prevalence and cancer incidence can produce quite different dwell times and very different answers with respect to the effectiveness of interventions. When conducting applied analyses to inform policy, using multiple models provides a sensitivity analysis on key (unobserved) "deep" model parameters and can provide guidance about specific areas in need of additional research and validation.

Clarifying differences in natural history between models of screening: the case of colorectal cancer.

BACKGROUND: Microsimulation models are important decision support tools for screening. However, their complexity makes them difficult to understand and limits realization of their full potential. Therefore, it is important to develop documentation that clarifies their structure and assumptions. The authors demonstrate this problem and explore a solution for natural history using 3 independently developed colorectal cancer screening models. METHODS: The authors first project effectiveness and cost-effectiveness of colonoscopy screening for the 3 models (CRC-SPIN, SimCRC, and MISCAN). Next, they provide a conventional presentation of each model, including information on structure and parameter values. Finally, they report the simulated reduction in clinical cancer incidence following a one-time complete removal of adenomas and preclinical cancers for each model. They call this new measure the maximum clinical incidence reduction (MCLIR). RESULTS: Projected effectiveness varies widely across models. For example, estimated mortality reduction for colonoscopy screening every 10 years from age 50 to 80 years, with surveillance in adenoma patients, ranges from 65% to 92%. Given only conventional information, it is difficult to explain these differences, largely because differences in structure make parameter values incomparable. In contrast, the MCLIR clearly shows the impact of model differences on the key feature of natural history, the dwell time of preclinical disease. Dwell times vary from 8 to 25 years across models and help explain differences in projected screening effectiveness. CONCLUSIONS: The authors propose a new measure, the MCLIR, which summarizes the implications for predicted screening effectiveness of differences in natural history assumptions. Including the MCLIR in the standard description of a screening model would improve the transparency of these models.

An evidence-based microsimulation model for colorectal cancer: validation and application.

BACKGROUND: The Colorectal Cancer Simulated Population model for Incidence and Natural history (CRC-SPIN) is a new microsimulation model for the natural history of colorectal cancer that can be used for comparative effectiveness studies of colorectal cancer screening modalities. METHODS: CRC-SPIN simulates individual event histories associated with colorectal cancer, based on the adenoma-carcinoma sequence: adenoma initiation and growth, development of preclinical invasive colorectal cancer, development of clinically detectable colorectal cancer, death from colorectal cancer, and death from other causes. We present the CRC-SPIN structure and parameters, data used for model calibration, and model validation. We also provide basic model outputs to further describe CRC-SPIN, including annual transition probabilities between various disease states and dwell times. We conclude with a simple application that predicts the impact of a one-time colonoscopy at age 50 on the incidence of colorectal cancer assuming three different operating characteristics for colonoscopy. RESULTS: CRC-SPIN provides good prediction of both the calibration and the validation data. Using CRC-SPIN, we predict that a one-time colonoscopy greatly reduces colorectal cancer incidence over the subsequent 35 years. CONCLUSIONS: CRC-SPIN is a valuable new tool for combining expert opinion with observational and experimental results to predict the comparative effectiveness of alternative colorectal cancer screening modalities. IMPACT: Microsimulation models such as CRC-SPIN can serve as a bridge between screening and treatment studies and health policy decisions by predicting the comparative effectiveness of different interventions. As such, it is critical to publish model descriptions that provide insight into underlying assumptions along with validation studies showing model performance.

Cost-effectiveness of computed tomographic colonography screening for colorectal cancer in the medicare population.

BACKGROUND: The Centers for Medicare and Medicaid Services (CMS) considered whether to reimburse computed tomographic colonography (CTC) for colorectal cancer screening of Medicare enrollees. To help inform its decision, we evaluated the reimbursement rate at which CTC screening could be cost-effective compared with the colorectal cancer screening tests that are currently reimbursed by CMS and are included in most colorectal cancer screening guidelines, namely annual fecal occult blood test (FOBT), flexible sigmoidoscopy every 5 years, flexible sigmoidoscopy every 5 years in conjunction with annual FOBT, and colonoscopy every 10 years. METHODS: We used three independently developed microsimulation models to assess the health outcomes and costs associated with CTC screening and with currently reimbursed colorectal cancer screening tests among the average-risk Medicare population. We assumed that CTC was performed every 5 years (using test characteristics from either a Department of Defense CTC study or the National CTC Trial) and that individuals with findings of 6 mm or larger were referred to colonoscopy. We computed incremental cost-effectiveness ratios for the currently reimbursed screening tests and calculated the maximum cost per scan (ie, the threshold cost) for the CTC strategy to lie on the efficient frontier. Sensitivity analyses were performed on key parameters and assumptions. RESULTS: Assuming perfect adherence with all tests, the undiscounted number life-years gained from CTC screening ranged from 143 to 178 per 1000 65-year-olds, which was slightly less than the number of life-years gained from 10-yearly colonoscopy (152-185 per 1000 65-year-olds) and comparable to that from 5-yearly sigmoidoscopy with annual FOBT (149-177 per 1000 65-year-olds). If CTC screening was reimbursed at $488 per scan (slightly less than the reimbursement for a colonoscopy without polypectomy), it would be the most costly strategy. CTC screening could be cost-effective at $108-$205 per scan, depending on the microsimulation model used. Sensitivity analyses showed that if relative adherence to CTC screening was 25% higher than adherence to other tests, it could be cost-effective if reimbursed at $488 per scan. CONCLUSIONS: CTC could be a cost-effective option for colorectal cancer screening among Medicare enrollees if the reimbursement rate per scan is substantially less than that for colonoscopy or if a large proportion of otherwise unscreened persons were to undergo screening by CTC.

Bayesian Calibration of Microsimulation Models.

Microsimulation models that describe disease processes synthesize information from multiple sources and can be used to estimate the effects of screening and treatment on cancer incidence and mortality at a population level. These models are characterized by simulation of individual event histories for an idealized population of interest. Microsimulation models are complex and invariably include parameters that are not well informed by existing data. Therefore, a key component of model development is the choice of parameter values. Microsimulation model parameter values are selected to reproduce expected or known results though the process of model calibration. Calibration may be done by perturbing model parameters one at a time or by using a search algorithm. As an alternative, we propose a Bayesian method to calibrate microsimulation models that uses Markov chain Monte Carlo. We show that this approach converges to the target distribution and use a simulation study to demonstrate its finite-sample performance. Although computationally intensive, this approach has several advantages over previously proposed methods, including the use of statistical criteria to select parameter values, simultaneous calibration of multiple parameters to multiple data sources, incorporation of information via prior distributions, description of parameter identifiability, and the ability to obtain interval estimates of model parameters. We develop a microsimulation model for colorectal cancer and use our proposed method to calibrate model parameters. The microsimulation model provides a good fit to the calibration data. We find evidence that some parameters are identified primarily through prior distributions. Our results underscore the need to incorporate multiple sources of variability (i.e., due to calibration data, unknown parameters, and estimated parameters and predicted values) when calibrating and applying microsimulation models.