Markov models for clinical decision-making in radiation oncology: A systematic review.

The intrinsic stochasticity of patients' response to treatment is a major consideration for clinical decision-making in radiation therapy. Markov models are powerful tools to capture this stochasticity and render effective treatment decisions. This paper provides an overview of the Markov models for clinical decision analysis in radiation oncology. A comprehensive literature search was conducted within MEDLINE using PubMed, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Only studies published from 2000 to 2023 were considered. Selected publications were summarized in two categories: (i) studies that compare two (or more) fixed treatment policies using Monte Carlo simulation and (ii) studies that seek an optimal treatment policy through Markov Decision Processes (MDPs). Relevant to the scope of this study, 61 publications were selected for detailed review. The majority of these publications (n = 56) focused on comparative analysis of two or more fixed treatment policies using Monte Carlo simulation. Classifications based on cancer site, utility measures and the type of sensitivity analysis are presented. Five publications considered MDPs with the aim of computing an optimal treatment policy; a detailed statement of the analysis and results is provided for each work. As an extension of Markov model-based simulation analysis, MDP offers a flexible framework to identify an optimal treatment policy among a possibly large set of treatment policies. However, the applications of MDPs to oncological decision-making have been understudied, and the full capacity of this framework to render complex optimal treatment decisions warrants further consideration.

Markov decision process design: A framework for integrating strategic and operational decisions.

We consider the problem of optimally designing a system for repeated use under uncertainty. We develop a modeling framework that integrates the design and operational phases, which are represented by a mixed-integer program and discounted-cost infinite-horizon Markov decision processes, respectively. We seek to simultaneously minimize the design costs and the subsequent expected operational costs. This problem setting arises naturally in several application areas, as we illustrate through examples. We derive a bilevel mixed-integer linear programming formulation for the problem and perform a computational study to demonstrate that realistic instances can be solved numerically.

Surveillance imaging for patients with head and neck cancer treated with definitive radiotherapy: A partially observed Markov decision process model.

BACKGROUND: A possible surveillance model for patients with head and neck cancer (HNC) who received definitive radiotherapy was created using a partially observed Markov decision process. The goal of this model is to guide surveillance imaging policies after definitive radiotherapy. METHODS: The partially observed Markov decision process model was formulated to determine the optimal times to scan patients. Transition probabilities were computed using a data set of 1508 patients with HNC who received definitive radiotherapy between the years 2000 and 2010. Kernel density estimation was used to smooth the sample distributions. The reward function was derived using cost estimates from the literature. Additional model parameters were estimated using either data from the literature or clinical expertise. RESULTS: When considering all forms of relapse, the model showed that the optimal time between scans was longer than the time intervals used in the institutional guidelines. The optimal policy dictates that there should be less time between surveillance scans immediately after treatment compared with years after treatment. Comparable results also held when only locoregional relapses were considered as relapse events in the model. Simulation results for the inclusive relapse cases showed that <15% of patients experienced a relapse over a simulated 36-month surveillance program. CONCLUSIONS: This model suggests that less frequent surveillance scan policies can maintain adequate information on relapse status for patients with HNC treated with radiotherapy. This model could potentially translate into a more cost-effective surveillance program for this group of patients.

An Online Calculator to Estimate the Impact of Changes in Breastfeeding Rates on Population Health and Costs.

OBJECTIVE: We sought to determine the impact of changes in breastfeeding rates on population health. MATERIALS AND METHODS: We used a Monte Carlo simulation model to estimate the population-level changes in disease burden associated with marginal changes in rates of any breastfeeding at each month from birth to 12 months of life, and in rates of exclusive breastfeeding from birth to 6 months of life. We used these marginal estimates to construct an interactive online calculator (available at www.usbreastfeeding.org/saving-calc ). The Institutional Review Board of the Cambridge Health Alliance exempted the study. RESULTS: Using our interactive online calculator, we found that a 5% point increase in breastfeeding rates was associated with statistically significant differences in child infectious morbidity for the U.S. population, including otitis media (101,952 cases, 95% confidence interval [CI] 77,929-131,894 cases) and gastrointestinal infection (236,073 cases, 95% CI 190,643-290,278 cases). Associated medical cost differences were $31,784,763 (95% CI $24,295,235-$41,119,548) for otitis media and $12,588,848 ($10,166,203-$15,479,352) for gastrointestinal infection. The state-level impact of attaining Healthy People 2020 goals varied by population size and current breastfeeding rates. CONCLUSION: Modest increases in breastfeeding rates substantially impact healthcare costs in the first year of life.

Suboptimal breastfeeding in the United States: Maternal and pediatric health outcomes and costs.

The aim of this study was to quantify the excess cases of pediatric and maternal disease, death, and costs attributable to suboptimal breastfeeding rates in the United States. Using the current literature on the associations between breastfeeding and health outcomes for nine pediatric and five maternal diseases, we created Monte Carlo simulations modeling a hypothetical cohort of U.S. women followed from age 15 to age 70 years and their children from birth to age 20 years. We examined disease outcomes using (a) 2012 breastfeeding rates and (b) assuming that 90% of infants were breastfed according to medical recommendations. We measured annual excess cases, deaths, and associated costs, in 2014 dollars, using a 2% discount rate. Annual excess deaths attributable to suboptimal breastfeeding total 3,340 (95% confidence interval [1,886 to 4,785]), 78% of which are maternal due to myocardial infarction (n = 986), breast cancer (n = 838), and diabetes (n = 473). Excess pediatric deaths total 721, mostly due to Sudden Infant Death Syndrome (n = 492) and necrotizing enterocolitis (n = 190). Medical costs total $3.0 billion, 79% of which are maternal. Costs of premature death total $14.2 billion. The number of women needed to breastfeed as medically recommended to prevent an infant gastrointestinal infection is 0.8; acute otitis media, 3; hospitalization for lower respiratory tract infection, 95; maternal hypertension, 55; diabetes, 162; and myocardial infarction, 235. For every 597 women who optimally breastfeed, one maternal or child death is prevented. Policies to increase optimal breastfeeding could result in substantial public health gains. Breastfeeding has a larger impact on women's health than previously appreciated.

Impact of Optimized Breastfeeding on the Costs of Necrotizing Enterocolitis in Extremely Low Birthweight Infants.

OBJECTIVE: To estimate risk of necrotizing enterocolitis (NEC) for extremely low birth weight (ELBW) infants as a function of preterm formula (PF) and maternal milk intake and calculate the impact of suboptimal feeding on the incidence and costs of NEC. STUDY DESIGN: We used aORs derived from the Glutamine Trial to perform Monte Carlo simulation of a cohort of ELBW infants under current suboptimal feeding practices, compared with a theoretical cohort in which 90% of infants received at least 98% human milk. RESULTS: NEC incidence among infants receiving ≥98% human milk was 1.3%; 11.1% among infants fed only PF; and 8.2% among infants fed a mixed diet (P = .002). In adjusted models, compared with infants fed predominantly human milk, we found an increased risk of NEC associated with exclusive PF (aOR = 12.1, 95% CI 1.5, 94.2), or a mixed diet (aOR 8.7, 95% CI 1.2-65.2). In Monte Carlo simulation, current feeding of ELBW infants was associated with 928 excess NEC cases and 121 excess deaths annually, compared with a model in which 90% of infants received ≥98% human milk. These models estimated an annual cost of suboptimal feeding of ELBW infants of $27.1 million (CI $24 million, $30.4 million) in direct medical costs, $563 655 (CI $476 191, $599 069) in indirect nonmedical costs, and $1.5 billion (CI $1.3 billion, $1.6 billion) in cost attributable to premature death. CONCLUSIONS: Among ELBW infants, not being fed predominantly human milk is associated with an increased risk of NEC. Efforts to support milk production by mothers of ELBW infants may prevent infant deaths and reduce costs.

Comparative effectiveness of guidelines for the management of hyperlipidemia and hypertension for type 2 diabetes patients.

BACKGROUND: Several guidelines to reduce cardiovascular risk in diabetes patients exist in North America, Europe, and Australia. Their ability to achieve this goal efficiently is unclear. METHODS AND FINDINGS: Decision analysis was used to compare the efficiency and effectiveness of international contemporary guidelines for the management of hypertension and hyperlipidemia for patients aged 40-80 with type 2 diabetes. Measures of comparative effectiveness included the expected probability of a coronary or stroke event, incremental medication costs per event, and number-needed-to-treat (NNT) to prevent an event. All guidelines are equally effective, but they differ significantly in their medication costs. The range of NNT to prevent an event was small across guidelines (6.5-7.6 for males and 6.5-7.5 for females); a larger range of differences were observed for expected cost per event avoided (ranges, $117,269-$157,186 for males and $115,999-$163,775 for females). Australian and U.S. guidelines result in the highest and lowest expected costs, respectively. CONCLUSIONS: International guidelines based on the same evidence and seeking the same goal are similar in their effectiveness; however, there are large differences in expected medication costs.

Markov decision processes: a tool for sequential decision making under uncertainty.

We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

Use of dynamic microsimulation to predict disease progression in patients with pneumonia-related sepsis.

INTRODUCTION: Sepsis is the leading cause of death in critically ill patients and often affects individuals with community-acquired pneumonia. To overcome the limitations of earlier mathematical models used to describe sepsis and predict outcomes, we designed an empirically based Monte Carlo model that simulates the progression of sepsis in hospitalized patients over a 30-day period. METHODS: The model simulates changing health over time, as represented by the Sepsis-related Organ Failure Assessment (SOFA) score, as a function of a patient's previous health state and length of hospital stay. We used data from patients enrolled in the GenIMS (Genetic and Inflammatory Markers of Sepsis) study to calibrate the model, and tested the model's ability to predict deaths, discharges, and daily SOFA scores over time using different algorithms to estimate the natural history of sepsis. We evaluated the stability of the methods using bootstrap sampling techniques. RESULTS: Of the 1,888 patients originally enrolled, most were elderly (mean age 67.77 years) and white (80.72%). About half (47.98%) were female. Most were relatively ill, with a mean Acute Physiology and Chronic Health Evaluation III score of 56 and Pneumonia Severity Index score of 73.5. The model's estimates of the daily pattern of deaths, discharges, and SOFA scores over time were not statistically different from the actual pattern when information about how long patients had been ill was included in the model (P = 0.91 to 0.98 for discharges; P = 0.26 to 0.68 for deaths). However, model estimates of these patterns were different from the actual pattern when the model did not include data on the duration of illness (P < 0.001 for discharges; P = 0.001 to 0.040 for deaths). Model results were stable to bootstrap validation. CONCLUSION: An empiric simulation model of sepsis can predict complex longitudinal patterns in the progression of sepsis, most accurately by models that contain data representing both organ-system levels of and duration of illness. This work supports the incorporation into mathematical models of disease of the clinical intuition that the history of disease in an individual matters, and represents an advance over several prior simulation models that assume a constant rate of disease progression.

Increasing the efficiency of Monte Carlo cohort simulations with variance reduction techniques.

The authors discuss techniques for Monte Carlo (MC) cohort simulations that reduce the number of simulation replications required to achieve a given degree of precision for various output measures. Known as variance reduction techniques, they are often used in industrial engineering and operations research models, but they are seldom used in medical models. However, most MC cohort simulations are well suited to the implementation of these techniques. The authors discuss the cost of implementation versus the benefit of reduced replications.

A clinically based discrete-event simulation of end-stage liver disease and the organ allocation process.

BACKGROUND: The optimal allocation of scarce donor livers is a contentious health care issue requiring careful analysis. The objective of this article was to design a biologically based discrete-event simulation to test proposed changes in allocation policies. METHODS: The authors used data from multiple sources to simulate end-stage liver disease and the complex allocation system. To validate the model, they compared simulation output with historical data. RESULTS: Simulation outcomes were within 1% to 2% of actual results for measures such as new candidates, donated livers, and transplants by year. The model overestimated the yearly size of the waiting list by 5% in the last year of the simulation and the total number of pretransplant deaths by 10%. CONCLUSION: The authors created a discrete-event simulation model that represents the biology of end-stage liver disease and the health care organization of transplantation in the United States.

Simulation and critical care modeling.

PURPOSE OF REVIEW: Decisions made in critical care are often complicated, requiring an in-depth understanding of the relations between complex diseases, available interventions, and patients with a wide range of characteristics. Standard modeling techniques such as decision trees and statistical modeling have difficulty in capturing these interactions as the complexity of the problem increases. RECENT FINDINGS: Recent models in the literature suggest that simulation modeling techniques such as Markov modeling, Monte Carlo simulation, and discrete-event simulation are useful tools for analyzing complex systems in critical care. These simulation techniques are reviewed briefly, and examples from the literature are presented to demonstrate their usefulness in understanding real problems in critical care. SUMMARY: Simulation models provide useful tools for organizing and analyzing the interactions between therapies, tradeoffs, and outcomes.