Using measures of race to make clinical predictions: Decision making, patient health, and fairness.

The use of race measures in clinical prediction models is contentious. We seek to inform the discourse by evaluating the inclusion of race in probabilistic predictions of illness that support clinical decision making. Adopting a static utilitarian framework to formalize social welfare, we show that patients of all races benefit when clinical decisions are jointly guided by patient race and other observable covariates. Similar conclusions emerge when the model is extended to a two-period setting where prevention activities target systemic drivers of disease. We also discuss non-utilitarian concepts that have been proposed to guide allocation of health care resources.

Balancing data privacy and usability in the federal statistical system.

The federal statistical system is experiencing competing pressures for change. On the one hand, for confidentiality reasons, much socially valuable data currently held by federal agencies is either not made available to researchers at all or only made available under onerous conditions. On the other hand, agencies which release public databases face new challenges in protecting the privacy of the subjects in those databases, which leads them to consider releasing fewer data or masking the data in ways that will reduce their accuracy. In this essay, we argue that the discussion has not given proper consideration to the reduced social benefits of data availability and their usability relative to the value of increased levels of privacy protection. A more balanced benefit-cost framework should be used to assess these trade-offs. We express concerns both with synthetic data methods for disclosure limitation, which will reduce the types of research that can be reliably conducted in unknown ways, and with differential privacy criteria that use what we argue is an inappropriate measure of disclosure risk. We recommend that the measure of disclosure risk used to assess all disclosure protection methods focus on what we believe is the risk that individuals should care about, that more study of the impact of differential privacy criteria and synthetic data methods on data usability for research be conducted before either is put into widespread use, and that more research be conducted on alternative methods of disclosure risk reduction that better balance benefits and costs.

Toward Patient-Centered Drug Approval for Treatment of Rare Diseases.

OBJECTIVES: This commentary seeks to improve the design and analysis of trials undertaken to obtain approval of drugs for treatment of rare diseases. METHODS: Methodological analysis reveals that use of hypothesis testing in the Food and Drug Administration drug approval process is harmful. Conventional asymmetric error probabilities bias the approval process against approval of new drugs. Hypothesis testing is inattentive to the relative magnitudes of losses to patient welfare when types 1 and 2 errors occur. Requiring the sample size to be large enough to guarantee the specified statistical power particularly inhibits the development of new drugs for treating rare diseases. Rarity of a disease makes it difficult to enroll the number of trial subjects needed to meet the statistical power standards for drug approval. RESULTS: Use of statistical decision theory in drug approval would overcome these serious deficiencies of hypothesis testing. Sample size would remain relevant to drug approval, but the criterion used to evaluate sample size would change. Rather than judging sample size by statistical power, the Food and Drug Administration could require a sample to be large enough to provide a specified nearness to optimality of the approval decision. CONCLUSIONS: Using nearness to optimality to set sample size and making approval decisions to minimize distance from optimality would particularly benefit the evaluation of drugs for treatment of rare diseases. It would enable a dramatic reduction in sample size relative to current norms, without compromising the clinical informativeness of trials.

Addressing partial identification in climate modeling and policy analysis.

Numerical simulations of the global climate system provide inputs to integrated assessment modeling for estimating the impacts of greenhouse gas mitigation and other policies to address global climate change. While essential tools for this purpose, computational climate models are subject to considerable uncertainty, including intermodel "structural" uncertainty. Structural uncertainty analysis has emphasized simple or weighted averaging of the outputs of multimodel ensembles, sometimes with subjective Bayesian assignment of probabilities across models. However, choosing appropriate weights is problematic. To use climate simulations in integrated assessment, we propose, instead, framing climate model uncertainty as a problem of partial identification, or "deep" uncertainty. This terminology refers to situations in which the underlying mechanisms, dynamics, or laws governing a system are not completely known and cannot be credibly modeled definitively even in the absence of data limitations in a statistical sense. We propose the min-max regret (MMR) decision criterion to account for deep climate uncertainty in integrated assessment without weighting climate model forecasts. We develop a theoretical framework for cost-benefit analysis of climate policy based on MMR, and apply it computationally with a simple integrated assessment model. We suggest avenues for further research.

Partial Identification of Personalized Treatment Response with Trial-reported Analyses of Binary Subgroups.

Medical journals have adhered to a reporting practice that seriously limits the usefulness of published trial findings. Medical decision makers commonly observe many patient covariates and seek to use this information to personalize treatment choices. Yet standard summaries of trial findings only partition subjects into broad subgroups, typically binary categories. Given this reporting practice, we study the problem of inference on long mean treatment outcomes E[y(t)|x], where t is a treatment, y(t) is a treatment outcome, and the covariate vector x has length K, each component being a binary variable. The available data are estimates of {E[y(t)|x k = 0], E[y(t)|x k = 1], P(x k )}, k = 1,..., K reported in journal articles. We show that reported trial findings partially identify {E[y(t)|x], P(x)}. Illustrative computations demonstrate that the summaries of trial findings in journal articles may imply only wide bounds on long mean outcomes. One can realistically tighten inferences if one can combine reported trial findings with credible assumptions having identifying power, such as bounded-variation assumptions.

Patient-centered appraisal of race-free clinical risk assessment.

Until recently, there has been a consensus that clinicians seeking to assess patient risks of illness should condition risk assessments on all observed patient covariates with predictive power. The broad idea is that knowing more about patients enables more accurate predictions of their health risks and, hence, better clinical decisions. This consensus has recently unraveled with respect to a specific covariate, namely race. There have been increasing calls for race-free risk assessment, arguing that using race to predict health risks contributes to racial disparities and inequities in health care. In some medical fields, leading institutions have recommended race-free risk assessment. An important open question is how race-free risk assessment would affect the quality of clinical decisions. Considering the matter from the patient-centered perspective of medical economics yields a disturbing conclusion: Race-free risk assessment would harm patients of all races.

Communicating uncertainty in policy analysis.

The term "policy analysis" describes scientific evaluations of the impacts of past public policies and predictions of the outcomes of potential future policies. A prevalent practice has been to report policy analysis with incredible certitude. That is, exact predictions of policy outcomes are routine, while expressions of uncertainty are rare. However, predictions and estimates often are fragile, resting on unsupported assumptions and limited data. Therefore, the expressed certitude is not credible. This paper summarizes my work documenting incredible certitude and calling for transparent communication of uncertainty. I present a typology of practices that contribute to incredible certitude, give illustrative examples, and offer suggestions on how to communicate uncertainty.

Tuberculosis diagnosis and treatment under uncertainty.

In 2017, 1.6 million people worldwide died from tuberculosis (TB). A new TB diagnostic test-Xpert MTB/RIF from Cepheid-was endorsed by the World Health Organization in 2010. Trials demonstrated that Xpert is faster and has greater sensitivity and specificity than smear microscopy-the most common sputum-based diagnostic test. However, subsequent trials found no impact of introducing Xpert on morbidity and mortality. We present a decision-theoretic model of how a clinician might decide whether to order Xpert or other tests for TB, and whether to treat a patient, with or without test results. Our first result characterizes the conditions under which it is optimal to perform empirical treatment; that is, treatment without diagnostic testing. We then examine the implications for decision making of partial knowledge of TB prevalence or test accuracy. This partial knowledge generates ambiguity, also known as deep uncertainty, about the best testing and treatment policy. In the presence of such ambiguity, we show the usefulness of diversification of testing and treatment.

Predicting kidney transplant outcomes with partial knowledge of HLA mismatch.

We consider prediction of graft survival when a kidney from a deceased donor is transplanted into a recipient, with a focus on the variation of survival with degree of human leukocyte antigen (HLA) mismatch. Previous studies have used data from the Scientific Registry of Transplant Recipients (SRTR) to predict survival conditional on partial characterization of HLA mismatch. Whereas earlier studies assumed proportional hazards models, we used nonparametric regression methods. These do not make the unrealistic assumption that relative risks are invariant as a function of time since transplant, and hence should be more accurate. To refine the predictions possible with partial knowledge of HLA mismatch, it has been suggested that HaploStats statistics on the frequencies of haplotypes within specified ethnic/national populations be used to impute complete HLA types. We counsel against this, showing that it cannot improve predictions on average and sometimes yields suboptimal transplant decisions. We show that the HaploStats frequency statistics are nevertheless useful when combined appropriately with the SRTR data. Analysis of the ecological inference problem shows that informative bounds on graft survival probabilities conditional on refined HLA typing are achievable by combining SRTR and HaploStats data with immunological knowledge of the relative effects of mismatch at different HLA loci.

Tail and Center Rounding of Probabilistic Expectations in the Health and Retirement Study.

We study rounding of numerical expectations in the Health and Retirement Study (HRS) between 2002 and 2014. We document that respondent-specific rounding patterns across questions in individual waves are quite stable across waves. We discover a tendency by about half of the respondents to provide more refined responses in the tails of the 0-100 scale than the center. In contrast, only about five percent of the respondents give more refined responses in the center than the tails. We find that respondents tend to report the values 25 and 75 more frequently than other values ending in 5. We also find that rounding practices vary somewhat across question domains and respondent characteristics. We propose an inferential approach that assumes stability of response tendencies across questions and waves to infer person-specific rounding in each question domain and scale segment and that replaces each point-response with an interval representing the range of possible values of the true latent belief. Using expectations from the 2016 wave of the HRS, we validate our approach. To demonstrate the consequences of rounding on inference, we compare best-predictor estimates from face-value expectations with those implied by our intervals.

Bounding the Accuracy of Diagnostic Tests, With Application to COVID-19 Antibody Tests.

Tests used to diagnose illness commonly have imperfect accuracy, with some false-positive and negative results. For risk assessment and clinical decisions, predictive values are of interest. Positive predictive value (PPV) is the chance that a member of a relevant population who tests positive has been ill. Negative predictive value (NPV) is the chance that someone who tests negative has not been ill. The medical literature regularly reports sensitivity and specificity. Sensitivity is the chance that an ill person receives a positive test result. Specificity is the chance that a nonill person receives a negative result. Knowledge of sensitivity and specificity enables one to predict the test result given a person's illness status. These predictions are not directly relevant to patient care but, given knowledge of sensitivity and specificity, PPV and NPV can be derived if one knows the prevalence of the disease, the population rate of illness. There is considerable uncertainty about the prevalence of some diseases, a notable case being COVID-19. This paper addresses the problem of identification of PPV and NPV given knowledge of sensitivity and specificity and given bounds on prevalence. I explain the problem and show how to bound PPV and NPV as well as the risk ratio and difference, which are functions thereof. I apply the findings to COVID-19 antibody tests. I question the realism of supposing that sensitivity and specificity are known.

Statistical Decision Properties of Imprecise Trials Assessing Coronavirus Disease 2019 (COVID-19) Drugs.

OBJECTIVES: Researchers studying treatment of coronavirus disease 2019 (COVID-19) have reported findings of randomized trials comparing standard care with care augmented by experimental drugs. Many trials have small sample sizes, so estimates of treatment effects are imprecise. Hence, clinicians may find it difficult to decide when to treat patients with experimental drugs. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. We study treatment choice from the perspective of statistical decision theory, which considers treatment options symmetrically when assessing trial findings. METHODS: We use the concept of near-optimality to evaluate criteria for treatment choice. This concept jointly considers the probability and magnitude of decision errors. An appealing criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial. RESULTS: Considering the design of some COVID-19 trials, we show that the empirical success rule yields treatment choices that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests. CONCLUSION: Using trial findings to make near-optimal treatment choices rather than perform hypothesis tests should improve clinical decision making.

Estimating the COVID-19 infection rate: Anatomy of an inference problem.

As a consequence of missing data on tests for infection and imperfect accuracy of tests, reported rates of cumulative population infection by the SARS CoV-2 virus are lower than actual rates of infection. Hence, reported rates of severe illness conditional on infection are higher than actual rates. Understanding the time path of the COVID-19 pandemic has been hampered by the absence of bounds on infection rates that are credible and informative. This paper explains the logical problem of bounding these rates and reports illustrative findings, using data from Illinois, New York, and Italy. We combine the data with assumptions on the infection rate in the untested population and on the accuracy of the tests that appear credible in the current context. We find that the infection rate might be substantially higher than reported. We also find that, assuming accurate reporting of deaths, the infection fatality rates in Illinois, New York, and Italy are substantially lower than reported.

Toward Credible Patient-centered Meta-analysis.

Meta-analysis is widely used to combine the findings of multiple disparate studies of health risks or treatment response. Meta-analysis often uses a random-effects model to express heterogeneity across studies. The model interprets a weighted average of study-specific estimates as an estimate of a mean parameter across a hypothetical population of studies. The relevance of this methodology to patient care is not evident. Clinicians need to assess risks and choose treatments for populations of patients, not for populations of studies. This article draws on econometric research on partial identification to propose principles for patient-centered meta-analysis. One specifies a patient prediction of concern and determines what each available study reveals. Given common imperfections in internal and external validity, studies typically yield credible set-valued rather than point predictions. Thus, a study may enable one to conclude that a probability of disease, or mean treatment response, lies within a range of possibilities. Patient-centered meta-analysis would combine the findings of multiple studies by computing the intersection of the set-valued predictions that they yield.

Assessing benefits, costs, and disparate racial impacts of confrontational proactive policing.

Effective policing in a democratic society must balance the sometime conflicting objectives of public safety and community trust. This paper uses a formal model of optimal policing to explore how society might reasonably resolve the tension between these two objectives as well as evaluate disparate racial impacts. We do so by considering the social benefits and costs of confrontational types of proactive policing, such as stop, question, and frisk. Three features of the optimum that are particularly relevant to policy choices are explored: (i) the cost of enforcement against the innocent, (ii) the baseline level of crime rate without confrontational enforcement, and (iii) differences across demographic groups in the optimal rate of enforcement.

Sufficient trial size to inform clinical practice.

Medical research has evolved conventions for choosing sample size in randomized clinical trials that rest on the theory of hypothesis testing. Bayesian statisticians have argued that trials should be designed to maximize subjective expected utility in settings of clinical interest. This perspective is compelling given a credible prior distribution on treatment response, but there is rarely consensus on what the subjective prior beliefs should be. We use Wald's frequentist statistical decision theory to study design of trials under ambiguity. We show that ε-optimal rules exist when trials have large enough sample size. An ε-optimal rule has expected welfare within ε of the welfare of the best treatment in every state of nature. Equivalently, it has maximum regret no larger than ε We consider trials that draw predetermined numbers of subjects at random within groups stratified by covariates and treatments. We report exact results for the special case of two treatments and binary outcomes. We give simple sufficient conditions on sample sizes that ensure existence of ε-optimal treatment rules when there are multiple treatments and outcomes are bounded. These conditions are obtained by application of Hoeffding large deviations inequalities to evaluate the performance of empirical success rules.

Reasonable patient care under uncertainty.

This paper discusses how limited ability to predict illness and treatment response may affect the welfare achieved in patient care. The discussion covers both decentralized clinical decision making and care that adheres to clinical practice guidelines. I explain why predictive ability has been limited, calling attention to questionable methodological practices in the research that supports evidence-based medicine. I summarize research on identification whose objective is to yield credible prediction of patient outcomes. Recognizing that uncertainty will continue to afflict medical decision making, I apply basic decision theory to suggest reasonable decision criteria with well-understood welfare properties. Previous research on medical decision making has largely embraced Bayesian decision theory. I summarize research studying the minimax-regret criterion, which seeks uniformly near-optimal decisions.

Diagnostic testing and treatment under ambiguity: using decision analysis to inform clinical practice.

Partial knowledge of patient health status and treatment response is a pervasive concern in medical decision making. Clinical practice guidelines (CPGs) make recommendations intended to optimize patient care, but optimization typically is infeasible with partial knowledge. Decision analysis shows that a clinician's objective, knowledge, and decision criterion should jointly determine the care he prescribes. To demonstrate, this paper studies a common scenario regarding diagnostic testing and treatment. A patient presents to a clinician, who obtains initial evidence on health status. The clinician can prescribe a treatment immediately or he can order a test yielding further evidence that may be useful in predicting treatment response. In the latter case, he prescribes a treatment after observation of the test result. I analyze this scenario in three steps. The first poses a welfare function and characterizes optimal care. The second describes partial knowledge of response to testing and treatment that might realistically be available. The third considers decision criteria. I conclude with reconsideration of clinical practice guidelines.