Envisioning an improved research strategy for detecting moderators/mediators in intervention studies: A commentary on McClure et al.

The goal of moderator/mediator research in treatment evaluation is to provide guidance to clinicians to choose the best treatment for each patient with a disorder (moderators), and to advise on its optimal protocol or implementation (mediators): personalized/precision medicine. McClure et al. report a systematic review of studies addressing moderators/mediators of the treatment effect of digital interventions for eating disorders, finding no robust moderators or mediators. They attribute this failure to methodological problems, an assessment with which I concur. The focus of this discussion is to clarify which methodological approaches are not likely to be successful, and to envision a research strategy encompassing both hypothesis-generating (exploratory) and hypothesis-testing approaches likely to produce better results not only for eating disorders, but also for all medical treatments.

Moderators of remission with interpersonal counselling or drug treatment in primary care patients with depression: randomised controlled trial.

BACKGROUND: Despite depressive disorders being very common there has been little research to guide primary care physicians on the choice of treatment for patients with mild to moderate depression. AIMS: To evaluate the efficacy of interpersonal counselling compared with selective serotonin reuptake inhibitors (SSRIs), in primary care attenders with major depression and to identify moderators of treatment outcome. METHOD: A randomised controlled trial in nine centres (DEPICS, Australian New Zealand Clinical Trials Registry number: ACTRN12608000479303). The primary outcome was remission of the depressive episode (defined as a Hamilton Rating Scale for Depression score ≤7 at 2 months). Daily functioning was assessed using the Work and Social Adjustment Scale. Logistic regression models were used to identify moderators of treatment outcome. RESULTS: The percentage of patients who achieved remission at 2 months was significantly higher in the interpersonal counselling group compared with the SSRI group (58.7% v. 45.1%, P = 0.021). Five moderators of treatment outcome were found: depression severity, functional impairment, anxiety comorbidity, previous depressive episodes and smoking habit. CONCLUSIONS: We identified some patient characteristics predicting a differential outcome with pharmacological and psychological interventions. Should our results be confirmed in future studies, these characteristics will help clinicians to define criteria for first-line treatment of depression targeted to patients' characteristics.

The reliability of clinical diagnoses: state of the art.

Reliability of clinical diagnosis is essential for good clinical decision making as well as productive clinical research. The current review emphasizes the distinction between a disorder and a diagnosis and between validity and reliability of diagnoses, and the relationships that exist between them. What is crucial is that reliable diagnoses are essential to establishing valid diagnoses. The present review discusses the theoretical background underlying the evaluation of diagnoses, possible designs of reliability studies, estimation of the reliability coefficient, the standards for assessment of reliability, and strategies for improving reliability without compromising validity.

Toward sound objective evaluation of clinical measures.

The quality of all clinical decision-making, as well as power and precision in clinical research results, depends fundamentally on the quality of the measures used. Yet evaluations of the quality of clinical measures likely to be used either in clinic or research applications are difficult to execute or to critique because the criteria for judging such studies are so ill-defined. Here a set of guidelines is proposed, modeled on CONSORT guidelines for randomized clinical trials, first defining the terms often inconsistently used in the research literature and then identifying certain errors that seem to recur in evaluation studies.

Discovering, comparing, and combining moderators of treatment on outcome after randomized clinical trials: a parametric approach.

No one treatment is likely to affect all patients with a disorder in the same way. A treatment highly effective for some may be ineffective or even harmful for others. Statistically significant or not, the effect sizes of many treatments tend to be small. Consequently, emphasis in clinical research is gradually shifting (1) to increased focus on effect sizes and (2) to discovery and documentation of moderators of treatment effect on outcome in randomized clinical trials, that is, personalized medicine, in which individual differences between patients are explicitly acknowledged. How to test a null hypothesis of moderation of treatment outcome is reasonably well known. The focus here is on how, under parametric assumptions, to define the strength of moderation, that is, a moderator effect size, either for scientific purposes or for assessment of clinical significance, in order to compare moderators and choose among them and to develop a composite moderator, which might more strongly moderate the effect of a treatment on outcome than any single moderator that might ultimately provide guidance for clinicians as to whom to prescribe what treatment.

DSM categories and dimensions in clinical and research contexts.

An enhancement to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-V) is currently under consideration, one that would enhance both the reliability and validity of the Diagnostic and Statistical Manual (DSM) diagnoses: the addition of a dimensional adjunct to each of the traditional categorical diagnoses of the DSM. We first review the history and context of this proposal and define the concepts on which this dimensional proposal is based. The advantages of dimensional measures over categorical measures have long been known, but we here illustrate what is known with a theoretical and a practical demonstration of the potential effects of this addition. Possible objections to the proposal are discussed, concluding with some general criteria for implementing this proposal.

Lifetime prevalence and pseudocomorbidity in psychiatric research.

CONTEXT: Comorbidity is the rule rather than the exception with psychiatric disorders and is consequently of great interest to both researchers and clinicians. However, many studies of psychiatric comorbidity have been based on lifetime prevalence with mixed-age samples, a practice that (1) biases the assessment of epidemiologic comorbidity and (2) creates the appearance of comorbidity even when disorders are randomly associated. This bias is what we refer to as pseudocomorbidity. OBJECTIVES: To clarify the source of the problem and to discuss strategies that might be adopted to deal hereafter with lifetime prevalence data. METHODS: A simulated example is presented to show that even when there is only random association between disorders, there will appear to be nonrandom comorbidity when lifetime prevalence is used with mixed-age samples. An actual example relating psychosis to phobia is presented to show the bias that can result and to illustrate one way of dealing with lifetime prevalence data. CONCLUSIONS: Use of lifetime prevalence with mixed-age samples, used almost exclusively in psychiatric research, generates problematic results, especially when used for assessment of comorbidity, and should be viewed with some skepticism. Hereafter, we recommend that any future use of lifetime prevalence should require determination of the age of onset, even if only by retrospective report. Comorbidity then should be reported by age.

Caution regarding the use of pilot studies to guide power calculations for study proposals.

Clinical researchers often propose (or review committees demand) pilot studies to determine whether a study is worth performing and to guide power calculations. The most likely outcomes are that (1) studies worth performing are aborted and (2) studies that are not aborted are underpowered. There are many excellent reasons for performing pilot studies. The argument herein is not meant to discourage clinical researchers from performing pilot studies (or review committees from requiring them) but simply to caution against their use for the objective of guiding power calculations.

Developing the diagnostic and statistical manual V: what will "statistical" mean in DSM-V?

In February of 2004, the American Psychiatric Institute for Research and Education (APIRE) hosted a Launch and Methodology Conference to discuss the role statistics might play in the eventual revision of the Fourth Edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) and the Ninth Edition of the International Classification of Diseases (ICD9). The conference consisted of talks on specific topics by statisticians and epidemiologists from North America and Great Britain, followed by group discussion by experts in nosology and psychopathology. We report here on the development of specific themes related to the future interaction between statisticians and nosologists in DSM-V development that arose as a result of that meeting. The themes are related to (1) the nature of the statistician/nosologist interaction; (2) specific areas of concern in that interaction, and (3) the use of novel and complex statistical methods to challenge and inspire new avenues of thinking among nosologists.

Size of treatment effects and their importance to clinical research and practice.

In randomized clinical trails (RCTs), effect sizes seen in earlier studies guide both the choice of the effect size that sets the appropriate threshold of clinical significance and the rationale to believe that the true effect size is above that threshold worth pursuing in an RCT. That threshold is used to determine the necessary sample size for the proposed RCT. Once the RCT is done, the data generated are used to estimate the true effect size and its confidence interval. Clinical significance is assessed by comparing the true effect size to the threshold effect size. In subsequent meta-analysis, this effect size is combined with others, ultimately to determine whether treatment (T) is clinically significantly better than control (C). Thus, effect sizes play an important role both in designing RCTs and in interpreting their results; but specifically which effect size? We review the principles of statistical significance, power, and meta-analysis, and commonly used effect sizes. The commonly used effect sizes are limited in conveying clinical significance. We recommend three equivalent effect sizes: number needed to treat, area under the receiver operating characteristic curve comparing T and C responses, and success rate difference, chosen specifically to convey clinical significance.

Factors associated with psychiatric hospitalization of individuals diagnosed with dementia and comorbid bipolar disorder.

The objective was to determine risk factors of psychiatric hospitalization among a Veterans Administration database of patients with dementia and comorbid bipolar disorder (D+BD). Patients with D+BD had a greater prevalence of psychiatric hospitalization (28% vs 4%). The strongest predictor of psychiatric hospitalization was the presence of an alcohol use disorder (51% risk); patients without alcohol use disorders but under the age of 70 had the next highest risk (33% risk). However, patients with an alcohol use disorder had shorter psychiatric hospitalizations than those without. Compared with patients without BD, D+BD patients were more likely to have alcohol use disorders (15% vs 3%) and any other substance use problem (10% vs 1%). In patients diagnosed with dementia and bipolar disorder, the strongest risk factor for psychiatric hospitalization was an alcohol abuse disorder. These findings suggest that disorders with increased frequency in BD affect the course of dementia.

Correlation coefficients in medical research: from product moment correlation to the odds ratio.

OBJECTIVE: Presentation of effect sizes that can be interpreted in terms of clinical or practical significance is currently urged whenever statistical significance (a 'p-value') is reported in research journals. However, which effect size and how to interpret it are not yet clearly delineated. The present focus is on effect sizes indicating strength of correlation, that is, effect sizes that describe the strength of monotonic association between two random variables X and Y in a population. METHODS: A logical structure of measures of association is traced, showing the interrelationships among the many measures of association. Advantages and disadvantages of each are discussed. CONCLUSIONS: Suggestions are made for the future use of measures of association in research to facilitate considerations of clinical significance, emphasizing distribution-free effect sizes such as the Spearman correlation coefficient and Kendall's coefficient of concordance for ordinal versus ordinal associations, weighted and intraclass kappa for binary versus binary associations and risk difference (RD) for binary versus ordinal association.

Messages for Clinicians: Moderators and Mediators of Treatment Outcome in Randomized Clinical Trials.

Many problems in randomized clinical trial design, execution, analysis, presentation and interpretation stem in part from an inadequate understanding of the roles of moderators and mediators of treatment outcome. As a result, 1) the results of clinical research are slow to have an impact on clinical decision making and thus to benefit patients; 2) it is difficult for clinicians or patients to apply randomized clinical trial results comparing two treatments (treatment versus control); 3) when such trials are conducted at various sites, the results often do not replicate; 4) when the results influence clinical decision making, the results clinicians obtain do not match what researchers report; and 5) the treatment effects comparing treatment and control conditions, particularly for psychiatric treatments, often seem trivial. In this review article, the author reviews and integrates the methodological literature concerning dealing with covariates in trials to emphasize their impact on clinical decision making. The goal of trials should ultimately be to establish who should get the treatment condition rather than the control condition (moderators) and to determine how to obtain the best outcomes with whatever is the preferred treatment (mediators). The author makes recommendations to clinicians as to which trials might best be ignored and which carefully considered, and urges clinical researchers to focus on studies best designed to reduce the burden of mental illness on patients.

Mediators and moderators of treatment effects in randomized clinical trials.

Randomized clinical trials (RCTs) not only are the gold standard for evaluating the efficacy and effectiveness of psychiatric treatments but also can be valuable in revealing moderators and mediators of therapeutic change. Conceptually, moderators identify on whom and under what circumstances treatments have different effects. Mediators identify why and how treatments have effects. We describe an analytic framework to identify and distinguish between moderators and mediators in RCTs when outcomes are measured dimensionally. Rapid progress in identifying the most effective treatments and understanding on whom treatments work and do not work and why treatments work or do not work depends on efforts to identify moderators and mediators of treatment outcome. We recommend that RCTs routinely include and report such analyses.

Are certain multicenter randomized clinical trial structures misleading clinical and policy decisions?

Multicenter studies involving randomized clinical trials (RCTs) may have different structures. We discuss four general types. The first two, an "ideal" multicenter RCT and decentralized multicenter collaborative RCTs, we feel are, in different circumstances, highly recommended approaches. The other two, the multicenter RCT that ignores site differences and centralized multicenter collaborative RCTs, we argue, are not only not cost-effective but may also produce misleading results, thus impeding scientific progress and possibly putting patients at unnecessary risk.

A simple effect size indicator for two-group comparisons? A comment on r equivalent.

R. Rosenthal and D. B. Rubin (2003) proposed an effect size, r equivalent, to be used when (a) only sample size and p values are known for a study, (b) there are no generally accepted effect size indicators, or (c) sample sizes are so small or the data so non-normal that the directly computed effect sizes would be more misleading than the simple effect size. The limitations of their proposal, however, are many, and much more serious than the authors suggested, and should be carefully considered before this effect size is applied, as well as in developing other effect sizes using similar methods.