Surrogate endpoint metaregression: useful statistics for regulators and trialists.

OBJECTIVES: The main purpose of using a surrogate endpoint is to estimate the treatment effect on the true endpoint sooner than with a true endpoint. Based on a metaregression of historical randomized trials with surrogate and true endpoints, we discuss statistics for applying and evaluating surrogate endpoints. METHODS: We computed statistics from 2 types of linear metaregressions for trial-level data: simple random effects and novel random effects with correlations among estimated treatment effects in trials with more than 2 arms. A key statistic is the estimated intercept of the metaregression line. An intercept that is small or not statistically significant increases confidence when extrapolating to a new treatment because of consistency with a single causal pathway and invariance to labeling of treatments as controls. For a regulator applying the metaregression to a new treatment, a useful statistic is the 95% prediction interval. For a clinical trialist planning a trial of a new treatment, useful statistics are the surrogate threshold effect proportion, the sample size multiplier adjusted for dropouts, and the novel true endpoint advantage. RESULTS: We illustrate these statistics with surrogate endpoint metaregressions involving antihypertension treatment, breast cancer screening, and colorectal cancer treatment. CONCLUSION: Regulators and trialists should consider using these statistics when applying and evaluating surrogate endpoints.

COVID-19 vaccine hesitancy in inflammatory arthritis patients: serial surveys from a large longitudinal national Australian cohort.

OBJECTIVES: To determine COVID-19 vaccine hesitancy rates in inflammatory arthritis patients and identify factors associated with changing vaccine hesitancy over time. METHODS: This investigation was a prospective cohort study of inflammatory arthritis patients from community and public hospital outpatient rheumatology clinics enrolled in the Australian Rheumatology Association Database (ARAD). Two surveys were conducted, one immediately prior to (pre-pandemic) and another approximately 1 year after the start of the pandemic (follow-up). Coronavirus disease 2019 (COVID-19) vaccine hesitancy was measured at follow-up, and general vaccine hesitancy was inferred pre-pandemic; these were used to identify factors associated with fixed and changing vaccine beliefs, including sources of information and broader beliefs about medication. RESULTS: Of the 594 participants who completed both surveys, 74 (12%) were COVID-19 vaccine hesitant. This was associated with pre-pandemic beliefs about medications being harmful (P < 0.001) and overused (P = 0.002), with stronger beliefs resulting in vaccine hesitancy persistent over two time points (P = 0.008, P = 0.005). For those not vaccine hesitant pre-pandemic, the development of COVID-19 vaccine hesitancy was associated with a lower likelihood of seeking out vaccine information from health-care professionals (P < 0.001). COVID-19 vaccine hesitancy was not associated with new influenza vaccine hesitancy (P = 0.138). CONCLUSION: In this study of vaccine beliefs before and during the COVID-19 pandemic, factors associated with COVID-19 vaccine hesitancy in inflammatory arthritis patients varied, depending on vaccine attitudes immediately prior to the start of the pandemic. Fixed beliefs reflected broader views about medications, while fluid beliefs were highly influenced by whether they sought out information from health-care professionals, including rheumatologists.

Bias, coverage, and asymptotic behaviour of random effects meta-analysis: a clinically driven simulation study.

AIMS: The study of foundational features of meta-analysis is incomplete and continues to remain important. Using simulations we study bias and coverage and the asymptotic behaviour of the DerSimonian and Laird (D&L) meta-analysis with varying trial numbers and sizes, levels of risk, and extent of treatment effects. METHODS: With simulated data we model risk of untoward events in randomized controlled trials in meta-analyses. Treatment effect is expressed as relative risk reduction, with effect size estimated by the odds ratio which is then compared with the known population odds ratio. Performance is measured as bias, standardized bias and coverage, with thresholds for desirable results being prespecified. RESULTS: Bias, standardized bias, and coverage varied substantially across meta-analyses of different trial size and number, risk mean and distribution, and relative risk reduction. Although improvements were observed with increasing trial size and number, there was widespread lack of satisfactory performance. Performance using normal risk distributions was worse compared with performance using constant or narrow uniform risk distributions. Asymptotic behaviour using very large trial numbers failed to show bias that appeared to approach zero for any distribution. CONCLUSION: The D&L random effects meta-analysis method performed modestly at best. We were unable to demonstrate asymptotic normality. These results question the validity of the random effects method. The findings need replication and extension, which, if confirmed, would warn against generic use of the D&L method.

Is blood pressure reduction a valid surrogate endpoint for stroke prevention? An analysis incorporating a systematic review of randomised controlled trials, a by-trial weighted errors-in-variables regression, the surrogate threshold effect (STE) and the Biomarker-Surrogacy (BioSurrogate) Evaluation Schema (BSES).

BACKGROUND: Blood pressure is considered to be a leading example of a valid surrogate endpoint. The aims of this study were to (i) formally evaluate systolic and diastolic blood pressure reduction as a surrogate endpoint for stroke prevention and (ii) determine what blood pressure reduction would predict a stroke benefit. METHODS: We identified randomised trials of at least six months duration comparing any pharmacologic anti-hypertensive treatment to placebo or no treatment, and reporting baseline blood pressure, on-trial blood pressure, and fatal and non-fatal stroke. Trials with fewer than five strokes in at least one arm were excluded. Errors-in-variables weighted least squares regression modelled the reduction in stroke as a function of systolic blood pressure reduction and diastolic blood pressure reduction respectively. The lower 95% prediction band was used to determine the minimum systolic blood pressure and diastolic blood pressure difference, the surrogate threshold effect (STE), below which there would be no predicted stroke benefit. The STE was used to generate the surrogate threshold effect proportion (STEP), a surrogacy metric, which with the R-squared trial-level association was used to evaluate blood pressure as a surrogate endpoint for stroke using the Biomarker-Surrogacy Evaluation Schema (BSES3). RESULTS: In 18 qualifying trials representing all pharmacologic drug classes of antihypertensives, assuming a reliability coefficient of 0.9, the surrogate threshold effect for a stroke benefit was 7.1 mmHg for systolic blood pressure and 2.4 mmHg for diastolic blood pressure. The trial-level association was 0.41 and 0.64 and the STEP was 66% and 78% for systolic and diastolic blood pressure respectively. The STE and STEP were more robust to measurement error in the independent variable than R-squared trial-level associations. Using the BSES3, assuming a reliability coefficient of 0.9, systolic blood pressure was a B + grade and diastolic blood pressure was an A grade surrogate endpoint for stroke prevention. In comparison, using the same stroke data sets, no STEs could be estimated for cardiovascular (CV) mortality or all-cause mortality reduction, although the STE for CV mortality approached 25 mmHg for systolic blood pressure. CONCLUSIONS: In this report we provide the first surrogate threshold effect (STE) values for systolic and diastolic blood pressure. We suggest the STEs have face and content validity, evidenced by the inclusivity of trial populations, subject populations and pharmacologic intervention populations in their calculation. We propose that the STE and STEP metrics offer another method of evaluating the evidence supporting surrogate endpoints. We demonstrate how surrogacy evaluations are strengthened if formally evaluated within specific-context evaluation frameworks using the Biomarker- Surrogate Evaluation Schema (BSES3), and we discuss the implications of our evaluation of blood pressure on other biomarkers and patient-reported instruments in relation to surrogacy metrics and trial design.

Clinicians adopting evidence based guidelines: a case study with thromboprophylaxis.

BACKGROUND: Venous Thromboembolism (VTE) is a cause of hospital mortality and managing its morbidity is associated with significant expenditure. Uptake of evidenced based guideline recommendations intended to prevent VTE in hospital settings is sub-optimal. This study was conducted to explore clinicians' attitudes and the clinical environment in which they work to understand their reluctance to adopt VTE prophylaxis guidelines. METHODS: Between February and November 2009, 40 hospital employed doctors from 2 Australian metropolitan hospitals were interviewed in depth. Qualitative data were analysed according to thematic methodology. RESULTS: Analysis of interviews revealed that barriers to evidence based practice include i) the fragmented system of care delivery where multiple members of teams and multiple teams are responsible for each patient's care, and in the case of VTE, where everyone shares responsibility and no-one in particular is responsible; ii) the culture of practice where team practice is tailored to that of the team head, and where medicine is considered an 'art' in which guidelines should be adapted to each patient rather than applied universally. Interviewees recommend clear allocation of responsibility and reminders to counteract VTE risk assessment being overlooked. CONCLUSIONS: Senior clinicians are the key enablers for practice change. They will need to be convinced that guideline compliance adds value to their patient care. Then with the support of systems in the organisation designed to minimize the effects of care fragmentation, they will drive practice changes in their teams. We believe that evidence based practice is only possible with a coordinated program that addresses individual, cultural and organisational constraints.

A pathway and approach to biomarker validation and qualification for osteoarthritis clinical trials.

This narrative review outlines the work done in other fields with regards biomarker validation and qualification and the lessons that we may learn from this experience. Defining a universally agreed upon path for biomarker validation and qualification is urgently needed to circumvent many of the hurdles faced in OA therapeutic development irrespective of whether we are discussing biochemical markers, imaging markers or other measures. This review proposes a path that may be suitable for osteoarthritis and poses some logical next steps that will take us in this direction.

LDL-cholesterol differences predicted survival benefit in statin trials by the surrogate threshold effect (STE).

OBJECTIVE: We describe a new statistical method called the surrogate threshold effect (STE) that estimates the threshold level of a surrogate needed in a clinical trial to predict a benefit in the target clinical outcome. In this article, we apply this method to the LDL-cholesterol biomarker surrogate and survival benefit-target outcome in statin trials. STUDY DESIGN AND SETTING: We identified randomized trials comparing statin treatment to placebo treatment or no treatment and reporting all-cause and cardiovascular mortality. Trials with fewer than five all-cause deaths in at least one arm were excluded. Multiple regression modeled the reduction in all-cause and cardiovascular mortality as a function of LDL-cholesterol difference. The 95% confidence and 95% prediction bands were calculated and graphed to determine the minimum LDL-cholesterol difference (the surrogate threshold) below which there would be no predicted survival benefit. RESULTS: In 16 qualifying trials, regression analysis yielded an all-cause mortality model whose prediction bands demonstrated no overall survival gain with LDL-cholesterol difference values below 1.5 mmol/L. The cardiovascular mortality model yielded prediction bands that demonstrated no cardiovascular survival benefit with LDL-cholesterol difference values below 1.4 mmol/L. CONCLUSIONS: In a multitrial setting, the STE approach is a promising yet straightforward statistical method for evaluating the surrogate validity of biomarkers.

Imaging: the need for standardization.

The five stages in the evolution of a new method or measure are discovery (by design or inadvertent), development, testing, standardization and application. However, measures may be accepted, disseminated and used before they have been formally evaluated and standardized. This chapter describes the properties of measurement in the medical sciences and the process of standardization. It includes an example of the development and standardization of a magnetic resonance imaging rheumatoid arthritis score, and ends with a matrix that can serve as a guide for systematic appraisal and standardization of outcome measures, such as imaging outcomes. Using the matrix, one can determine the gaps in knowledge and what further evaluation is needed in one or more domains or metrics.

The Biomarker-Surrogacy Evaluation Schema: a review of the biomarker-surrogate literature and a proposal for a criterion-based, quantitative, multidimensional hierarchical levels of evidence schema for evaluating the status of biomarkers as surrogate endpoints.

There are clear advantages to using biomarkers and surrogate endpoints, but concerns about clinical and statistical validity and systematic methods to evaluate these aspects hinder their efficient application. Section 2 is a systematic, historical review of the biomarker-surrogate endpoint literature with special reference to the nomenclature, the systems of classification and statistical methods developed for their evaluation. In Section 3 an explicit, criterion-based, quantitative, multidimensional hierarchical levels of evidence schema - Biomarker-Surrogacy Evaluation Schema - is proposed to evaluate and co-ordinate the multiple dimensions (biological, epidemiological, statistical, clinical trial and risk-benefit evidence) of the biomarker clinical endpoint relationships. The schema systematically evaluates and ranks the surrogacy status of biomarkers and surrogate endpoints using defined levels of evidence. The schema incorporates the three independent domains: Study Design, Target Outcome and Statistical Evaluation. Each domain has items ranked from zero to five. An additional category called Penalties incorporates additional considerations of biological plausibility, risk-benefit and generalizability. The total score (0-15) determines the level of evidence, with Level 1 the strongest and Level 5 the weakest. The term ;surrogate' is restricted to markers attaining Levels 1 or 2 only. Surrogacy status of markers can then be directly compared within and across different areas of medicine to guide individual, trial-based or drug-development decisions. This schema would facilitate communication between clinical, researcher, regulatory, industry and consumer participants necessary for evaluation of the biomarker-surrogate-clinical endpoint relationship in their different settings.

Definitions and validation criteria for biomarkers and surrogate endpoints: development and testing of a quantitative hierarchical levels of evidence schema.

OBJECTIVE: There are clear advantages to using biomarkers and surrogate endpoints, but concerns about clinical and statistical validity and systematic methods to evaluate these aspects hinder their efficient application. Our objective was to review the literature on biomarkers and surrogates to develop a hierarchical schema that systematically evaluates and ranks the surrogacy status of biomarkers and surrogates; and to obtain feedback from stakeholders. METHODS: After a systematic search of Medline and Embase on biomarkers, surrogate (outcomes, endpoints, markers, indicators), intermediate endpoints, and leading indicators, a quantitative surrogate validation schema was developed and subsequently evaluated at a stakeholder workshop. RESULTS: The search identified several classification schema and definitions. Components of these were incorporated into a new quantitative surrogate validation level of evidence schema that evaluates biomarkers along 4 domains: Target, Study Design, Statistical Strength, and Penalties. Scores derived from 3 domains the Target that the marker is being substituted for, the Design of the (best) evidence, and the Statistical strength are additive. Penalties are then applied if there is serious counterevidence. A total score (0 to 15) determines the level of evidence, with Level 1 the strongest and Level 5 the weakest. It was proposed that the term "surrogate" be restricted to markers attaining Levels 1 or 2 only. Most stakeholders agreed that this operationalization of the National Institutes of Health definitions of biomarker, surrogate endpoint, and clinical endpoint was useful. CONCLUSION: Further development and application of this schema provides incentives and guidance for effective biomarker and surrogate endpoint research, and more efficient drug discovery, development, and approval.

Synovial membrane cytokine expression is predictive of joint damage progression in rheumatoid arthritis: a two-year prospective study (the DAMAGE study cohort).

OBJECTIVE: The primary aim of this prospective 2-year study was to explain the wide variability in joint damage progression in patients with rheumatoid arthritis (RA) from measures of pathologic changes in the synovial membrane. METHODS: Patients underwent clinical measurements and joint damage assessments by magnetic resonance imaging (MRI) and radiography at enrollment and at year 2. Synovial membrane was obtained by knee biopsy and assessed histologically by hematoxylin and eosin staining. Interleukin-1beta (IL-1beta), IL-10, IL-16, IL-17, RANKL, tumor necrosis factor alpha (TNFalpha), and interferon-gamma (IFNgamma) messenger RNA (mRNA) expression was determined by quantitative reverse transcription-polymerase chain reaction. The relationship of synovial measurements to joint damage progression was determined by multivariate analysis. RESULTS: Sixty patients were enrolled. Histologic features had no relationship to damage progression. Multivariate analysis by several different methods consistently demonstrated that synovial membrane mRNA levels of IL-1beta, TNFalpha, IL-17, and IL-10 were predictive of damage progression. IL-17 was synergistic with TNFalpha. TNFalpha and IL-17 effects were most pronounced with shorter disease duration, and IL-1beta effects were most pronounced with longer disease duration. IFNgamma was protective. These factors explained 57% of the MRI joint damage progression over 2 years. CONCLUSION: We have demonstrated for the first time in a prospective study that synovial membrane cytokine mRNA expression is predictive of joint damage progression in RA. The findings for IL-1beta and TNFalpha are consistent with results of previous clinical research, but the protective role of IFNgamma, the differing effects of disease duration, and IL-17-cytokine interactions had only been demonstrated previously by animal and in vitro research. These findings explain some of the variability of joint damage in RA and identify new targets for therapy.

Magnetic resonance imaging in rheumatoid arthritis advances and research priorities.

This article updates the work and results of the OMERACT MRI in RA Working Group as presented at the OMERACT 7 meeting in May 2004, focusing on the development of the EULAR-OMERACT rheumatoid arthritis magnetic resonance imaging reference image atlas, and on areas for future research.

The evidence for magnetic resonance imaging as an outcome measure in proof-of-concept rheumatoid arthritis studies.

Magnetic resonance imaging (MRI) has now been used extensively in cross-sectional and observational studies as well as in controlled clinical trials to assess disease activity and joint damage in rheumatoid arthritis (RA). MRI measurements or scores for erosions, bone edema, and synovitis have been developed and validated by several groups. The OMERACT criteria require that outcome measures demonstrate adequate validity, discriminative power, and feasibility if they are to be useful in clinical trials. Specific performance targets for these criteria depend on the scientific, regulatory, logistical, and financial context of the study in question. We review the extent to which MRI assessments of joint erosion, bone edema, and synovitis fulfil these criteria, particularly as they relate to proof-of-concept RA clinical trials.

Musculoskeletal ultrasound including definitions for ultrasonographic pathology.

Ultrasound (US) has great potential as an outcome in rheumatoid arthritis trials for detecting bone erosions, synovitis, tendon disease, and enthesopathy. It has a number of distinct advantages over magnetic resonance imaging, including good patient tolerability and ability to scan multiple joints in a short period of time. However, there are scarce data regarding its validity, reproducibility, and responsiveness to change, making interpretation and comparison of studies difficult. In particular, there are limited data describing standardized scanning methodology and standardized definitions of US pathologies. This article presents the first report from the OMERACT ultrasound special interest group, which has compared US against the criteria of the OMERACT filter. Also proposed for the first time are consensus US definitions for common pathological lesions seen in patients with inflammatory arthritis.

Minimal disease activity for rheumatoid arthritis: a preliminary definition.

Agreement on response criteria in rheumatoid arthritis (RA) has allowed better standardization and interpretation of clinical trial reports. With recent advances in therapy, the proportion of patients achieving a satisfactory state of minimal disease activity (MDA) is becoming a more important measure with which to compare different treatment strategies. The threshold for MDA is between high disease activity and remission and, by definition, anyone in remission will also be in MDA. True remission is still rare in RA; in addition, the American College of Rheumatology definition is difficult to apply in the context of trials. Participants at OMERACT 6 in 2002 agreed on a conceptual definition of minimal disease activity (MDA): "that state of disease activity deemed a useful target of treatment by both the patient and the physician, given current treatment possibilities and limitations." To prepare for a preliminary operational definition of MDA for use in clinical trials, we asked rheumatologists to assess 60 patient profiles describing real RA patients seen in routine clinical practice. Based on their responses, several candidate definitions for MDA were designed and discussed at the OMERACT 7 in 2004. Feedback from participants and additional on-site analyses in a cross-sectional database allowed the formulation of 2 preliminary, equivalent definitions of MDA: one based on the Disease Activity Score 28 (DAS28) index, and one based on meeting cutpoints in 5 out the 7 WHO/ILAR core set measures. Researchers applying these definitions first need to choose whether to use the DAS28 or the core set definition, because although each selects a similar proportion in a population, these are not always the same patients. In both MDA definitions, an initial decision node places all patients in MDA who have a tender joint count of 0 and a swollen joint count of 0, and an erythrocyte sedimentation rate (ESR) no greater than 10 mm. If this condition is not met: * The DAS28 definition places patients in MDA when DAS28 < or = 2.85; * The core set definition places patients in MDA when they meet 5 of 7 criteria: (1) Pain (0-10) < or = 2; (2) Swollen joint count (0-28) < or = 1; (3) Tender joint count (0-28) < or = 1; (4) Health Assessment Questionnaire (HAQ, 0-3) < or = 0.5; (5) Physician global assessment of disease activity (0-10) < or = 1.5; (6) Patient global assessment of disease activity (0-10) < or = 2; (7) ESR < or = 20. This set of 2 definitions gained approval of 73% of the attendees. These (and other) definitions will now be subject to further validation in other databases.

Generic patient self-report and investigator report instruments of therapeutic safety and tolerability.

A patient self-report instrument was designed as a patient event index that maps to a parallel investigator instrument. Event importance (a composite of severity, frequency, and duration) was reported, but attribution was not required. The patient instrument used a checklist but also allowed for spontaneous reporting for new or unusual events. The investigator instrument (also a checklist) includes all events reported by the patient, as well as events such as signs, investigations, and diagnoses that would not generally be known to the patient. Presently, both patient and investigator instruments are to be used alongside current methods of adverse event reporting in clinical trials. The patient instrument would serve as a safety/tolerability index, whereas the investigator instrument would be a fully quantifiable (appropriately weighted), standardized adverse event index. As in many methodological projects in medicine, the overriding problem was the tradeoff between validity (comprehensiveness and accuracy) and feasibility (clarity and short administration time) in instrument development. A summary of pilot studies and results of instrument reliability and validity are presented.

Challenges and progress in adverse event ascertainment and reporting in clinical trials.

Toxicity, safety, and tolerability are integral facets of patient risk/benefit decisions, yet the capacity to define, measure, and compare these aspects is underdeveloped compared to aspects of efficacy. There are many reasons for this, scientific and administrative, but all are surmountable. Probably the greatest primary obstacle is the absence of a measurement instrument designed specifically for this purpose. There are increasing calls from various stakeholders for better evidence, and therefore better ascertainment, in this area, especially in randomized trials, and for these reasons OMERACT began deliberations about these concepts in 1994. A prototype coding instrument (the Rheumatology Common Toxicity Criteria) was developed and discussed at OMERACT 5. In the 2 years before OMERACT 7, a process of concept development and iterative design and testing were conducted to develop a patient self-report and investigator-reported adverse event instruments designed for use in trials at the time of visit. The predominant workload is performed by the patient in a self-report checklist, which is then mapped by the trialist onto a medically sophisticated version. This article presents background on the process of developing a dual adverse event instrument, which was presented and critically discussed in detail at OMERACT 7.

Standardized assessment of adverse events in rheumatology clinical trials: summary of the OMERACT 7 drug safety module update.

A presentation, demonstration, and discussion of recently developed adverse event instruments were the topics for the OMERACT 7 Drug Safety Module. The module began with a plenary introducing the needs and challenges of adverse event ascertainment. It was followed by a review of module work from previous OMERACT meetings on a prototype coding instrument (Rheumatology Common Toxicity Criteria), then a brief description of the process behind the recently developed patient self-report and investigator report adverse event instruments. These current instruments are designed for use in controlled trials although they could be used in other settings. The instruments rely primarily on patient self-reporting using a checklist, which the investigator then folds into a parallel structured but more medically sophisticated instrument. In pilot testing, this innovative dual-use system has shown reliability and acceptability, while preserving validity. A "stakeholder panel" of representatives from 8 sectors followed--patient, nurse investigator, regulator, clinician scientist, industry, OMERACT, global public health/WHO, and Cochrane Collaboration--for their perspectives on the needs, challenges, and potential ways forward for adverse event ascertainment and reporting in clinical trials. At the breakout session small focus groups participated in hands-on interactive testing of one of 3 versions of the instruments, which differ in degree of comprehensiveness. Each focus group had a participatory patient with rheumatoid arthritis. At a second plenary there was group feedback by rapporteurs and presentation of results from pilot studies of iterative testing of validity, reliability, and feasibility of the instruments. During plenary discussion a frequent suggestion for improvement was to refine the process so that event ascertainment could be done entirely using the patient instrument with minimal input from the investigator at the visit, if patient-investigator agreement was high. Most found the patient checklist attractive, particularly if the patient instrument was shown to be reliable and valid. Finally, a future research agenda was discussed.