A systematic comparison of different composite measures (DAS 28, CDAI, SDAI, and Boolean approach) for determining treatment effects on low disease activity and remission in rheumatoid arthritis.

BACKGROUND: Some composite measures for determining the treatment effects of disease-modifying antirheumatic drugs on remission and low disease activity (LDA) in rheumatoid arthritis (RA) may produce misleading results if they include an acute phase reactant (APR). To inform the choice of appropriate measure, we performed a systematic comparison of treatment effects using different composite measures. METHODS: We used data generated for a systematic review of biologics in RA conducted by the Institute for Quality and Efficiency in Health Care and data from systematic reviews of newer biologics and Janus kinase (JAK) inhibitors provided by sponsors. The studies included had been conducted up to 2020 and investigated comparisons of biologics with placebo and head-to-head comparisons of biologics. Treatment effects on LDA and remission in studies investigating biologics or JAK inhibitors in RA were compared among 4 composite measures: the disease activity score 28 (DAS 28), the simplified disease activity index (SDAI), the Boolean approach (remission only), and the clinical disease activity index (CDAI)-only the latter does not include an APR. RESULTS: 49 placebo-controlled studies included 9 different biologics; 48 studies (16,233 patients) investigated LDA and 49 (16,338 patients) investigated remission. 11 active-controlled studies (5996 patients) investigated both LDA and remission and included 5 different head-to-head comparisons of biologics and 5 different comparisons (6 studies) of biologics with JAK inhibitors. Statistically significantly larger treatment effects were found for biologics or JAK inhibitors versus placebo or active control in 16% of pairwise comparisons of composite measures (27 of 168). Most of these larger effects were observed for composite measures with an APR, i.e. the DAS 28 (19 comparisons) followed by the SDAI (n = 7). Larger effects were most frequently detected in favour of interleukin (IL)-6 inhibitors and to a lesser extent for JAK inhibitors versus treatments with different modes of action. CONCLUSIONS: The use of the DAS 28 and SDAI in clinical studies may generate results favouring certain treatments based on their mode of action (e.g. IL-6 inhibitors versus other biologics). To enable unbiased comparative effectiveness research, a composite measure without an APR (i.e. the CDAI) should thus be the measure of choice.

Results on patient-reported outcomes are underreported in summaries of product characteristics for new drugs.

BACKGROUND: Summaries of product characteristics (SmPCs) are regulatory documents published upon drug approval. They should report all relevant study data and advise how to use drugs safely and effectively. Patient-reported outcomes (PROs) are increasingly used in clinical trials to incorporate the patient perspective-SmPCs should thus adequately report PROs. In Germany, new drugs undergo mandatory early benefit assessment. Pharmaceutical companies submit dossiers containing all evidence; the subsequent dossier assessments focus on patient-relevant outcomes and comprehensively report PROs. OBJECTIVE: The primary aim was to investigate to what extent PROs recorded as outcomes in clinical trials of new drugs are reported in SmPCs. METHODS: We analysed dossier assessments with randomized controlled trials (RCTs) of new drugs entering the market between 01/2014 and 07/2018 and the corresponding SmPCs, and compared PRO reporting in both document types. For this purpose, we evaluated dossier assessment characteristics (e.g. drug name, indication, disease category) and study characteristics (e.g. evaluable PROs available?). PROs were divided into symptoms and health-related quality of life (HRQoL). SmPCs were screened to identify RCTs. We conducted 3 main evaluation steps: (1) Did the RCT included in the dossier assessment contain evaluable PROs? (2) If yes, was the RCT included in the SmPC? (3) If yes, were the PROs reported in the SmPC? Results are presented descriptively. RESULTS: 88 dossier assessments including 143 RCTs on 72 drugs were considered: 109 (76.2%) RCTs included evaluable PROs, of which 89 were included in SmPCs. 38 RCTs (42.7%) investigated oncologics, 18 (20.2%) anti-infectives, and 33 (37.1%) other drugs. The RCTs considered symptoms more often than HRQoL (82 vs. 66 RCTs). In SmPCs, PROs were reported for 41 RCTs (46.1%), with a slightly higher reporting rate for RCTs considering HRQoL (43.9%) than for RCTs considering symptoms (41.5%). In oncologic indications, PROs were reported for 36.7% of RCTs considering HRQoL and 33.3% of RCTs considering symptoms. In infectious diseases, the rates were 21.4% (symptoms) and 0% (HRQoL), and for other diseases about 60% (symptoms) to 70% (HRQoL). CONCLUSION: Even though a large amount of PRO data on new drugs is available from clinical trials included in SmPCs, the corresponding results are underreported.

Guideline appraisal with AGREE II: online survey of the potential influence of AGREE II items on overall assessment of guideline quality and recommendation for use.

BACKGROUND: The AGREE II instrument is the most commonly used guideline appraisal tool. It includes 23 appraisal criteria (items) organized within six domains. AGREE II also includes two overall assessments (overall guideline quality, recommendation for use). Our aim was to investigate how strongly the 23 AGREE II items influence the two overall assessments. METHODS: An online survey of authors of publications on guideline appraisals with AGREE II and guideline users from a German scientific network was conducted between 10th February 2015 and 30th March 2015. Participants were asked to rate the influence of the AGREE II items on a Likert scale (0 = no influence to 5 = very strong influence). The frequencies of responses and their dispersion were presented descriptively. RESULTS: Fifty-eight of the 376 persons contacted (15.4%) participated in the survey and the data of the 51 respondents with prior knowledge of AGREE II were analysed. Items 7-12 of Domain 3 (rigour of development) and both items of Domain 6 (editorial independence) had the strongest influence on the two overall assessments. In addition, Items 15-17 (clarity of presentation) had a strong influence on the recommendation for use. Great variations were shown for the other items. The main limitation of the survey is the low response rate. CONCLUSIONS: In guideline appraisals using AGREE II, items representing rigour of guideline development and editorial independence seem to have the strongest influence on the two overall assessments. In order to ensure a transparent approach to reaching the overall assessments, we suggest the inclusion of a recommendation in the AGREE II user manual on how to consider item and domain scores. For instance, the manual could include an a-priori weighting of those items and domains that should have the strongest influence on the two overall assessments. The relevance of these assessments within AGREE II could thereby be further specified.

Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports.

BACKGROUND: When a new drug becomes available, patients and doctors require information on its benefits and harms. In 2011, Germany introduced the early benefit assessment of new drugs through the act on the reform of the market for medicinal products (AMNOG). At market entry, the pharmaceutical company responsible must submit a standardised dossier containing all available evidence of the drug's added benefit over an appropriate comparator treatment. The added benefit is mainly determined using patient relevant outcomes. The "dossier assessment" is generally performed by the Institute for Quality and Efficiency in Health Care (IQWiG) and then published online. It contains all relevant study information, including data from unpublished clinical study reports contained in the dossiers. The dossier assessment refers to the patient population for which the new drug is approved according to the summary of product characteristics. This patient population may comprise either the total populations investigated in the studies submitted to regulatory authorities in the drug approval process, or the specific subpopulations defined in the summary of product characteristics ("approved subpopulations"). OBJECTIVE: To determine the information gain from AMNOG documents compared with non-AMNOG documents for methods and results of studies available at market entry of new drugs. AMNOG documents comprise dossier assessments done by IQWiG and publicly available modules of company dossiers; non-AMNOG documents comprise conventional, publicly available sources-that is, European public assessment reports, journal publications, and registry reports. The analysis focused on the approved patient populations. DESIGN: Retrospective analysis. DATA SOURCES: All dossier assessments conducted by IQWiG between 1 January 2011 and 28 February 2013 in which the dossiers contained suitable studies allowing for a full early benefit assessment. We also considered all European public assessment reports, journal publications, and registry reports referring to these studies and included in the dossiers. DATA ANALYSIS: We assessed reporting quality for each study and each available document for eight methods and 11 results items (three baseline characteristics and eight patient relevant outcomes), and dichotomised them as "completely reported" or "incompletely reported (including items not reported at all)." For each document type we calculated the proportion of items with complete reporting for methods and results, for each item and overall, and compared the findings.Results 15 out of 27 dossiers were eligible for inclusion and contained 22 studies. The 15 dossier assessments contained 28 individual assessments of 15 total study populations and 13 approved subpopulations. European public assessment reports were available for all drugs. Journal publications were available for 14 out of 15 drugs and 21 out of 22 studies. A registry report in ClinicalTrials.gov was available for all drugs and studies; however, only 11 contained results. In the analysis of total study populations, the AMNOG documents reached the highest grade of completeness, with about 90% of methods and results items completely reported. In non-AMNOG documents, the rate was 75% for methods and 52% for results items; journal publications achieved the best rates, followed by European public assessment reports and registry reports. The analysis of approved subpopulations showed poorer complete reporting of results items, particularly in non-AMNOG documents (non-AMNOG versus AMNOG: 11% v 71% for overall results items and 5% v 70% for patient relevant outcomes). The main limitation of our analysis is the small sample size. CONCLUSION: Conventional, publicly available sources provide insufficient information on new drugs, especially on patient relevant outcomes in approved subpopulations. This type of information is largely available in AMNOG documents, albeit only partly in English. The AMNOG approach could be used internationally to develop a comprehensive publication model for clinical studies and thus represents a key open access measure.

[Patient-relevant outcomes and surrogates in the early benefit assessment of drugs: first experiences]. / Patientenrelevante Endpunkte und Surrogate in der frühen Nutzenbewertung von Arzneimitteln: erste Erfahrungen.

The Act on the Reform of the Market for Medicinal Products (AMNOG) became effective in Germany on January 1, 2011. Since then, the assessment of the added benefit of new drugs versus a therapeutic standard on the basis of dossiers submitted by pharmaceutical companies has been required by law. The Federal Joint Committee (G-BA) generally commissions the Institute for Quality and Efficiency in Health Care (IQWiG) with this task. The added benefit is primarily to be demonstrated on the basis of patient-relevant outcomes. The aim of this paper is to describe the feasibility of the early benefit assessment on the basis of patient-relevant outcomes by systematically characterising the outcomes available in company dossiers and comparing the companies' and IQWiG's evaluations regarding patient relevance and surrogate validity. Dossier assessments published between October 2011 and June 2012 were used for this purpose. The outcomes available and the respective evaluations were extracted and compared. 12 out of 22 submitted dossiers contained sufficient data to assess outcomes; all 12 assessable dossiers provided data on patient-relevant outcomes. Data on mortality and adverse events were available in all dossiers, except that one dossier did not contain adverse event data on the relevant subpopulation. In contrast, data on morbidity and health-related quality of life were available in 8 and 7 dossiers, respectively. Of a total of 214 outcomes extracted by IQWiG, 124 patient-relevant and 3 surrogate outcomes were included in IQWiG's assessment (companies: a total of 183 outcomes included, of which 172 were patient-relevant and 11 were surrogates). The first experiences with AMNOG have shown that in principle an early benefit assessment of drugs based on patient-relevant outcomes is feasible. The companies' and IQWiG's evaluations regarding patient relevance and surrogate validity of outcomes partly deviated from each other. By increasingly considering patient-relevant outcomes in approval studies, pharmaceutical companies can create the necessary data basis for the early benefit assessment.

Impact of inclusion of industry trial results registries as an information source for systematic reviews.

BACKGROUND: Clinical trial results registries may contain relevant unpublished information. Our main aim was to investigate the potential impact of the inclusion of reports from industry results registries on systematic reviews (SRs). METHODS: We identified a sample of 150 eligible SRs in PubMed via backward selection. Eligible SRs investigated randomized controlled trials of drugs and included at least 2 bibliographic databases (original search date: 11/2009). We checked whether results registries of manufacturers and/or industry associations had also been searched. If not, we searched these registries for additional trials not considered in the SRs, as well as for additional data on trials already considered. We reanalysed the primary outcome and harm outcomes reported in the SRs and determined whether results had changed. A "change" was defined as either a new relevant result or a change in the statistical significance of an existing result. We performed a search update in 8/2013 and identified a sample of 20 eligible SRs to determine whether mandatory results registration from 9/2008 onwards in the public trial and results registry ClinicalTrials.gov had led to its inclusion as a standard information source in SRs, and whether the inclusion rate of industry results registries had changed. RESULTS: 133 of the 150 SRs (89%) in the original analysis did not search industry results registries. For 23 (17%) of these SRs we found 25 additional trials and additional data on 31 trials already included in the SRs. This additional information was found for more than twice as many SRs of drugs approved from 2000 as approved beforehand. The inclusion of the additional trials and data yielded changes in existing results or the addition of new results for 6 of the 23 SRs. Of the 20 SRs retrieved in the search update, 8 considered ClinicalTrials.gov or a meta-registry linking to ClinicalTrials.gov, and 1 considered an industry results registry. CONCLUSION: The inclusion of industry and public results registries as an information source in SRs is still insufficient and may result in publication and outcome reporting bias. In addition to an essential search in ClinicalTrials.gov, authors of SRs should consider searching industry results registries.

Early benefit assessment of new drugs in Germany - results from 2011 to 2012.

Rising drug costs in Germany led to the Act on the Reform of the Market for Medicinal Products (AMNOG) in January 2011. For new drugs, pharmaceutical companies have to submit dossiers containing all available evidence to demonstrate an added benefit versus an appropriate comparator therapy. The Federal Joint Committee (G-BA), the main decision-making body of the statutory healthcare system, is responsible for the overall procedure of "early benefit assessment". The Institute for Quality and Efficiency in Health Care (IQWiG) largely conducts the dossier assessments, which inform decisions by the G-BA on added benefit and support price negotiations. Of the 25 dossiers (excluding orphan drugs) assessed until 31 December 2012, 14 contained sufficient data from randomized active-controlled trials investigating patient-relevant outcomes or at least acceptable surrogates; 11 contained insufficient data. The most common indications were oncology (6) and viral infections (4). For the 14 drugs assessed, the extent of added benefit was rated as minor, considerable, and non-quantifiable in 3, 8, and 2 cases; the remaining drug showed no added benefit. Despite some shortcomings, for the first time it has been possible in Germany to implement a systematic procedure for assessing new drugs at market entry, thus providing support for price negotiations and informed decision-making for patients, clinicians and policy makers.

Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data.

BACKGROUND: Access to unpublished clinical study reports (CSRs) is currently being discussed as a means to allow unbiased evaluation of clinical research. The Institute for Quality and Efficiency in Health Care (IQWiG) routinely requests CSRs from manufacturers for its drug assessments. Our objective was to determine the information gain from CSRs compared to publicly available sources (journal publications and registry reports) for patient-relevant outcomes included in IQWiG health technology assessments (HTAs) of drugs. METHODS AND FINDINGS: We used a sample of 101 trials with full CSRs received for 16 HTAs of drugs completed by IQWiG between 15 January 2006 and 14 February 2011, and analyzed the CSRs and the publicly available sources of these trials. For each document type we assessed the completeness of information on all patient-relevant outcomes included in the HTAs (benefit outcomes, e.g., mortality, symptoms, and health-related quality of life; harm outcomes, e.g., adverse events). We dichotomized the outcomes as "completely reported" or "incompletely reported." For each document type, we calculated the proportion of outcomes with complete information per outcome category and overall. We analyzed 101 trials with CSRs; 86 had at least one publicly available source, 65 at least one journal publication, and 50 a registry report. The trials included 1,080 patient-relevant outcomes. The CSRs provided complete information on a considerably higher proportion of outcomes (86%) than the combined publicly available sources (39%). With the exception of health-related quality of life (57%), CSRs provided complete information on 78% to 100% of the various benefit outcomes (combined publicly available sources: 20% to 53%). CSRs also provided considerably more information on harms. The differences in completeness of information for patient-relevant outcomes between CSRs and journal publications or registry reports (or a combination of both) were statistically significant for all types of outcomes. The main limitation of our study is that our sample is not representative because only CSRs provided voluntarily by pharmaceutical companies upon request could be assessed. In addition, the sample covered only a limited number of therapeutic areas and was restricted to randomized controlled trials investigating drugs. CONCLUSIONS: In contrast to CSRs, publicly available sources provide insufficient information on patient-relevant outcomes of clinical trials. CSRs should therefore be made publicly available. Please see later in the article for the Editors' Summary.

Access to regulatory data from the European Medicines Agency: the times they are a-changing.

Systematic reviewers are increasingly trying to obtain regulatory clinical study reports (CSRs) to correct for publication bias. For instance, our organization, the Institute for Quality and Efficiency in Health Care, routinely asks drug manufacturers to provide full CSRs of studies considered in health technology assessments. However, since cooperation is voluntary, CSRs are available only for a subset of studies analysed. In the case of the inhaled insulin Exubera, the manufacturer refused to cooperate and in 2007 we asked the European Medicines Agency (EMA) to provide the relevant CSRs, but EMA denied access. Other researchers have reported similar experiences.In 2010 EMA introduced a new policy on access to regulatory documents, including CSRs, and has also undertaken further steps. The new policy has already borne fruit: in 2011, by providing additional sections of relevant CSRs, EMA made an important contribution to a review of oseltamivir (Tamiflu).Unfortunately, speedy implementation of the new policy may be endangered. We define a CSR following the International Conference on Harmonisation (ICH) E3 guideline. Although this guideline requires individual patient data listings, it does not necessarily require that these listings be made available in a computer-readable format, as proposed by some regulators from EMA and other agencies. However, access to raw data in a computer-readable format poses additional problems; merging this issue with that of access to CSRs could hamper the relatively simple implementation of the EMA policy. Moreover, EMA plans to release CSRs only on request; we suggest making these documents routinely available on the EMA website.Public access to regulatory data also carries potential risks. In our view, the issue of patient confidentiality has been largely resolved by current European legislation. The risk of other problems, such as conflicts of interest (CoIs) of independent researchers or quality issues can be reduced by transparency measures, such as the implementation of processes to evaluate CoIs and the publication of methods and protocols.In conclusion, regulatory data are an indispensable source for systematic reviews. Because of EMA's policy change, a milestone for data transparency in clinical research is within reach; let's hope it is not unnecessarily delayed.

Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications.

OBJECTIVE: To investigate to what extent three types of documents for reporting clinical trials provide sufficient information for trial evaluation. DESIGN: Retrospective analysis DATA SOURCES: Primary studies and corresponding documents (registry reports, clinical study reports, journal publications) from 16 health technology assessments of drugs conducted by the German Institute for Quality and Efficiency in Health Care between 2006 and February 2011. Data analysis We assessed reporting quality for each study and each available document for six items on methods and six on outcomes, and dichotomised them as "completely reported" or "incompletely reported." For each document type, we calculated the proportion of studies with complete reporting for methods and outcomes, per item and overall, and compared the findings. RESULTS: We identified 268 studies. Publications, study reports and registry reports were available for 192 (72%), 101 (38%), and 78 (29%) studies, respectively. Reporting quality was highest in study reports, which overall provided complete information for 90% of items (1086/1212). Registry reports provided more complete information on outcomes than on methods (overall 330/468 (71%) v 147/468 (31%)); the same applied to publications (594/1152 (52%) v 458/1152 (40%)). In the matched pairs analysis, reporting quality was poorer in registry reports than in study reports for overall methods and outcomes (P<0.001 in each case). Compared with publications, reporting quality was poorer in registry reports for overall methods (P<0.001), but better for outcomes (P=0.005). CONCLUSION: Registry reports and publications insufficiently report clinical trials but may supplement each other. Measures to improve reporting include the mandatory worldwide implementation of adequate standards for results registration.