Consistency of genome-wide associations across major ancestral groups.

It is not well known whether genetic markers identified through genome-wide association studies (GWAS) confer similar or different risks across people of different ancestry. We screened a regularly updated catalog of all published GWAS curated at the NHGRI website for GWAS-identified associations that had reached genome-wide significance (p ≤ 5 × 10(-8)) in at least one major ancestry group (European, Asian, African) and for which replication data were available for comparison in at least two different major ancestry groups. These groups were compared for the correlation between and differences in risk allele frequencies and genetic effects' estimates. Data on 108 eligible GWAS-identified associations with a total of 900 datasets (European, n = 624; Asian, n = 217; African, n = 60) were analyzed. Risk-allele frequencies were modestly correlated between ancestry groups, with >10% absolute differences in 75-89% of the three pairwise comparisons of ancestry groups. Genetic effect (odds ratio) point estimates between ancestry groups correlated modestly (pairwise comparisons' correlation coefficients: 0.20-0.33) and point estimates of risks were opposite in direction or differed more than twofold in 57%, 79%, and 89% of the European versus Asian, European versus African, and Asian versus African comparisons, respectively. The modest correlations, differing risk estimates, and considerable between-association heterogeneity suggest that differential ancestral effects can be anticipated and genomic risk markers may need separate further evaluation in different ancestry groups.

Assessment of claims of improved prediction beyond the Framingham risk score.

CONTEXT: With heightened interest in predictive medicine, many studies try to document information that can improve prediction of major clinical outcomes. OBJECTIVE: To evaluate the reported design and analysis of studies that examined whether additional predictors improve predictive performance when added to the Framingham risk score (FRS), one of the most widely validated and cited clinical prediction scores. STUDY SELECTION: Two independent investigators searched 1908 articles citing the article that described the FRS in 1998 until September 2009 through the ISI Web of Knowledge database. Articles were eligible if they included any analyses comparing the predictive performance of the FRS vs the FRS plus some additional predictor for a prospectively assessed outcome. Data Analyses We recorded information on FRS calculation, modeling of additional predictors, outcomes assessed, population evaluated, subgroup analysis documentation, and flaws in the methods that may have affected the reported improvements in predictive ability. We also evaluated the correlation of reported design and analysis features with the predictive model discrimination and improvements with the additional predictors. RESULTS: We evaluated 79 eligible articles. Forty-nine studies (62%) did not calculate the FRS as it has been proposed, 15 (19%) modeled the additional predictor in more than 1 way and presented only the best-fit or area-under-the-curve (AUC) results for only 1 model, 41 (52%) did not examine the original outcome that the FRS was developed for, 33 (42%) studied a population different from what the FRS was intended for, and 25 (32%) claimed improved prediction in 1 subgroup but only 7 (9%) formally tested subgroup differences. Evaluation of independence in multivariable regressions, discrimination in AUC, calibration, and reclassification were reported in 77, 36, 7, and 7 studies, respectively, but these methods were adequately documented in only 60, 13, 4, and 2 studies, respectively. Overall, 63 studies (80%) claimed some improved prediction. Increase in AUC was larger when the predictive performance of the FRS was lower (rho = -0.57, P < .001). Increase in AUC was significantly larger when evaluation of independence in multivariable regression or discrimination in AUC analysis was not adequately documented and when the additional predictor had been modeled in more than 1 way and only 1 model was reported for AUC. CONCLUSION: The majority of examined studies claimed that they found factors that could offer additional predictive value beyond what the FRS could achieve; however, most had flaws in their design, analyses, and reporting that cast some doubt on the reliability of the claims for improved prediction.

Selection in reported epidemiological risks: an empirical assessment.

BACKGROUND: Epidemiological studies may be subject to selective reporting, but empirical evidence thereof is limited. We empirically evaluated the extent of selection of significant results and large effect sizes in a large sample of recent articles. METHODS AND FINDINGS: We evaluated 389 articles of epidemiological studies that reported, in their respective abstracts, at least one relative risk for a continuous risk factor in contrasts based on median, tertile, quartile, or quintile categorizations. We examined the proportion and correlates of reporting statistically significant and nonsignificant results in the abstract and whether the magnitude of the relative risks presented (coined to be consistently > or =1.00) differs depending on the type of contrast used for the risk factor. In 342 articles (87.9%), > or =1 statistically significant relative risk was reported in the abstract, while only 169 articles (43.4%) reported > or =1 statistically nonsignificant relative risk in the abstract. Reporting of statistically significant results was more common with structured abstracts, and was less common in US-based studies and in cancer outcomes. Among 50 randomly selected articles in which the full text was examined, a median of nine (interquartile range 5-16) statistically significant and six (interquartile range 3-16) statistically nonsignificant relative risks were presented (p = 0.25). Paradoxically, the smallest presented relative risks were based on the contrasts of extreme quintiles; on average, the relative risk magnitude was 1.41-, 1.42-, and 1.36-fold larger in contrasts of extreme quartiles, extreme tertiles, and above-versus-below median values, respectively (p < 0.001). CONCLUSIONS: Published epidemiological investigations almost universally highlight significant associations between risk factors and outcomes. For continuous risk factors, investigators selectively present contrasts between more extreme groups, when relative risks are inherently lower.

International ranking systems for universities and institutions: a critical appraisal.

BACKGROUND: Ranking of universities and institutions has attracted wide attention recently. Several systems have been proposed that attempt to rank academic institutions worldwide. METHODS: We review the two most publicly visible ranking systems, the Shanghai Jiao Tong University 'Academic Ranking of World Universities' and the Times Higher Education Supplement 'World University Rankings' and also briefly review other ranking systems that use different criteria. We assess the construct validity for educational and research excellence and the measurement validity of each of the proposed ranking criteria, and try to identify generic challenges in international ranking of universities and institutions. RESULTS: None of the reviewed criteria for international ranking seems to have very good construct validity for both educational and research excellence, and most don't have very good construct validity even for just one of these two aspects of excellence. Measurement error for many items is also considerable or is not possible to determine due to lack of publication of the relevant data and methodology details. The concordance between the 2006 rankings by Shanghai and Times is modest at best, with only 133 universities shared in their top 200 lists. The examination of the existing international ranking systems suggests that generic challenges include adjustment for institutional size, definition of institutions, implications of average measurements of excellence versus measurements of extremes, adjustments for scientific field, time frame of measurement and allocation of credit for excellence. CONCLUSION: Naïve lists of international institutional rankings that do not address these fundamental challenges with transparent methods are misleading and should be abandoned. We make some suggestions on how focused and standardized evaluations of excellence could be improved and placed in proper context.

Use of reclassification for assessment of improved prediction: an empirical evaluation.

BACKGROUND: An increasing number of studies evaluate the ability of predictors to change risk stratification and alter medical decisions, i.e. reclassification performance. We examined the reported design and analysis of recent studies of reclassification and the robustness of their claims for improved reclassification. METHODS: Two independent investigators searched PubMed and citations to the article that introduced the currently most popular reclassification metric (net reclassification index, NRI) to identify studies performing reclassification analysis (January 2006-January 2010). We focused on articles that included any analyses comparing the performance of a baseline predictive model vs the baseline model plus some additional predictor for a prospectively assessed outcome. We recorded information on the baseline model used, outcomes assessed, choice of risk thresholds and features of reclassification analyses. RESULTS: Of 58 baseline models used in 51 eligible papers, only 14 (24%) were previously described, used as described and had same outcomes as originally intended. Calibration was examined in 53% of the studies. Sixteen studies (31%) provided a reference for the choice of risk thresholds and only six used the previously proposed categories or justified the use of alternative thresholds. Only 14 studies (27%) stated that the chosen risk thresholds had different therapeutic intervention implications. NRI was calculated in 38 studies and was smaller in studies with adequately referenced or justified risk thresholds vs others (P < 0.0001). CONCLUSIONS: Reclassification studies would benefit from more rigorous methodological standards; otherwise claims for improved reclassification may remain spurious.

The elderly were under-represented in osteoarthritis clinical trials.

OBJECTIVES: Osteoarthritis is the most common disease affecting joints in the elderly. We aimed to evaluate if elderly patients are properly represented in clinical trials of diverse osteoarthritis interventions. STUDY DESIGN AND SETTING: Clinical trials of osteoarthritis interventions were retrieved from Cochrane Library systematic reviews (2006, issue 2). We examined the age distribution of the trial participants and eligibility criteria. RESULTS: We analyzed data from 219 eligible trials from 18 systematic reviews. The average mean age of the participants was 63 years. Only 13 trials (6.4%) had a mean age between 71 and 80 years and only one trial had a mean age exceeding 80 years. Among trials where the age range of participants was available or could be approximately inferred, we estimated that 66 (38%) trials had not included any patients over 80 years old. Only 23 trials specifically excluded patients over 70 based on reported eligibility criteria, but 168 trials excluded patients with various comorbidities and 142 trials excluded patients receiving other specific treatments. CONCLUSIONS: Elderly patients are considerably under-represented in clinical trials of osteoarthritis. This causes an important deficit in the utility, relevance, and generalizability of trial results for this very common condition.