The impact of mandatory conflict of interest disclosures on editors' manuscript acceptance decisions: A cross-sectional observational study.

Objective: Although current ethical standards mandate conflict of interest (COI) disclosure by authors of peer-reviewed publications, it is unknown whether disclosure affects a manuscript's fate. Our objective was to identify associations between author COI disclosure and editorial decision to publish. Methods: We performed a cross-sectional observational study of editorial decisions for original research and brief research report manuscripts submitted to Annals of Emergency Medicine between June 2014 and January 2018 using data from the journal's editorial decision software and data from a prior study that characterized author COI for the same manuscripts. Outcomes of interest included final editor decision to publish (primary), initial editor decision, and number of revisions. We compared outcomes for manuscripts with COI versus those without and by type of COI (commercial/government/other). Results: Out of 1312 manuscripts in the sample, 65.1% had no COI declarations, and 34.9% had one or more. Overall likelihood of editorial decision to publish was 13.5% (115/854) for articles without COI and 26.9% (123/458) for those with COI. Overall likelihood of editorial decision to publish was 19.8% (19/96) for articles with commercial COI only versus 33.3% (35/105) for those with government COI only. Conclusions: Articles with author-reported COI were more likely to be published than those without such a declaration. Additionally, results suggest that reports of government COI are associated with improved chance of publication. Authorities should consider relaxing COI requirements temporarily to allow investigators to perform larger scale, randomized controlled studies of the impact of mandated COI disclosure.

Effect of revealing authors' conflicts of interests in peer review: randomized controlled trial.

OBJECTIVE: To assess the effect of disclosing authors' conflict of interest declarations to peer reviewers at a medical journal. DESIGN: Randomized controlled trial. SETTING: Manuscript review process at the Annals of Emergency Medicine. PARTICIPANTS: Reviewers (n=838) who reviewed manuscripts submitted between 2 June 2014 and 23 January 2018 inclusive (n=1480 manuscripts). INTERVENTION: Reviewers were randomized to either receive (treatment) or not receive (control) authors' full International Committee of Medical Journal Editors format conflict of interest disclosures before reviewing manuscripts. Reviewers rated the manuscripts as usual on eight quality ratings and were then surveyed to obtain "counterfactual scores"-that is, the scores they believed they would have given had they been assigned to the opposite arm-as well as attitudes toward conflicts of interest. MAIN OUTCOME MEASURE: Overall quality score that reviewers assigned to the manuscript on submitting their review (1 to 5 scale). Secondary outcomes were scores the reviewers submitted for the seven more specific quality ratings and counterfactual scores elicited in the follow-up survey. RESULTS: Providing authors' conflict of interest disclosures did not affect reviewers' mean ratings of manuscript quality (Mcontrol=2.70 (SD 1.11) out of 5; Mtreatment=2.74 (1.13) out of 5; mean difference 0.04, 95% confidence interval -0.05 to 0.14), even for manuscripts with disclosed conflicts (Mcontrol= 2.85 (1.12) out of 5; Mtreatment=2.96 (1.16) out of 5; mean difference 0.11, -0.05 to 0.26). Similarly, no effect of the treatment was seen on any of the other seven quality ratings that the reviewers assigned. Reviewers acknowledged conflicts of interest as an important matter and believed that they could correct for them when they were disclosed. However, their counterfactual scores did not differ from actual scores (Mactual=2.69; Mcounterfactual=2.67; difference in means 0.02, 0.01 to 0.02). When conflicts were reported, a comparison of different source types (for example, government, for-profit corporation) found no difference in effect. CONCLUSIONS: Current ethical standards require disclosure of conflicts of interest for all scientific reports. As currently implemented, this practice had no effect on any quality ratings of real manuscripts being evaluated for publication by real peer reviewers.

Industry payments to physician journal editors.

BACKGROUND: Open Payments is a United States federal program mandating reporting of medical industry payments to physicians, increasing transparency of physician conflicts of interest (COI). Study objectives were to assess industry payments to physician-editors, and to compare their financial COI rate to all physicians within the specialty. METHODS AND FINDINGS: We performed a retrospective analysis of prospectively collected data, reviewing Open Payments from August 1, 2013 to December 31, 2016. We reviewed general payments ("… not made in connection with a research agreement") and research funding to "top tier" physician-editors of highly-cited medical journals. We compared payments to physician-editors and physicians-by-specialty. In 35 journals, 333 (74.5%) of 447 "top tier" US-based editors met inclusion criteria. Of these, 212 (63.7%) received industry-associated payments in the study period. In an average year, 141 (42.3%) of physician-editors received any direct payments to themselves including general payments and research payments, 66 (19.8%) received direct payments >$5,000 (National Institutes of Health threshold for a Significant Financial Interest) and 51 (15.3%) received >$10,000. Mean annual general payments to physician-editors was $55,157 (median 3,512, standard deviation 561,885, range 10-10,981,153). Median general payments to physician-editors were mostly higher compared to all physicians within their specialty. Mean annual direct research payment to the physician-editor was $14,558 (median 4,000, range 15-174,440). Mean annual indirect research funding to the physician-editor's institution (highly valued by academic leaders such as departmental chairs and deans) was $175,282 (median 49,107, range 0.18-5,000,000). The main study limitation was difficulty identifying physician-editors primarily responsible for making manuscript decisions. CONCLUSIONS: A substantial minority of physician-editors receive payments from industry within any given year, sometimes quite large. Most editors received payment of some kind during the four-year study period. Given the extent of editors' influences on the medical literature, more robust and accessible editor financial COI declarations are recommended.

What Does It Take to Change an Editor's Mind? Identifying Minimally Important Difference Thresholds for Peer Reviewer Rating Scores of Scientific Articles.

STUDY OBJECTIVE: We define a minimally important difference for the Likert-type scores frequently used in scientific peer review (similar to existing minimally important differences for scores in clinical medicine). The magnitude of score change required to change editorial decisions has not been studied, to our knowledge. METHODS: Experienced editors at a journal in the top 6% by impact factor were asked how large a change of rating in "overall desirability for publication" was required to trigger a change in their initial decision on an article. Minimally important differences were assessed twice for each editor: once assessing the rating change required to shift the editor away from an initial decision to accept, and the other assessing the magnitude required to shift away from an initial rejection decision. RESULTS: Forty-one editors completed the survey (89% response rate). In the acceptance frame, the median minimally important difference was 0.4 points on a scale of 1 to 5. Editors required a greater rating change to shift from an initial rejection decision; in the rejection frame, the median minimally important difference was 1.2 points. Within each frame, there was considerable heterogeneity: in the acceptance frame, 38% of editors did not change their decision within the maximum available range; in the rejection frame, 51% did not. CONCLUSION: To our knowledge, this is the first study to determine the minimally important difference for Likert-type ratings of research article quality, or in fact any nonclinical scientific assessment variable. Our findings may be useful for future research assessing whether changes to the peer review process produce clinically meaningful differences in editorial decisionmaking.

Does mentoring new peer reviewers improve review quality? A randomized trial.

BACKGROUND: Prior efforts to train medical journal peer reviewers have not improved subsequent review quality, although such interventions were general and brief. We hypothesized that a manuscript-specific and more extended intervention pairing new reviewers with high-quality senior reviewers as mentors would improve subsequent review quality. METHODS: Over a four-year period we randomly assigned all new reviewers for Annals of Emergency Medicine to receive our standard written informational materials alone, or these materials plus a new mentoring intervention. For this program we paired new reviewers with a high-quality senior reviewer for each of their first three manuscript reviews, and asked mentees to discuss their review with their mentor by email or phone. We then compared the quality of subsequent reviews between the control and intervention groups, using linear mixed effects models of the slopes of review quality scores over time. RESULTS: We studied 490 manuscript reviews, with similar baseline characteristics between the 24 mentees who completed the trial and the 22 control reviewers. Mean quality scores for the first 3 reviews on our 1 to 5 point scale were similar between control and mentee groups (3.4 versus 3.5), as were slopes of change of review scores (-0.229 versus -0.549) and all other secondary measures of reviewer performance. CONCLUSIONS: A structured training intervention of pairing newly recruited medical journal peer reviewers with senior reviewer mentors did not improve the quality of their subsequent reviews.

Medical journal editors lacked familiarity with scientific publication issues despite training and regular exposure.

OBJECTIVE: To characterize medical editors by determining their demographics, training, potential sources of conflict of interest (COI), and familiarity with ethical standards. STUDY DESIGN AND SETTING: We selected editors of clinical medical journals with the highest annual citation rates. One hundred eighty-three editors were electronically surveyed (response rate, 52%) on demographics and experiences with editorial training, publication ethics, industry, and scientific publication organizations. RESULTS: Editors reported formal (76%) and informal (89%) training in medical editing topics. Most editors saw publication ethics issues (e.g., authorship, COIs) at least once a year. When presented with four questions about editorial issues discussed in commonly cited authoritative policy sources, performance was poor on topics of authorship (30% answered correctly), COI (15%), peer review (16%), and plagiarism (17%). Despite this, confidence level in editorial skills on a Likert scale from the beginning to the end of the survey dropped only slightly from 4.2 to 3.9 (P<0.0001). CONCLUSION: Our study presents a current look at editors of major clinical medical journals. Most editors reported training in medical editing topics, saw ethical issues regularly, and were aware of scientific publication organizations, but their knowledge of four common and well-disseminated publication ethics topics appears poor.