Toward more rigorous and informative nutritional epidemiology: The rational space between dismissal and defense of the status quo.

To date, nutritional epidemiology has relied heavily on relatively weak methods including simple observational designs and substandard measurements. Despite low internal validity and other sources of bias, claims of causality are made commonly in this literature. Nutritional epidemiology investigations can be improved through greater scientific rigor and adherence to scientific reporting commensurate with research methods used. Some commentators advocate jettisoning nutritional epidemiology entirely, perhaps believing improvements are impossible. Still others support only normative refinements. But neither abolition nor minor tweaks are appropriate. Nutritional epidemiology, in its present state, offers utility, yet also needs marked, reformational renovation. Changing the status quo will require ongoing, unflinching scrutiny of research questions, practices, and reporting-and a willingness to admit that "good enough" is no longer good enough. As such, a workshop entitled "Toward more rigorous and informative nutritional epidemiology: the rational space between dismissal and defense of the status quo" was held from July 15 to August 14, 2020. This virtual symposium focused on: (1) Stronger Designs, (2) Stronger Measurement, (3) Stronger Analyses, and (4) Stronger Execution and Reporting. Participants from several leading academic institutions explored existing, evolving, and new better practices, tools, and techniques to collaboratively advance specific recommendations for strengthening nutritional epidemiology.

Amputation for Complex Regional Pain Syndrome: Meta-Analysis and Validation of a Histopathology Scoring System.

OBJECTIVE: Pathology can provide crucial insights into the etiology of disease. The goal of this review is to evaluate the rigor of histopathology reports of Complex Regional Pain Syndrome (CRPS). METHODS: A systematic search of multiple databases identified papers that described amputation for CRPS with pathology findings. Control pathology articles were randomly chosen from the same journals. Landmark articles in Surgical Pathology were previously identified. Papers were categorized by the use of histology: Anatomic (microscopic description), Diagnostic (binary result), and Substrate (special studies only). A novel Histopathology Score assigned 1 point for each of 10 History elements and 15 Pathology elements. All articles were scored and analyzed by appropriate statistics. RESULTS: The search identified 22 CRPS, 50 Control and 50 Landmark articles. Multivariable analysis of the Pathology Score showed a significantly higher score for Anatomic vs Non-Anatomic papers (Incidence Rate Ratio (IRR) 1.54, P < .001) and Landmark vs CRPS articles (IRR 1.39, P value .003). CRPS papers reported some elements infrequently: diagnostic criteria (31.8%), routine stain (50%), any clinic-pathologic correlation (40.9%), and sample size >2 (27.3%). CONCLUSIONS: The Pathology Score is a useful quality assessment tool to evaluate studies. As expected, Anatomic papers scored significantly higher than Non-Anatomic papers. CRPS papers had small sample sizes (median 1) and infrequent reporting of diagnostic criteria, routine stain, any clinical pathologic correlation. These particular elements are crucial for analyzing and reviewing pathologic features. The analysis explains why it is quite difficult to write a meaningful systematic review of CRPS histology at this time.

Improving transparency and scientific rigor in academic publishing.

Progress in basic and clinical research is slowed when researchers fail to provide a complete and accurate report of how a study was designed, executed, and the results analyzed. Publishing rigorous scientific research involves a full description of the methods, materials, procedures, and outcomes. Investigators may fail to provide a complete description of how their study was designed and executed because they may not know how to accurately report the information or the mechanisms are not in place to facilitate transparent reporting. Here, we provide an overview of how authors can write manuscripts in a transparent and thorough manner. We introduce a set of reporting criteria that can be used for publishing, including recommendations on reporting the experimental design and statistical approaches. We also discuss how to accurately visualize the results and provide recommendations for peer reviewers to enhance rigor and transparency. Incorporating transparency practices into research manuscripts will significantly improve the reproducibility of the results by independent laboratories.

Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility.

BACKGROUND: Using internet search engines (such as Google search) in systematic literature reviews is increasingly becoming a ubiquitous part of search methodology. In order to integrate the vast quantity of available knowledge, literature mostly focuses on systematic reviews, considered to be principal sources of scientific evidence at all practical levels. Any possible individual methodological flaws present in these systematic reviews have the potential to become systemic. MAIN TEXT: This particular bias, that could be referred to as (re)search bubble effect, is introduced because of inherent, personalized nature of internet search engines that tailors results according to derived user preferences based on unreproducible criteria. In other words, internet search engines adjust their user's beliefs and attitudes, leading to the creation of a personalized (re)search bubble, including entries that have not been subjected to rigorous peer review process. The internet search engine algorithms are in a state of constant flux, producing differing results at any given moment, even if the query remains identical. There are many more subtle ways of introducing unwanted variations and synonyms of search queries that are used autonomously, detached from user insight and intent. Even the most well-known and respected systematic literature reviews do not seem immune to the negative implications of the search bubble effect, affecting reproducibility. CONCLUSION: Although immensely useful and justified by the need for encompassing the entirety of knowledge, the practice of including internet search engines in systematic literature reviews is fundamentally irreconcilable with recent emphasis on scientific reproducibility and rigor, having a profound impact on the discussion of the limits of scientific epistemology. Scientific research that is not reproducible, may still be called science, but represents one that should be avoided. Our recommendation is to use internet search engines as an additional literature source, primarily in order to validate initial search strategies centered on bibliographic databases.

The effectiveness of transcranial magnetic stimulation for dysphagia in stroke patients: an umbrella review of systematic reviews and meta-analyses.

Numerous studies have explored the use of repetitive Transcranial Magnetic Stimulation (rTMS) intervention in post-stroke dysphagia. The primary aim of this umbrella review was to appraise the methodological quality of systematic reviews (SRs), with and without meta-analyses (MAs), that synthesized the findings of randomized controlled trials (RCTs) exploring the effectiveness of rTMS in the management of dysphagia post-stroke. A secondary aim of was to evaluate the consistency and reliability of translational implications of rTMS for swallowing recovery after stroke across these SRs and MAs. We searched several databases from inception to the 14th of May 2023, to identify SRs and MAs that examined the effectiveness of rTMS in the management of dysphagia post-stroke. The methodological quality of the included studies was evaluated utilizing the AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews) instrument. To investigate the extent of literature overlap among the primary studies included in the SRs, the Graphical Overview of Evidence (GROOVE) was utilized. Of the 19 SRs that were identified, two studies received low quality ratings, while the rest (17) were rated with critically low quality based on the AMSTAR 2 rating. A high literature overlap across the SRs was observed. In all SRs and MAs reviewed, there was a consistent presence of at least some significant evidence supporting the effectiveness of rTMS in enhancing swallowing outcomes for individuals with dysphagia post-stroke, that is, all MAs reported at least a moderate overall effect in favor of rTMS (SMD range = [0.59, 6.23]). While rTMS shows promise for improving dysphagia post-stroke, the current evidence remains limited and inconclusive due to the methodological flaws observed in the published SRs and their respective MAs on the topic so far. Concerning the limitations of our study, language restrictions and methodological shortcomings may affect the generalizability of our findings.

Blockchain-Based Dynamic Consent and its Applications for Patient-Centric Research and Health Information Sharing: Protocol for an Integrative Review.

BACKGROUND: Blockchain has been proposed as a critical technology to facilitate more patient-centric research and health information sharing. For instance, it can be applied to coordinate and document dynamic informed consent, a procedure that allows individuals to continuously review and renew their consent to the collection, use, or sharing of their private health information. Such has been suggested to facilitate ethical, compliant longitudinal research, and patient engagement. However, blockchain-based dynamic consent is a relatively new concept, and it is not yet clear how well the suggested implementations will work in practice. Efforts to critically evaluate implementations in health research contexts are limited. OBJECTIVE: The objective of this protocol is to guide the identification and critical appraisal of implementations of blockchain-based dynamic consent in health research contexts, thereby facilitating the development of best practices for future research, innovation, and implementation. METHODS: The protocol describes methods for an integrative review to allow evaluation of a broad range of quantitative and qualitative research designs. The PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) framework guided the review's structure and nature of reporting findings. We developed search strategies and syntax with the help of an academic librarian. Multiple databases were selected to identify pertinent academic literature (CINAHL, Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science) and gray literature (Electronic Theses Online Service, ProQuest Dissertations and Theses, Open Access Theses and Dissertations, and Google Scholar) for a comprehensive picture of the field's progress. Eligibility criteria were defined based on PROSPERO (International Prospective Register of Systematic Reviews) requirements and a criteria framework for technology readiness. A total of 2 reviewers will independently review and extract data, while a third reviewer will adjudicate discrepancies. Quality appraisal of articles and discussed implementations will proceed based on the validated Mixed Method Appraisal Tool, and themes will be identified through thematic data synthesis. RESULTS: Literature searches were conducted, and after duplicates were removed, 492 articles were eligible for screening. Title and abstract screening allowed the removal of 312 articles, leaving 180 eligible articles for full-text review against inclusion criteria and confirming a sufficient body of literature for project feasibility. Results will synthesize the quality of evidence on blockchain-based dynamic consent for patient-centric research and health information sharing, covering effectiveness, efficiency, satisfaction, regulatory compliance, and methods of managing identity. CONCLUSIONS: The review will provide a comprehensive picture of the progress of emerging blockchain-based dynamic consent technologies and the rigor with which implementations are approached. Resulting insights are expected to inform best practices for future research, innovation, and implementation to benefit patient-centric research and health information sharing. TRIAL REGISTRATION: PROSPERO CRD42023396983; http://tinyurl.com/cn8a5x7t. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/50339.

Eleven strategies for making reproducible research and open science training the norm at research institutions.

Reproducible research and open science practices have the potential to accelerate scientific progress by allowing others to reuse research outputs, and by promoting rigorous research that is more likely to yield trustworthy results. However, these practices are uncommon in many fields, so there is a clear need for training that helps and encourages researchers to integrate reproducible research and open science practices into their daily work. Here, we outline eleven strategies for making training in these practices the norm at research institutions. The strategies, which emerged from a virtual brainstorming event organized in collaboration with the German Reproducibility Network, are concentrated in three areas: (i) adapting research assessment criteria and program requirements; (ii) training; (iii) building communities. We provide a brief overview of each strategy, offer tips for implementation, and provide links to resources. We also highlight the importance of allocating resources and monitoring impact. Our goal is to encourage researchers - in their roles as scientists, supervisors, mentors, instructors, and members of curriculum, hiring or evaluation committees - to think creatively about the many ways they can promote reproducible research and open science practices in their institutions.

Evaluating the quality of studies reporting on clinical applications of stromal vascular fraction: A systematic review and proposed reporting guidelines (CLINIC-STRA-SVF).

Background: The stromal vascular fraction (SVF) has been widely explored in a number of therapeutic applications in several specialties. Its therapeutic potential is being increasingly demonstrated, although its mechanism of action is still unclear. Objective: To evaluate the quality of studies reporting on clinical applications of SVF. Method: This is a systematic literature review that followed the PRISMA guidelines with the search of the studies from December 1, 2012, to December 1, 2022, in the following databases: MEDLINE, LILACS and EMBASE. The level of evidence of the studies was assessed using the GRADE system, and the rigor used in the publication of the results was assessed in relation to adherence to the guidelines indicated by the EQUATOR Network Group. The CLINIC - STRA-SVF reporting guideline was developed after the completion of this systematic review. Results: A total of 538 articles were found, and 77 articles were selected after reading the titles and abstracts and removing duplicates. Then, 15 studies were removed for not meeting the inclusion criteria, leaving 62 studies. The CLINIC - STRA-SVF was developed and consists of 33 items and two tables. Conclusion: There is scientific evidence, although mostly with a low level of evidence, that the use of SVF in clinical applications is safe and effective. The information published in these studies should be standardized, and the CLINIC - STRA-SVF reporting guideline proposed in this study may assist in the design, conduct, recording and reporting of clinical trials and others clinical studies involving the SVF.

Optimized Replication of Arrayed Bacterial Mutant Libraries Increases Access to Biological Resources.

Biological collections, including arrayed libraries of single transposon (Tn) or deletion mutants, greatly accelerate the pace of bacterial genetic research. Despite the importance of these resources, few protocols exist for the replication and distribution of these materials. Here, we describe a protocol for creating multiple replicates of an arrayed bacterial Tn library consisting of approximately 6,800 mutants in 96-well plates (73 plates). Our protocol provides multiple checkpoints to guard against contamination and minimize genetic drift caused by freeze/thaw cycles. This approach can also be scaled for arrayed culture collections of various sizes. Overall, this protocol is a valuable resource for other researchers considering the construction and distribution of arrayed culture collection resources for the benefit of the greater scientific community. IMPORTANCE Arrayed mutant collections drive robust genetic screens, but few protocols exist for replication of these resources and subsequent quality control. Increasing the distribution of arrayed biological collections will increase the accessibility and use of these resources. Developing standardized techniques for replication of these resources is essential for ensuring their quality and usefulness to the scientific community.

Communication regarding the article "An efficient primary screening COVID-19 by serum Raman spectroscopy".

When performing computational modeling and machine learning experiments, it is imperative to follow a protocol that minimizes bias. In this communication, we share our concerns regarding the article "An efficient primary screening COVID-19 by serum Raman spectroscopy" published in this journal. We consider that the authors may have inadvertently biased their results by not guaranteeing complete independence of test samples from the training data. We corroborate our point by reproducing the experiment with the available data, showing that if full independence of the test set was ensured, the reported results should be lower. We ask the authors to provide more information regarding their article, as well as making available all code used to generate their results. Our experiments are available at https://doi.org/10.6084/m9.figshare.14124356.

Improving Reproducibility to Enhance Scientific Rigor through Consideration of Mouse Diet.

Animal husbandry conditions, including rodent diet, constitute an example highlighting the importance of reporting experimental variables to enhance scientific rigor. In the present study, we examine the effects of three common rodent diets including two chows (Purina 5015 and Teklad 2019) and one purified ingredient diet (AIN-76A) on growth anthropometrics (body weight), behavior (nest building, actigraphy, passive avoidance) and blood biomarkers (ketones, glucose, amino acid profiles) in male and female C57BL/6J mice. We find increased body weight in response to the chows compared to purified ingredient diet albeit selectively in male mice. We did not find significantly altered behavior in female or male wild type C57BL/6J mice. However, amino acid profiles changed as an effect of sex and diet. These data contribute to a growing body of knowledge indicating that rodent diet impacts experimental outcomes and needs to be considered in study design and reporting.

A Priori Justification for Effect Measures in Single-Case Experimental Designs.

Due to the complex nature of single-case experimental design data, numerous effect measures are available to quantify and evaluate the effectiveness of an intervention. An inappropriate choice of the effect measure can result in a misrepresentation of the intervention effectiveness and this can have far-reaching implications for theory, practice, and policymaking. As guidelines for reporting appropriate justification for selecting an effect measure are missing, the first aim is to identify the relevant dimensions for effect measure selection and justification prior to data gathering. The second aim is to use these dimensions to construct a user-friendly flowchart or decision tree guiding applied researchers in this process. The use of the flowchart is illustrated in the context of a preregistered protocol. This is the first study that attempts to propose reporting guidelines to justify the effect measure choice, before collecting the data, to avoid selective reporting of the largest quantifications of an effect. A proper justification, less prone to confirmation bias, and transparent and explicit reporting can enhance the credibility of the single-case design study findings.

Guidance for biostatisticians on their essential contributions to clinical and translational research protocol review.

Rigorous scientific review of research protocols is critical to making funding decisions, and to the protection of both human and non-human research participants. Given the increasing complexity of research designs and data analysis methods, quantitative experts, such as biostatisticians, play an essential role in evaluating the rigor and reproducibility of proposed methods. However, there is a common misconception that a statistician's input is relevant only to sample size/power and statistical analysis sections of a protocol. The comprehensive nature of a biostatistical review coupled with limited guidance on key components of protocol review motived this work. Members of the Biostatistics, Epidemiology, and Research Design Special Interest Group of the Association for Clinical and Translational Science used a consensus approach to identify the elements of research protocols that a biostatistician should consider in a review, and provide specific guidance on how each element should be reviewed. We present the resulting review framework as an educational tool and guideline for biostatisticians navigating review boards and panels. We briefly describe the approach to developing the framework, and we provide a comprehensive checklist and guidance on review of each protocol element. We posit that the biostatistical reviewer, through their breadth of engagement across multiple disciplines and experience with a range of research designs, can and should contribute significantly beyond review of the statistical analysis plan and sample size justification. Through careful scientific review, we hope to prevent excess resource expenditure and risk to humans and animals on poorly planned studies.

Theory and empiricism: A comment on "Interrogating the environmental affordances model" by Pamplin and colleagues.

We strongly support efforts to generate, rigorously test, and falsify hypotheses derived from the Environmental Affordances (EA) Model of Health Disparities, as originated by the late Dr. James S. Jackson (1940-2020). Such efforts are critical to establishing robust, theoretically grounded scientific frameworks that explain the fundamental causes of racial disparities in health and wellbeing. Pamplin et al. (2021) fundamentally misrepresents the EA Model as a framework that (falsely) reifies the role of race as a determinant of health behaviors and health outcomes. Further, both their study design and analytic approach are inappropriate for testing predictions of this framework. We address these issues with the goal of recentering the scholarly conversation about how stress contributes to health, and disparities in health, over the life course.

Open Science Indicators as Metadata Fields?

Since 2000, there has been qualitative growth in the field of scientometrics. Innovations such as the DOI and the ORCID have irrevocably changed the scientific landscape. They have enabled analyses previously unheard of, in the decades preceding the new millennium. This paper proposes open science indicators (open data, open material, preregistration) as article-specific metadata fields. The authors reference the history of funding information, from bare acknowledgements to metadata field. The authors describe the mission of the Center for Open Science, and its TOP Factor database, as well as the performance of open science badges. Possibilities for a pilot study are explored, with an acknowledgement of the complexity of this undertaking.

The Symbiotic Relationship Between Scientific Quality and Animal Research Ethics.

Researchers have worked with animals as models for decades to expand our knowledge of basic biological processes and to systematically study the physiology of disease. In general, the public has an expectation that work with animals has a purpose and will ultimately reap benefits. The likelihood of such an outcome is directly dependent on the quality of the science being conducted with those animals. However, not all frameworks for consideration of the ethics around animal research overtly consider scientific quality. In the following review, we explore the complex relationship between scientific quality and animal research ethics. We advocate for the development of a detailed "Harm-Yield Analysis" for the evaluation of biomedical animal research that emphasizes scientific quality along with societal benefit in the ethical justification of the research. We reflect on the lost opportunity to establish best practices in animal research early in the career of scientists by introducing in the curriculum and encouraging the use of a paradigm of the iterative consideration of the ethics of animal research alongside other aspects of experimental design.

Moving Toward Health Policy that Respects Both Science and People Living with Obesity.

Through four decades of rising obesity, health policy has been mostly ineffective. Prevention policies failed to reverse rising trends in prevalence, partly because they are often based on biased mental models about what should work to prevent obesity, rather than empiric evidence for what does work. Bias toward people living with obesity harms health, while contributing to poor access to effective care that might serve to improve it. Better public policy will come from an increased application of objective obesity science, research to fill knowledge gaps, and respect for the human dignity of people who live with obesity.

The reporting quality, scientific rigor, and ethics of randomized placebo-controlled trials of traditional Chinese medicine compound formulations and the differences between Chinese and non-Chinese trials.

BACKGROUND: An increasing number of randomized placebo-controlled trials involving traditional Chinese medicine (TCM) compound formulations have been implemented worldwide. OBJECTIVE: The aim of this study was to assess the reporting quality, scientific rigor, and ethics of randomized placebo-controlled trials of TCM compound formulations and compare these differences between Chinese and non-Chinese trials. METHODS: English-language databases included the following: PubMed, OVID, EMBASE, and Science Citation Index Expanded. Chinese-language databases included the following: Chinese Biomedical Literature Database, Wanfang Database, Chinese Scientific and Technological Periodical Database, and the China National Knowledge Infrastructure. All were searched from respective inception to March 2009 to identify randomized placebo-controlled trials involving TCM compound prescriptions. Two reviewers independently assessed the retrieved trials via a modified Consolidated Standard of Reporting Trials (CONSORT) checklist and some evaluation indices that embodied the TCM characteristics or the scientific rigor and ethics of placebo-controlled trials. Trial publishing time was divided into 3 intervals: phase 1 (≤1999); phase 2 (2000-2004); and phase 3 (2005-2009). The number and percentage of trials reporting each item and the corresponding differences between Chinese (mainland China, Hong Kong, and Taiwan) and non-Chinese (eg, Japan, United States, Australia, Korea, and United Kingdom) trials were calculated. Moreover, the influence of trial publishing time on the reporting of CONSORT items and the differences in the number of items reported for each time interval between Chinese and non-Chinese trials were assessed. RESULTS: A total of 324 trials from China and 51 trials from other countries were included. A mean of 39.7% of the CONSORT items across all Chinese trials and 50.2% of the items across all non-Chinese trials were reported. The number of the reported CONSORT items all increased over time in both groups and the gap between Chinese articles and non-Chinese articles gradually decreased. Additionally, of the 324 Chinese articles, 137 (42.28%) reported TCM syndrome type, 113 (34.88%) reported the diagnostic criteria of diseases for TCM, and 69 (21.30%) reported efficacy evaluation indices of TCM. Of the non-Chinese articles, 3 (5.88%) reported TCM syndrome type and 1 (1.96%) reported the diagnostic criteria of diseases and evaluation indices of efficacy for TCM. It was found that 45.37% and 6.17% of Chinese articles reported the standard intervention for the diseases being treated and the emergency plan, respectively, compared with 23.53% and 9.80% for the non-Chinese articles; 33.02% and 10.49% of Chinese articles reported informed consent and ethics committee approval, respectively, compared with 92.16% and 82.35% for the non-Chinese articles. With regard to placebo ethics, 38.89% of the Chinese trials and 23.53% of the non-Chinese trials found it would not be ethically acceptable to use placebo alone in the control group. CONCLUSIONS: The data indicate that the reporting quality of the included trials on TCM compounds has improved over time, but still remains poor regardless of Chinese or non-Chinese trials. Across all trials, particularly Chinese trials, the reporting of the CONSORT items was inadequate (39.7%). The difference in the mean number of the reported CONSORT items between Chinese trials and non-Chinese trials narrowed from phase 1 (10.0 vs 13.8) to phase 3 (14.4 vs 17.4). Moreover, a large number of trials, especially non-Chinese trials (94.1%), were lacking syndrome differentiation of TCM. More importantly, in many placebo-controlled trials, especially Chinese trials, the use of placebo was not justified and was ethically contradictory.

The Impact of Complexity on Methods and Findings in Psychological Science.

The study of human behavior is severely hampered by logistical problems, ethical and legal constraints, and funding shortfalls. However, the biggest difficulty of conducting social and behavioral research is the extraordinary complexity of the study phenomena. In this article, we review the impact of complexity on research design, hypothesis testing, measurement, data analyses, reproducibility, and the communication of findings in psychological science. The systematic investigation of the world often requires different approaches because of the variability in complexity. Confirmatory testing, multi-factorial designs, survey methods, large samples, and modeling are frequently needed to study complex social and behavioral topics. Complexity impedes the measurement of general constructs, the reproducibility of results and scientific reporting, and the general rigor of research. Many of the benchmarks established by classic work in physical science are not attainable in studies of more complex phenomena. Consequently, the standards used to evaluate scientific research should be tethered to the complexity of the study topic.