The Rigor and Transparency Index Quality Metric for Assessing Biological and Medical Science Methods.

The reproducibility crisis is a multifaceted problem involving ingrained practices within the scientific community. Fortunately, some causes are addressed by the author's adherence to rigor and reproducibility criteria, implemented via checklists at various journals. We developed an automated tool (SciScore) that evaluates research articles based on their adherence to key rigor criteria, including NIH criteria and RRIDs, at an unprecedented scale. We show that despite steady improvements, less than half of the scoring criteria, such as blinding or power analysis, are routinely addressed by authors; digging deeper, we examined the influence of specific checklists on average scores. The average score for a journal in a given year was named the Rigor and Transparency Index (RTI), a new journal quality metric. We compared the RTI with the Journal Impact Factor and found there was no correlation. The RTI can potentially serve as a proxy for methodological quality.

DataMed - an open source discovery index for finding biomedical datasets.

OBJECTIVE: Finding relevant datasets is important for promoting data reuse in the biomedical domain, but it is challenging given the volume and complexity of biomedical data. Here we describe the development of an open source biomedical data discovery system called DataMed, with the goal of promoting the building of additional data indexes in the biomedical domain. MATERIALS AND METHODS: DataMed, which can efficiently index and search diverse types of biomedical datasets across repositories, is developed through the National Institutes of Health-funded biomedical and healthCAre Data Discovery Index Ecosystem (bioCADDIE) consortium. It consists of 2 main components: (1) a data ingestion pipeline that collects and transforms original metadata information to a unified metadata model, called DatA Tag Suite (DATS), and (2) a search engine that finds relevant datasets based on user-entered queries. In addition to describing its architecture and techniques, we evaluated individual components within DataMed, including the accuracy of the ingestion pipeline, the prevalence of the DATS model across repositories, and the overall performance of the dataset retrieval engine. RESULTS AND CONCLUSION: Our manual review shows that the ingestion pipeline could achieve an accuracy of 90% and core elements of DATS had varied frequency across repositories. On a manually curated benchmark dataset, the DataMed search engine achieved an inferred average precision of 0.2033 and a precision at 10 (P@10, the number of relevant results in the top 10 search results) of 0.6022, by implementing advanced natural language processing and terminology services. Currently, we have made the DataMed system publically available as an open source package for the biomedical community.

The Function Biomedical Informatics Research Network Data Repository.

The Function Biomedical Informatics Research Network (FBIRN) developed methods and tools for conducting multi-scanner functional magnetic resonance imaging (fMRI) studies. Method and tool development were based on two major goals: 1) to assess the major sources of variation in fMRI studies conducted across scanners, including instrumentation, acquisition protocols, challenge tasks, and analysis methods, and 2) to provide a distributed network infrastructure and an associated federated database to host and query large, multi-site, fMRI and clinical data sets. In the process of achieving these goals the FBIRN test bed generated several multi-scanner brain imaging data sets to be shared with the wider scientific community via the BIRN Data Repository (BDR). The FBIRN Phase 1 data set consists of a traveling subject study of 5 healthy subjects, each scanned on 10 different 1.5 to 4 T scanners. The FBIRN Phase 2 and Phase 3 data sets consist of subjects with schizophrenia or schizoaffective disorder along with healthy comparison subjects scanned at multiple sites. In this paper, we provide concise descriptions of FBIRN's multi-scanner brain imaging data sets and details about the BIRN Data Repository instance of the Human Imaging Database (HID) used to publicly share the data.

Function biomedical informatics research network recommendations for prospective multicenter functional MRI studies.

This report provides practical recommendations for the design and execution of multicenter functional MRI (MC-fMRI) studies based on the collective experience of the Function Biomedical Informatics Research Network (FBIRN). The study was inspired by many requests from the fMRI community to FBIRN group members for advice on how to conduct MC-fMRI studies. The introduction briefly discusses the advantages and complexities of MC-fMRI studies. Prerequisites for MC-fMRI studies are addressed before delving into the practical aspects of carefully and efficiently setting up a MC-fMRI study. Practical multisite aspects include: (i) establishing and verifying scan parameters including scanner types and magnetic fields, (ii) establishing and monitoring of a scanner quality program, (iii) developing task paradigms and scan session documentation, (iv) establishing clinical and scanner training to ensure consistency over time, (v) developing means for uploading, storing, and monitoring of imaging and other data, (vi) the use of a traveling fMRI expert, and (vii) collectively analyzing imaging data and disseminating results. We conclude that when MC-fMRI studies are organized well with careful attention to unification of hardware, software and procedural aspects, the process can be a highly effective means for accessing a desired participant demographics while accelerating scientific discovery.

Brain-performance correlates of working memory retrieval in schizophrenia: a cognitive modeling approach.

Correlations of cognitive functioning with brain activation during a sternberg item recognition paradigm (SIRP) were investigated in patients with schizophrenia and in healthy controls studied at 8 sites. To measure memory scanning times, 4 response time models were fit to SIRP data. The best fitting model assumed exhaustive serial memory scanning followed by self-terminating memory search and involved one intercept parameter to represent SIRP processes not contributing directly to memory scanning. Patients displayed significantly longer response times with increasing memory load and differed on the memory scanning, memory search, and intercept parameters of the best fitting probability model. Groups differed in the correlation between the memory scanning parameter and linear brain response to increasing memory load within left inferior and left middle frontal gyrus, bilateral caudate, and right precuneus. The pattern of findings in these regions indicated that high scanning capacity was associated with high neural capacity among healthy subjects but that scanning speed was uncoupled from brain response to increasing memory load among schizophrenia patients. Group differences in correlation of the best fitting model's scanning parameter with a quadratic trend in brain response to increasing memory load suggested inefficient or disordered patterns of neural inhibition among individuals with schizophrenia, especially in the left perirhinal and entorhinal cortices. The results show at both cognitive and neural levels that disordered memory scanning contributes to deficient SIRP performance among schizophrenia patients.