Assessing scientists for hiring, promotion, and tenure.

Assessment of researchers is necessary for decisions of hiring, promotion, and tenure. A burgeoning number of scientific leaders believe the current system of faculty incentives and rewards is misaligned with the needs of society and disconnected from the evidence about the causes of the reproducibility crisis and suboptimal quality of the scientific publication record. To address this issue, particularly for the clinical and life sciences, we convened a 22-member expert panel workshop in Washington, DC, in January 2017. Twenty-two academic leaders, funders, and scientists participated in the meeting. As background for the meeting, we completed a selective literature review of 22 key documents critiquing the current incentive system. From each document, we extracted how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion. This resulted in six principles for assessing scientists and associated research and policy implications. We hope the content of this paper will serve as a basis for establishing best practices and redesigning the current approaches to assessing scientists by the many players involved in that process.

The preregistration revolution.

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

Scientific progress despite irreproducibility: A seeming paradox.

It appears paradoxical that science is producing outstanding new results and theories at a rapid rate at the same time that researchers are identifying serious problems in the practice of science that cause many reports to be irreproducible and invalid. Certainly, the practice of science needs to be improved, and scientists are now pursuing this goal. However, in this perspective, we argue that this seeming paradox is not new, has always been part of the way science works, and likely will remain so. We first introduce the paradox. We then review a wide range of challenges that appear to make scientific success difficult. Next, we describe the factors that make science work-in the past, present, and presumably also in the future. We then suggest that remedies for the present practice of science need to be applied selectively so as not to slow progress and illustrate with a few examples. We conclude with arguments that communication of science needs to emphasize not just problems but the enormous successes and benefits that science has brought and is now bringing to all elements of modern society.

Deep learning early warning system for embryo culture conditions and embryologist performance in the ART laboratory.

Staff competency is a crucial component of the in vitro fertilization (IVF) laboratory quality management system because it impacts clinical outcomes and informs the key performance indicators (KPIs) used to continuously monitor and assess culture conditions. Contemporary quality control and assurance in the IVF lab can be automated (collect, store, retrieve, and analyze), to elevate quality control and assurance beyond the cursory monthly review. Here we demonstrate that statistical KPI monitoring systems for individual embryologist performance and culture conditions can be detected by artificial intelligence systems to provide systemic, early detection of adverse outcomes, and identify clinically relevant shifts in pregnancy rates, providing critical validation for two statistical process controls proposed in the Vienna Consensus Document; intracytoplasmic sperm injection (ICSI) fertilization rate and day 3 embryo quality.

A computer model for professional competence assessment according to ISO 15189.

Background As defined by ISO 15189 competence is the "demonstrated ability to apply knowledge and skills" thus, its assessment is fundamental for ensuring the quality of the total testing process in order to reduce the risk for the patient. We have developed a functional software for the measurement of professional competences in order to standardize the procedure and to collect all the data in a single platform, avoiding redundancy and dispersion. Methods Our model objectively assesses the skills, as they become measurable and comparable with appropriate standards and involves both managers and operators, to increase their active engagement. The assessment concerns everyone, but the standards to be met (numerical values) can vary according to the responsibilities. Several subjective and objective criteria are evaluated: each parameter can contribute in a variable proportion to the total skills measured according to the needs of the organization. Results The data are automatically analyzed and can be easily monitored in real time in the form of indicators, thanks to dashboards. The comparison between the skills required and those measured allows highlighting the gap useful for planning personalized training paths. Conclusions Our tool is reliable and highly adaptable to laboratories about competences to track criteria, standards and monitored indicators. The computerized management is a strategic action as it fulfills the requirements of registration, traceability, communication, data analysis and indicators development, which are the tenets of continuous improvement, and allows planning to be made on the basis of the actual training needs.

Biases encountered in long-term monitoring studies of invertebrates and microflora: Australian examples of protocols, personnel, tools and site location.

Monitoring forms the basis for understanding ecological change. It relies on repeatability of methods to ensure detected changes accurately reflect the effect of environmental drivers. However, operator bias can influence the repeatability of field and laboratory work. We tested this for invertebrates and diatoms in three trials: (1) two operators swept invertebrates from heath vegetation, (2) four operators picked invertebrates from pyrethrum knockdown samples from tree trunk and (3) diatom identifications by eight operators in three laboratories. In each trial, operators were working simultaneously and their training in the field and laboratory was identical. No variation in catch efficiency was found between the two operators of differing experience using a random number of net sweeps to catch invertebrates when sequence, location and size of sweeps were random. Number of individuals and higher taxa collected by four operators from tree trunks varied significantly between operators and with their 'experience ranking'. Diatom identifications made by eight operators were clustered together according to which of three laboratories they belonged. These three tests demonstrated significant potential bias of operators in both field and laboratory. This is the first documented case demonstrating the significant influence of observer bias on results from invertebrate field-based studies. Examples of two long-term trials are also given that illustrate further operator bias. Our results suggest that long-term ecological studies using invertebrates need to be rigorously audited to ensure that operator bias is accounted for during analysis and interpretation. Further, taxonomic harmonisation remains an important step in merging field and laboratory data collected by different operators.

Science faculty's subtle gender biases favor male students.

Despite efforts to recruit and retain more women, a stark gender disparity persists within academic science. Abundant research has demonstrated gender bias in many demographic groups, but has yet to experimentally investigate whether science faculty exhibit a bias against female students that could contribute to the gender disparity in academic science. In a randomized double-blind study (n = 127), science faculty from research-intensive universities rated the application materials of a student-who was randomly assigned either a male or female name-for a laboratory manager position. Faculty participants rated the male applicant as significantly more competent and hireable than the (identical) female applicant. These participants also selected a higher starting salary and offered more career mentoring to the male applicant. The gender of the faculty participants did not affect responses, such that female and male faculty were equally likely to exhibit bias against the female student. Mediation analyses indicated that the female student was less likely to be hired because she was viewed as less competent. We also assessed faculty participants' preexisting subtle bias against women using a standard instrument and found that preexisting subtle bias against women played a moderating role, such that subtle bias against women was associated with less support for the female student, but was unrelated to reactions to the male student. These results suggest that interventions addressing faculty gender bias might advance the goal of increasing the participation of women in science.

Evaluation of Diagnostic Microbiology Capacity and Barriers in Testing for HIV and TB at Peripheral Hospital-Based Laboratories in Oyo-State, Nigeria.

The prevalence of tuberculosis (TB) and human immunodeficiency virus (HIV) coinfection in Nigeria is currently around 19.1%. This indicates that the two diseases are still a burden on the nation"s health. The aim of this study was to evaluate the diagnostic microbiology capacity and the barriers in performing assay for TB and HIV at peripheral district-level hospital-based laboratories in Oyo State, Nigeria. Diagnostic microbiology capacity was estimated using a scale of 100-point where scores ≤ 49% were categorized as low, 50-79% fair and ≥80% good. Barriers to diagnosis were summarized in proportions. The diagnostic microbiology capacity revealed that 6 (35.3%) and 11 (64.7%) of the laboratories had "fair" and "low" capacity, respectively, to detect TB in cerebrospinal fluid/sputum. In testing for HIV, 3 (17.6%) of the laboratories had "fair capacity" and 14 (82.4%) had "low capacity" to detect CD4 count and HIV antibodies in blood serum. The major diagnostic barriers in almost all (94.1%) the laboratories were lack of culture supplies and nonavailability of reagents/testing kits. There was no diagnostic microbiology service with good capacity to facilitate case detection of HIV and TB at the peripheral hospitals. Hence there is a need to improve the supply of reagents, culture stock and testing kits. This will facilitate the detection of TB and HIV cases in peripheral communities. IMPORTANCE This study provided a snapshot knowledge of testing capabilities and commodity availability at state laboratories. The findings should inform the action of stakeholders to improve diagnostic microbiology capacity, consequently enhancing diagnostic measures in detecting human immunodeficiency virus and Mycobacterium tuberculosis.

A Comprehensive Approach to Assuring Quality of Laboratory Testing in HIV Surveys: Lessons Learned From the Population-Based HIV Impact Assessment Project.

BACKGROUND: Conducting HIV surveys in resource-limited settings is challenging because of logistics, limited availability of trained personnel, and complexity of testing. We described the procedures and systems deemed critical to ensure high-quality laboratory data in the population-based HIV impact assessments and large-scale household surveys. METHODS: Laboratory professionals were engaged in every stage of the surveys, including protocol development, site assessments, procurement, training, quality assurance, monitoring, analysis, and reporting writing. A tiered network of household, satellite laboratories, and central laboratories, accompanied with trainings, optimized process for blood specimen collection, storage, transport, and real-time monitoring of specimen quality, and test results at each level proved critical in maintaining specimen integrity and high-quality testing. A plausibility review of aggregate merged data was conducted to confirm associations between key variables as a final quality check for quality of laboratory results. RESULTS: Overall, we conducted a hands-on training for 3355 survey staff across 13 surveys, with 160-387 personnel trained per survey on biomarker processes. Extensive training and monitoring demonstrated that overall, 99% of specimens had adequate volume and 99.8% had no hemolysis, indicating high quality. We implemented quality control and proficiency testing for testing, resolved discrepancies, verified >300 Pima CD4 instruments, and monitored user errors. Aggregate data review for plausibility further confirmed the high quality of testing. CONCLUSIONS: Ongoing engagement of laboratory personnel to oversee processes at all levels of the surveys is critical for successful national surveys. High-quality population-based HIV impact assessments laboratory data ensured reliable results and demonstrated the impact of HIV programs in 13 countries.

Minimum standards for practices offering assisted reproductive technologies: a committee opinion.

This document is designed to provide a framework for assisted reproductive technology (ART) programs that meet or exceed the requirements suggested by the Centers for Disease Control and Prevention (CDC) for certification of ART laboratories. This document replaces the document, "Revised Minimum Standards for Practices Offering Assisted Reproductive Technologies: A Committee Opinion," published in 2014.

Implementation of p16/Ki67 dual stain cytology in a Danish routine screening laboratory: Importance of adequate training and experience.

BACKGROUND: Immunocytochemical staining with p16/Ki67 has been suggested as a promising triage biomarker in cervical cancer screening. As dual staining is a subjective method, proper training may be required to ensure safe implementation in routine laboratories and reduce risk of misclassification. We determined concordance between novice evaluators and an expert, stratified by number of slides reviewed at three reading points. METHODS: The study was conducted at the Department of Pathology, Randers, Denmark. Women were eligible if they were aged ≥45, had been enrolled in one of two ongoing clinical studies, and had a dual stain slide available. Dual staining was performed using the CINtec plus assay. Slides were randomly selected from three reading points at which novice evaluators had reviewed <30, ~300, and ≥500 dual stain slides respectively. Level of concordance was estimated using Cohen's Kappa, κ. RESULTS: Of 600 eligible slides, 50 slides were selected for review as recommended by the manufacturer. Median age was 68 years (range: 58-74). Overall concordance was good (κ = 0.68, 95% confidence interval [CI]: 0.60-0.76), with an overall agreement of 84% (95% CI: 70.9%-92.8%). Concordance improved with increasing number of slides reviewed at a given reading point, from a moderate concordance (κ = 0.47, 95% CI: 0.05-0.90) after reviewing <30 slides to a good concordance (κ = 0.66, 95% CI: 0.20-0.88) and a very good concordance (κ = 0.88, 95% CI: 0.66-1.00) after reviewing ~300 and ≥500 slides, respectively. CONCLUSIONS: When interpreting dual stain slides from older women, concordance increased slightly as novice evaluators received more training and experience. Although further evaluation is warranted, these findings indicate that a significant amount of training and experience of novice evaluators may be needed to ensure accurate dual stain interpretation in this age group. Future studies should accurately describe training and experience of evaluators to enable a better comparison of concordance and diagnostic accuracy across studies. TRIAL REGISTRATION: NCT04114968 and NCT04298957.