Tool to assess recognition and understanding of elements in Summary of Findings Table for health evidence synthesis: a cross-sectional study.

of Findings (SoF) tables concisely present the main findings of evidence synthesis of health evidence, but how users navigate it to understand and interpret the presented information is not clear. We quantified the interaction of medical students with an SoF table while answering a knowledge quiz. Read&Learn tool was used to measure the number of target and non-target table cells visited for each question and the time spent on these cells. Students positively identified target elements for quiz questions and answered simpler questions, but struggled with critical thinking and understanding study outcomes. The question on outcomes with the largest improvement post-intervention had the fewest correct answers, the longest interaction with table cells and the most opened cells before answering. Students spent a median of 72% of the time reading target table cells. A heatmap of the interactions showed that they were mostly answer-oriented. Further development of the tool and metrics is needed to use the tool and the metrics to study the cognitive processes during the assessment of health evidence.

Does peer review improve the statistical content of manuscripts? A study on 27 467 submissions to four journals.

Improving the methodological rigour and the quality of data analysis in manuscripts submitted to journals is key to ensure the validity of scientific claims. However, there is scant knowledge of how manuscripts change throughout the review process in academic journals. Here, we examined 27 467 manuscripts submitted to four journals from the Royal Society (2006-2017) and analysed the effect of peer review on the amount of statistical content of manuscripts, i.e. one of the most important aspects to assess the methodological rigour of manuscripts. We found that manuscripts with both initial low or high levels of statistical content increased their statistical content during peer review. The availability of guidelines on statistics in the review forms of journals was associated with an initial similarity of statistical content of manuscripts but did not have any relevant implications on manuscript change during peer review. We found that when reports were more concentrated on statistical content, there was a higher probability that these manuscripts were eventually rejected by editors.

On the Design and Validation of Assessing Tools for Measuring the Impact of Programs Promoting STEM Vocations.

This paper presents the design and validation process of a set of instruments to evaluate the impact of an informal learning initiative to promote Science, Technology, Engineering, and Mathematics (STEM) vocations in students, their families (parents), and teachers. The proposed set of instruments, beyond assessing the satisfaction of the public involved, allow collecting data to evaluate the impact in terms of changes in the consideration of the role of women in STEM areas and STEM vocations. The procedure followed to develop the set of instruments consisted of two phases. In the first phase, a preliminary version (v1) of the questionnaires was designed based on the objectives of the Girls4STEM initiative, an inclusive project promoting STEM vocations between 6 and 18 years old boys and girls. Five specific questionnaires were designed, one for the families (post activity), two for the students (pre and post activity) and two for the teachers (pre and post avitivity). A refined version (v2) of each questionnaire was obtained with evidence of content validity after undergoing an expert judgment process. The second phase was the refinement of the (v2) instruments, to ascertain the evidence of reliability and validity so that a final version (v3) was derived. In the paper, a high-quality set of good practices focused on promoting diversity and gender equality in the STEM sector are presented from a Higher Education Institution perspective, the University of Valencia. The main contribution of this work is the achievement of a set of instruments, rigorously designed for the evaluation of the implementation and effectiveness of a STEM promoting program, with sufficient validity evidence. Moreover, the proposed instruments can be a reference for the evaluation of other projects aimed at diversifying the STEM sector.

The effect of publishing peer review reports on referee behavior in five scholarly journals.

To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees' willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

Gender Diversity in STEM Disciplines: A Multiple Factor Problem.

Lack of diversity, and specifically, gender diversity, is one of the key problems that both technological companies and academia are facing these days. Moreover, recent studies show that the number of female students enrolled in science, technology, engineering and mathematics (STEM) related disciplines have been decreasing in the last twenty years, while the number of women resigning from technological job positions remains unacceptably high. As members of a higher education institution, we foresee that working towards increasing and retaining the number of female students enrolled in STEM disciplines can help to alleviate part of the challenges faced by women in STEM fields. In this paper, we first review the main barriers and challenges that women encounter in their professional STEM careers through different age stages. Next, we focus on the special case of the information theory field, discussing the potential of gendered innovation, and whether it can be applied in the Information Theory case. The working program developed by the School of Engineering at the University of Valencia (ETSE-UV), Spain, which aims at decreasing the gender diversity gap, is then presented and recommendations for practice are given. This program started in 2011 and it encompasses Bachelor, Master and PhD levels. Four main actions are implemented: Providing institutional encouragement and support, increasing the professional support network, promoting and supporting the leadership, and increasing the visibility of female role models. To assess the impact of these actions, a chi-square test of independence is included to evaluate whether there is a significant effect on the percentage of enrolled female students. The percentage of graduated female students in the information and Communications Technology Field is also positioned with respect to other universities and the Spanish reference value. This analysis establishes that, in part, this program has helped to achieve higher female graduation rates, especially among Bachelor students, as well as increasing the number of top-decision positions held by faculty women.