Are numerical scores important for grant assessment? A cross-sectional study.

Background: In the evaluation of research proposals, reviewers are often required to provide their opinions using various forms of quantitative and qualitative criteria. In 2020, the European Commission removed, for the Marie Sklodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) funding scheme, the numerical scores from the individual evaluations but retained them in the consensus report. This study aimed to assess whether there were any differences in reviewer comments' linguistic characteristics after the numerical scoring was removed, compared to comments from 2019 when numerical scoring was still present. Methods: This was an observational study and the data were collected for the Marie Sklodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) evaluation reports from the calls of 2019 and 2020, for both individual and consensus comments and numerical scores about the quality of the research proposal on three evaluation criteria: Excellence, Impact and Implementation. All comments were analyzed using the Linguistic Inquiry and Word Count (LIWC) program. Results: For both years, the comments for proposal's strengths were written in a style that reflects objectivity, clout, and positive affect, while in weaknesses cold and objective style dominated, and that pattern remained stable across proposal status and research domains. Linguistic variables explained a very small proportion of the variance of the differences between 2019 and 2020 (McFadden R 2=0.03). Conclusions: Removing the numerical scores was not associated with the differences in linguistic characteristics of the reviewer comments. Future studies should adopt a qualitative approach to assess whether there are conceptual changes in the content of the comments.

Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals.

Background: We assessed the ethics review of proposals selected for funding under the Marie Sklodowska-Curie Actions (MSCA) and the European Research Council (ERC) in Horizon 2020, EU's framework programme for research and innovation, 2014-2020. Methods: We analysed anonymized datasets for 3,054 MSCA individual fellowships (IF), 417 MSCA Innovative Training Networks (ITN), and 1,465 ERC main-listed proposals with ethics conditional clearance, over four years (2016 to 2019). The datasets included the information on ethics issues identified by applicants in their proposal and ethics issues and requirements identified by ethics experts during the ethics review. Results: 42% of proposals received ethical clearance. For proposals with conditional ethics clearance (n=3546), most of the identified ethics issues by both applicants and ethics experts were in the ethics categories related to humans; protection of personal data; environment, health and safety; and non-EU countries. Ethics experts identified twice as many ethics issues compared to applicants across funding schemes, years, and from high- and low-research performing countries. ERC grants had the highest number of ethics requirements per proposal (median (Md)=8, interquartile range (IQR=4-14), compared to ITN (Md=6, IQR=3-13) and IF grants (Md=3, IQR=2-6). The majority of requirements had to be fulfilled after grant agreement: 99.4% for IF, 99.5% for ITN, and 26.0% for ERC. For 9% of the proposals, the requirements included the appointment of an independent ethics advisor and 1% of the proposals had to appoint an ethics advisory board. Conclusions: Many applicants for highly competitive H2020 funding schemes lack awareness of ethics issues raised by their proposed research. There is a need for better training of researchers at all career stages about ethics issues in research, more support to researchers from research organizations to follow the funding agencies requirements, as well as further development and harmonization of the ethics appraisal process during grant assessment.

A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018.

Most funding agencies rely on peer review to evaluate grant applications and proposals, but research into the use of this process by funding agencies has been limited. Here we explore if two changes to the organization of peer review for proposals submitted to various funding actions by the European Union has an influence on the outcome of the peer review process. Based on an analysis of more than 75,000 applications to three actions of the Marie Curie programme over a period of 12 years, we find that the changes - a reduction in the number of evaluation criteria used by reviewers and a move from in-person to virtual meetings - had little impact on the outcome of the peer review process. Our results indicate that other factors, such as the type of grant or area of research, have a larger impact on the outcome.

Effects of seniority, gender and geography on the bibliometric output and collaboration networks of European Research Council (ERC) grant recipients.

Assessing the success and performance of researchers is a difficult task, as their grant output is influenced by a series of factors, including seniority, gender and geographical location of their host institution. In order to assess the effects of these factors, we analysed the publication and citation outputs, using Scopus and Web of Science, and the collaboration networks of European Research Council (ERC) starting (junior) and advanced (senior) grantees. For this study, we used a cohort of 355 grantees from the Life Sciences domain of years 2007-09. While senior grantees had overall greater publication output, junior grantees had a significantly greater pre-post grant award increase in their overall number of publications and in those on which they had last authorship. The collaboration networks size and the number of sub-communities increased for all grantees, although more pronounced for juniors, as they departed from smaller and more compact pre-award co-authorship networks. Both junior and senior grantees increased the size of the community within which they were collaborating in the post-award period. Pre-post grant award performance of grantees was not related to gender, although male junior grantees had more publications than female grantees before and after the grant award. Junior grantees located in lower research-performing countries published less and had less diverse collaboration networks than their peers located in higher research-performing countries. Our study suggests that research environment has greater influence on post-grant award publications than gender especially for junior grantees. Also, collaboration networks may be a useful complement to publication and citation outputs for assessing post-grant research performance, especially for grantees who already have a high publication output and who get highly competitive grants such as those from ERC.

Shiga toxin B-subunit sequential binding to its natural receptor in lipid membranes.

Shiga toxin B-subunit (STxB), a protein involved in the cell-binding and intracellular trafficking of Shiga holotoxin, binds to a specific glycolipid, the globotriaosyl ceramide (Gb(3)). Tryptophan residues of STxB, located at the protein-membrane interface, allow one to study its interaction with model membranes by means of spectroscopic methods with no need for chemical derivatisation with a fluorophore. The protein emits maximally around 346 nm and a blue shift of about 8 nm, as well as the occurrence of changes in the emission fluorescence intensity spectra, is indicative of insertion and partition into the membrane. However, the interaction seems to take place without pentamer dissociation. Acrylamide quenching experiments confirm tryptophan residues become less exposed to solvent when in the presence of vesicles, and the use of lipophilic probes suggests that they are located in a shallow position near the water/membrane interface. Fluorescence intensity and lifetime measurements upon STxB titration with Gb(3)-containing vesicles suggest a complex STxB/Gb(3) docking mechanism involving static quenching in the later stages. Based on our observations, a model of the protein-membrane interaction is proposed and the STxB membrane partition and binding constants were calculated.

Correlation between Shiga toxin B-subunit stability and antigen crosspresentation: a mutational analysis.

The homopentameric B-subunit of Shiga toxin (STxB) is used as a tool to deliver antigenic peptides and proteins to the cytosolic compartment of dendritic cells (DCs). In this study, a series of interface mutants of STxB has been constructed. All mutants retained their overall conformation, while a loss in thermal stability was observed. This effect was even more pronounced in trifluoroethanol solutions that mimic the membrane environment. Despite this, all mutants were equally efficient at delivering a model antigenic protein into the MHC class I-restricted antigen presentation pathway of mouse DCs, suggesting that the structural stability of STxB is not a key factor in the membrane translocation process.

Characterization of the non-native trifluoroethanol-induced intermediate conformational state of the Shiga toxin B-subunit.

The effect of increasing concentrations of 2,2,2-trifluoroethanol (TFE) on the conformational stability of the Shiga toxin B-subunit (STxB), a bacterial homopentameric protein involved in cell-surface binding and intracellular transport, has been studied by far-, near-UV circular dichroism (CD), intrinsic fluorescence, analytical ultracentrifugation, and differential scanning calorimetry (DSC) under equilibrium conditions. Our data show that the native structure of STxB is highly perturbed by the presence of TFE. In fact, at concentrations of TFE above 20% (v/v), the native pentameric conformation of the protein is cooperatively transformed into a helix-rich monomeric and partially folded conformational state with no significant tertiary structure. Additionally, no cooperative transition was detected upon a further increase in the TFE concentration (above 40% (v/v)). The thermal stability of STxB was investigated at several different TFE concentrations using DSC and CD spectroscopy. Thermal transitions at TFE concentrations of up to 20% (v/v) were successfully fitted to the two-state folding/unfolding coupled to oligomerization model consistent with the transition between a pentameric folded conformation to a monomeric state of the protein, which the presence of TFE stabilizes as a partially folded conformation.

Cholera and Shiga toxin B-subunits: thermodynamic and structural considerations for function and biomedical applications.

The B-subunits of cholera and Shiga toxins are functionally and structurally related proteins with different chain lengths and no sequence similarity. They are responsible for toxin binding to specific glycosphingolipid receptors and intracellular toxin trafficking. Indeed, it is clearly established that B-subunits have the unique capacity of targeting the toxins to a poorly explored intracellular pathway, the retrograde route, allowing the transfer to the cytosol of the associated catalytic A-subunits, by retro-translocation from the endoplasmic reticulum. The B-subunits have also been used as vectors for antigen presentation in immunotherapeutic approaches. It is, however, not known if and how the B-subunits intervene in membrane translocation of the A-subunits and/or antigens to the cytosol. Therefore, it is important to characterise the driving force of pentamer formation, its conformational stability, and toxin-receptor interactions. This review summarises recent studies that have dealt with these topics.

Peer Review Evaluation Process of Marie Curie Actions under EU's Seventh Framework Programme for Research.

We analysed the peer review of grant proposals under Marie Curie Actions, a major EU research funding instrument, which involves two steps: an independent assessment (Individual Evaluation Report, IER) performed remotely by 3 raters, and a consensus opinion reached during a meeting by the same raters (Consensus Report, CR). For 24,897 proposals evaluated from 2007 to 2013, the association between average IER and CR scores was very high across different panels, grant calls and years. Median average deviation (AD) index, used as a measure of inter-rater agreement, was 5.4 points on a 0-100 scale (interquartile range 3.4-8.3), overall, demonstrating a good general agreement among raters. For proposals where one rater disagreed with the other two raters (n=1424; 5.7%), or where all 3 raters disagreed (n=2075; 8.3%), the average IER and CR scores were still highly associated. Disagreement was more frequent for proposals from Economics/Social Sciences and Humanities panels. Greater disagreement was observed for proposals with lower average IER scores. CR scores for proposals with initial disagreement were also significantly lower. Proposals with a large absolute difference between the average IER and CR scores (≥10 points; n=368, 1.5%) generally had lower CR scores. An inter-correlation matrix of individual raters' scores of evaluation criteria of proposals indicated that these scores were, in general, a reflection of raters' overall scores. Our analysis demonstrated a good internal consistency and general high agreement among raters. Consensus meetings appear to be relevant for particular panels and subsets of proposals with large differences among raters' scores.

Structural stability of adenylate kinase from the sulfate-reducing bacteria Desulfovibrio gigas.

A novel adenylate kinase (AK) has recently been purified from Desulfovibrio gigas and characterized as a Co(2+)/Zn(2+)-containing enzyme: this is an unusual characteristic for AKs from Gram-negative bacteria, in which these enzymes are normally devoid of metals. Here, we studied the conformational stability of holo- and apo-AK as a function of temperature by differential scanning calorimetry (DSC), circular dichroism (CD), and intrinsic fluorescence spectroscopy. The thermal unfolding of AK is a cooperative two-state process, and is sufficiently reversible in the 9-11 pH range, that can be correctly interpreted in terms of a simple two-state thermodynamic model. The spectral parameters as monitored by ellipticity changes in the CD spectra of the enzyme as well as the decrease in tryptophan intensity emission upon heating were seen to be good complements to the highly sensitive but integral DSC-method.

Untangling the folding mechanism of the 5(2)-knotted protein UCH-L3.

Proteins possessing deeply embedded topological knots in their structure add a stimulating new challenge to the already complex protein-folding problem. The most complicated knotted topology observed to date belongs to the human enzyme ubiquitin C-terminal hydrolase UCH-L3, which is an integral part of the ubiquitin-proteasome system. The structure of UCH-L3 contains five distinct crossings of its polypeptide chain, and it adopts a 5(2)-knotted topology, making it a fascinating target for folding studies. Here, we provide the first in depth characterization of the stability and folding of UCH-L3. We show that the protein can unfold and refold reversibly in vitro without the assistance of molecular chaperones, demonstrating that all the information necessary for the protein to find its knotted native structure is encoded in the amino acid sequence, just as with any other globular protein, and that the protein does not enter into any deep kinetic traps. Under equilibrium conditions, the unfolding of UCH-L3 appears to be two-state, however, multiphasic folding and unfolding kinetics are observed and the data are consistent with a folding pathway in which two hyperfluorescent intermediates are formed. In addition, a very slow phase in the folding kinetics is shown to be limited by proline-isomerization events. Overall, the data suggest that a knotted topology, even in its most complex form, does not necessarily limit folding in vitro, however, it does seem to require a complex folding mechanism which includes the formation of several distinct intermediate species.

Thermodynamic characterization of the palm tree Roystonea regia peroxidase stability.

The structural stability of a peroxidase, a dimeric protein from royal palm tree (Roystonea regia) leaves, has been characterized by high-sensitivity differential scanning calorimetry, circular dichroism, steady-state tryptophan fluorescence and analytical ultracentifugation under different solvent conditions. It is shown that the thermal and chemical (using guanidine hydrochloride (Gdn-HCl)) folding/unfolding of royal palm tree peroxidase (RPTP) at pH 7 is a reversible process involving a highly cooperative transition between the folded dimer and unfolded monomers, with a free stabilization energy of about 23 kcal per mol of monomer at 25 degrees C. The structural stability of RPTP is pH-dependent. At pH 3, where ion pairs have disappeared due to protonation, the thermally induced denaturation of RPTP is irreversible and strongly dependent upon the scan rate, suggesting that this process is under kinetic control. Moreover, thermally induced transitions at this pH value are dependent on the protein concentration, allowing it to be concluded that in solution RPTP behaves as dimer, which undergoes thermal denaturation coupled with dissociation. Analysis of the kinetic parameters of RPTP denaturation at pH 3 was accomplished on the basis of the simple kinetic scheme N-->kD, where k is a first-order kinetic constant that changes with temperature, as given by the Arrhenius equation; N is the native state, and D is the denatured state, and thermodynamic information was obtained by extrapolation of the kinetic transition parameters to an infinite heating rate. Obtained in this way, the value of RPTP stability at 25 degrees C is ca. 8 kcal per mole of monomer lower than at pH 7. In all probability, this quantity reflects the contribution of ion pair interactions to the structural stability of RPTP. From a comparison of the stability of RPTP with other plant peroxidases it is proposed that one of the main factors responsible for the unusually high stability of RPTP which enhances its potential use for biotechnological purposes, is its dimerization.

Thermodynamic analysis of the structural stability of the shiga toxin B-subunit.

The conformational stability of Shiga toxin B-subunit (STxB), a pentameric protein from Shigella dysenteriae has been characterized by high sensitivity differential scanning calorimetry and circular dichroism spectroscopy under different solvent conditions. It is shown that the thermal folding/unfolding of STxB is a reversible process involving a highly cooperative transition between folded pentamer and unfolded monomers. The conformational stability of STxB is pH dependent and because of its pentameric nature is also concentration dependent. STxB is maximally stable in the pH range from 5 to 9 (Delta G upon unfolding is close to 13 kcal per mol of monomer at 25 degrees C), and its stability decreases both at lower and at higher pH values. The pH dependence of the Gibbs energy of stabilization between pH 2.5 and 5 is consistent with the change in the ionizable state of an average of four groups per monomer upon unfolding. Structural thermodynamic calculations show that the stabilization of the STxB pentamer is primarily due to the interactions established between monomers rather than intramonomer interactions. The folding of an isolated monomer into the conformation existing in the pentamer is unfavorable and expected to be characterized by a free-energy change upon folding in the order of 2.5 kcal mol(-1) at 25 degrees C. On the average, intersubunit interaction induced upon oligomerization of folded monomers should contribute close to -13.4 kcal per mol of monomer to bring the overall Gibbs energy to the experimentally determined value at this temperature.

Thermal stability of peroxidase from the african oil palm tree Elaeis guineensis.

The thermal stability of peroxidase from leaves of the African oil palm tree Elaeis guineensis (AOPTP) at pH 3.0 was studied by differential scanning calorimetry (DSC), intrinsic fluorescence, CD and enzymatic assays. The spectral parameters as monitored by ellipticity changes in the far-UV CD spectrum of the enzyme as well as the increase in tryptophan intensity emission upon heating, together with changes in enzymatic activity with temperature were seen to be good complements to the highly sensitive but integral method of DSC. The data obtained in this investigation show that thermal denaturation of palm peroxidase is an irreversible process, under kinetic control, that can be satisfactorily described by the two-state kinetic scheme, N -->(k) D, where k is a first-order kinetic constant that changes with temperature, as given by the Arrhenius equation; N is the native state, and D is the denatured state. On the basis of this model, the parameters of the Arrhenius equation were calculated.