Exploring the application of machine learning to expert evaluation of research impact.

The objective of this study is to investigate the application of machine learning techniques to the large-scale human expert evaluation of the impact of academic research. Using publicly available impact case study data from the UK's Research Excellence Framework (2014), we trained five machine learning models on a range of qualitative and quantitative features, including institution, discipline, narrative style (explicit and implicit), and bibliometric and policy indicators. Our work makes two key contributions. Based on the accuracy metric in predicting high- and low-scoring impact case studies, it shows that machine learning models are able to process information to make decisions that resemble those of expert evaluators. It also provides insights into the characteristics of impact case studies that would be favoured if a machine learning approach was applied for their automated assessment. The results of the experiments showed strong influence of institutional context, selected metrics of narrative style, as well as the uptake of research by policy and academic audiences. Overall, the study demonstrates promise for a shift from descriptive to predictive analysis, but suggests caution around the use of machine learning for the assessment of impact case studies.

Evolving Document Patterns in UK Research Assessment Cycles.

Researcher behavior is shown to change under assessment. An unexpected time-skew toward most recent papers in each census period was found among the outputs selected by UK academics for the research assessment cycles of the 1990s. This skew changed to a more even time-based distribution for scientists and engineers in later cycles. At the same time, engineers switched their preferred output type for submission, from conference proceedings to journal articles. Social scientists also switched, from monographs to journal art. There was no discussion of these output patterns at the time, or later, but the patterns and their evolution had marked consistency across subjects and institutions. These changes are discussed in terms of consensus and influences on researcher concepts of the evidence of excellence. The increasing availability of citation data in the 1990s and the likely role of citation analysis as a steering factor are noted.

Interpreting Bibliometric Data.

Many academic analyses of good practice in the use of bibliometric data address only technical aspects and fail to account for and appreciate user requirements, expectations, and actual practice. Bibliometric indicators are rarely the only evidence put before any user group. In the present state of knowledge, it is more important to consider how quantitative evaluation can be made simple, transparent, and readily understood than it is to focus unduly on precision, accuracy, or scholarly notions of purity. We discuss how the interpretation of 'performance' from a presentation using accurate but summary bibliometrics can change when iterative deconstruction and visualization of the same dataset is applied. From the perspective of a research manager with limited resources, investment decisions can easily go awry at governmental, funding program, and institutional levels. By exploring select real-life data samples we also show how the specific composition of each dataset can influence interpretive outcomes.

Follow #eHealth2011: Measuring the Role and Effectiveness of Online and Social Media in Increasing the Outreach of a Scientific Conference.

BACKGROUND: Social media promotion is increasingly adopted by organizers of industry and academic events; however, the success of social media strategies is rarely questioned or the real impact scientifically analyzed. OBJECTIVE: We propose a framework that defines and analyses the impact, outreach, and effectiveness of social media for event promotion and research dissemination to participants of a scientific event as well as to the virtual audience through the Web. METHODS: Online communication channels Twitter, Facebook, Flickr, and a Liveblog were trialed and their impact measured on outreach during five phases of an eHealth conference: the setup, active and last-minute promotion phases before the conference, the actual event, and after the conference. RESULTS: Planned outreach through online channels and social media before and during the event reached an audience several magnitudes larger in size than would have been possible using traditional means. In the particular case of eHealth 2011, the outreach using traditional means would have been 74 attendees plus 23 extra as sold proceedings and the number of downloaded articles from the online proceedings (4107 until October 2013). The audience for the conference reached via online channels and social media was estimated at more than 5300 in total during the event. The role of Twitter for promotion before the event was complemented by an increased usage of the website and Facebook during the event followed by a sharp increase of views of posters on Flickr after the event. CONCLUSIONS: Although our case study is focused on a particular audience around eHealth 2011, our framework provides a template for redefining "audience" and outreach of events, merging traditional physical and virtual communities and providing an outline on how these could be successfully reached in clearly defined event phases.