Applying Large Language Models to Interpret Qualitative Interviews in Healthcare.

To address the persistent challenges in healthcare, it is crucial to incorporate firsthand experiences and perspectives from stakeholders such as patients and healthcare professionals. However, the current process of collecting, analyzing and interpreting qualitative data, such as interviews, is slow and labor-intensive. To expedite this process and enhance efficiency, automated approaches aim to extract meaningful themes and accelerate interpretation, but current approaches such as topic modeling reduce the richness of the raw data. Here, we evaluate whether Large Language Models can be used to support the semi-automated interpretation of qualitative interview data. We compare a novel approach based on LLMs to topic modeling approaches and to manually identified themes across two different qualitative interview datasets. This exploratory study finds that LLMs have the potential to support incorporating human perspectives more widely in the advancement of sustainable healthcare systems.

Trace amine-associated receptor 1 (TAAR1) agonism for psychosis: a living systematic review and meta-analysis of human and non-human data.

Background: Trace amine-associated receptor 1 (TAAR1) agonism shows promise for treating psychosis, prompting us to synthesise data from human and non-human studies. Methods: We co-produced a living systematic review of controlled studies examining TAAR1 agonists in individuals (with or without psychosis/schizophrenia) and relevant animal models. Two independent reviewers identified studies in multiple electronic databases (until 17.11.2023), extracted data, and assessed risk of bias. Primary outcomes were standardised mean differences (SMD) for overall symptoms in human studies and hyperlocomotion in animal models. We also examined adverse events and neurotransmitter signalling. We synthesised data with random-effects meta-analyses. Results: Nine randomised trials provided data for two TAAR1 agonists (ulotaront and ralmitaront), and 15 animal studies for 10 TAAR1 agonists. Ulotaront and ralmitaront demonstrated few differences compared to placebo in improving overall symptoms in adults with acute schizophrenia (N=4 studies, n=1291 participants; SMD=0.15, 95%CI: -0.05, 0.34), and ralmitaront was less efficacious than risperidone (N=1, n=156, SMD=-0.53, 95%CI: -0.86, -0.20). Large placebo response was observed in ulotaront phase-III trials. Limited evidence suggested a relatively benign side-effect profile for TAAR1 agonists, although nausea and sedation were common after a single dose of ulotaront. In animal studies, TAAR1 agonists improved hyperlocomotion compared to control (N=13 studies, k=41 experiments, SMD=1.01, 95%CI: 0.74, 1.27), but seemed less efficacious compared to dopamine D 2 receptor antagonists (N=4, k=7, SMD=-0.62, 95%CI: -1.32, 0.08). Limited human and animal data indicated that TAAR1 agonists may regulate presynaptic dopaminergic signalling. Conclusions: TAAR1 agonists may be less efficacious than dopamine D 2 receptor antagonists already licensed for schizophrenia. The results are preliminary due to the limited number of drugs examined, lack of longer-term data, publication bias, and assay sensitivity concerns in trials associated with large placebo response. Considering their unique mechanism of action, relatively benign side-effect profile and ongoing drug development, further research is warranted. Registration: PROSPERO-ID: CRD42023451628.

There is a need for more effective treatments for psychosis, including schizophrenia. Psychosis is a collection of mental health symptoms, such as hearing voices, that can cause distress and impair functioning. These symptoms are thought to be caused by changes in a chemical messenger system in the brain called dopamine. Currently used antipsychotic medications target brain receptors that respond to dopamine. They are not effective in some people and can cause uncomfortable adverse events, such as weight gain and movement disorders, especially with long-term use. A new type of drug is the trace amine-associated receptor 1 (TAAR1) agonists. These drugs act on different brain receptors that can affect the activity of the dopamine system, but do not directly bind to dopamine receptors. We aimed to understand if TAAR1 agonists can reduce symptoms of psychosis, what adverse events they might have, and how they work. We did this by reviewing and collating all available evidence until November 2023. This is a "living" systematic review, so it will be regularly updated in the future. We looked at both human and animal studies investigating TAAR1 agonists. Human studies suggested that two TAAR1 agonists (namely, ulotaront or ralmitaront) might have little to no effect on reducing symptoms of psychosis compared to placebo in people with schizophrenia. They seemed to cause fewer adverse events than current antipsychotics. Data from animal studies suggested that TAAR1 agonists had some positive effects but potentially smaller than other antipsychotics. There were little to no data from both human and animal studies about how TAAR1 agonists actually work. From the current evidence we are uncertain about these results. With the ongoing development of new TAAR1 agonists, more evidence is needed to understand their potential role in the treatment of psychosis.

The Paradoxes of Digital Tools in Hospitals: Qualitative Interview Study.

BACKGROUND: Digital tools are progressively reshaping the daily work of health care professionals (HCPs) in hospitals. While this transformation holds substantial promise, it leads to frustrating experiences, raising concerns about negative impacts on clinicians' well-being. OBJECTIVE: The goal of this study was to comprehensively explore the lived experiences of HCPs navigating digital tools throughout their daily routines. METHODS: Qualitative in-depth interviews with 52 HCPs representing 24 medical specialties across 14 hospitals in Switzerland were performed. RESULTS: Inductive thematic analysis revealed 4 main themes: digital tool use, workflow and processes, HCPs' experience of care delivery, and digital transformation and management of change. Within these themes, 6 intriguing paradoxes emerged, and we hypothesized that these paradoxes might partly explain the persistence of the challenges facing hospital digitalization: the promise of efficiency and the reality of inefficiency, the shift from face to face to interface, juggling frustration and dedication, the illusion of information access and trust, the complexity and intersection of workflows and care paths, and the opportunities and challenges of shadow IT. CONCLUSIONS: Our study highlights the central importance of acknowledging and considering the experiences of HCPs to support the transformation of health care technology and to avoid or mitigate any potential negative experiences that might arise from digitalization. The viewpoints of HCPs add relevant insights into long-standing informatics problems in health care and may suggest new strategies to follow when tackling future challenges.

Electronic cigarettes and subsequent use of cigarettes in young people: An evidence and gap map.

BACKGROUND AND AIMS: The use of e-cigarettes may influence later smoking uptake in young people. Evidence and gap maps (EGMs) are interactive on-line tools that display the evidence and gaps in a specific area of policy or research. The aim of this study was to map clusters and gaps in evidence exploring the relationship between e-cigarette use or availability and subsequent combustible tobacco use in people aged < 30 years. METHODS: We conducted an EGM of primary studies and systematic reviews. A framework and an interactive EGM was developed in consultation with an expert advisory group. A systematic search of five databases retrieved 9057 records, from which 134 studies were included. Systematic reviews were appraised using AMSTAR-2, and all included studies were coded into the EGM framework resulting in the interactive web-based EGM. A descriptive analysis of key characteristics of the identified evidence clusters and gaps resulted in this report. RESULTS: Studies were completed between 2015 and 2023, with the first systematic reviews being published in 2017. Most studies were conducted in western high-income countries, predominantly the United States. Cohort studies were the most frequently used study design. The evidence is clustered on e-cigarette use as an exposure, with an absolute gap identified for evidence looking into the availability of e-cigarettes and subsequent cessation of cigarette smoking. We also found little evidence analysing equity factors, and little exploring characteristics of e-cigarette devices. CONCLUSIONS: This evidence and gap map (EGM) offers a tool to explore the available evidence regarding the e-cigarette use/availability and later cigarette smoking in people under the age of 30 years at the time of the search. The majority of the 134 reports is from high-income countries, with an uneven geographic distribution. Most of the systematic reviews are of lower quality, suggesting the need for higher-quality reviews. The evidence is clustered around e-cigarette use as an exposure and subsequent frequency/intensity of current combustible tobacco use. Gaps in evidence focusing on e-cigarette availability, as well as on the influence of equity factors may warrant further research. This EGM can support funders and researchers in identifying future research priorities, while guiding practitioners and policymakers to the current evidence base.

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain.

BACKGROUND: Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS: LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS: The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS: LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.

Chebifier: automating semantic classification in ChEBI to accelerate data-driven discovery.

Connecting chemical structural representations with meaningful categories and semantic annotations representing existing knowledge enables data-driven digital discovery from chemistry data. Ontologies are semantic annotation resources that provide definitions and a classification hierarchy for a domain. They are widely used throughout the life sciences. ChEBI is a large-scale ontology for the domain of biologically interesting chemistry that connects representations of chemical structures with meaningful chemical and biological categories. Classifying novel molecular structures into ontologies such as ChEBI has been a longstanding objective for data scientific methods, but the approaches that have been developed to date are limited in several ways: they are not able to expand as the ontology expands without manual intervention, and they are not able to learn from continuously expanding data. We have developed an approach for automated classification of chemicals in the ChEBI ontology based on a neuro-symbolic AI technique that harnesses the ontology itself to create the learning system. We provide this system as a publicly available tool, Chebifier, and as an API, ChEB-AI. We here evaluate our approach and show how it constitutes an advance towards a continuously learning semantic system for chemical knowledge discovery.

Proteomic profiling for biomarker discovery in heparin-induced thrombocytopenia.

ABSTRACT: New analytical techniques can assess hundreds of proteins simultaneously with high sensitivity, facilitating the observation of their complex interplay and role in disease mechanisms. We hypothesized that proteomic profiling targeting proteins involved in thrombus formation, inflammation, and the immune response would identify potentially new biomarkers for heparin-induced thrombocytopenia (HIT). Four existing panels of the Olink proximity extension assay covering 356 proteins involved in thrombus formation, inflammation, and immune response were applied to randomly selected patients with suspected HIT (confirmed HIT, n = 32; HIT ruled out, n = 38; and positive heparin/platelet factor 4 [H/PF4] antibodies, n = 28). The relative difference in protein concentration was analyzed using a linear regression model adjusted for sex and age. To confirm the test results, soluble P-selectin was determined using enzyme-linked immunosorbent assay (ELISA) in above mentioned patients and an additional second data set (n = 49). HIT was defined as a positive heparin-induced platelet activation assay (washed platelet assay). Among 98 patients of the primary data set, the median 4Ts score was 5 in patients with HIT, 4 in patients with positive H/PF4 antibodies, and 3 in patients without HIT. The median optical density of a polyspecific H/PF4 ELISA were 3.0, 0.9, and 0.3. Soluble P-selectin remained statistically significant after multiple test adjustments. The area under the receiver operating characteristic curve was 0.81 for Olink and 0.8 for ELISA. Future studies shall assess the diagnostic and prognostic value of soluble P-selectin in the management of HIT.

Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology.

Purpose: Technological progress of machine learning and natural language processing has led to the development of large language models (LLMs), capable of producing well-formed text responses and providing natural language access to knowledge. Modern conversational LLMs such as ChatGPT have shown remarkable capabilities across a variety of fields, including medicine. These models may assess even highly specialized medical knowledge within specific disciplines, such as radiation therapy. We conducted an exploratory study to examine the capabilities of ChatGPT to answer questions in radiation therapy. Methods and Materials: A set of multiple-choice questions about clinical, physics, and biology general knowledge in radiation oncology as well as a set of open-ended questions were created. These were given as prompts to the LLM ChatGPT, and the answers were collected and analyzed. For the multiple-choice questions, it was checked how many of the answers of the model could be clearly assigned to one of the allowed multiple-choice-answers, and the proportion of correct answers was determined. For the open-ended questions, independent blinded radiation oncologists evaluated the quality of the answers regarding correctness and usefulness on a 5-point Likert scale. Furthermore, the evaluators were asked to provide suggestions for improving the quality of the answers. Results: For 70 multiple-choice questions, ChatGPT gave valid answers in 66 cases (94.3%). In 60.61% of the valid answers, the selected answer was correct (50.0% of clinical questions, 78.6% of physics questions, and 58.3% of biology questions). For 25 open-ended questions, 12 answers of ChatGPT were considered as "acceptable," "good," or "very good" regarding both correctness and helpfulness by all 6 participating radiation oncologists. Overall, the answers were considered "very good" in 29.3% and 28%, "good" in 28% and 29.3%, "acceptable" in 19.3% and 19.3%, "bad" in 9.3% and 9.3%, and "very bad" in 14% and 14% regarding correctness/helpfulness. Conclusions: Modern conversational LLMs such as ChatGPT can provide satisfying answers to many relevant questions in radiation therapy. As they still fall short of consistently providing correct information, it is problematic to use them for obtaining medical information. As LLMs will further improve in the future, they are expected to have an increasing impact not only on general society, but also on clinical practice, including radiation oncology.

Experience of Health Care Professionals Using Digital Tools in the Hospital: Qualitative Systematic Review.

BACKGROUND: The digitalization of health care has many potential benefits, but it may also negatively impact health care professionals' well-being. Burnout can, in part, result from inefficient work processes related to the suboptimal implementation and use of health information technologies. Although strategies to reduce stress and mitigate clinician burnout typically involve individual-based interventions, emerging evidence suggests that improving the experience of using health information technologies can have a notable impact. OBJECTIVE: The aim of this systematic review was to collect evidence of the benefits and challenges associated with the use of digital tools in hospital settings with a particular focus on the experiences of health care professionals using these tools. METHODS: We conducted a systematic literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to explore the experience of health care professionals with digital tools in hospital settings. Using a rigorous selection process to ensure the methodological quality and validity of the study results, we included qualitative studies with distinct data that described the experiences of physicians and nurses. A panel of 3 independent researchers performed iterative data analysis and identified thematic constructs. RESULTS: Of the 1175 unique primary studies, we identified 17 (1.45%) publications that focused on health care professionals' experiences with various digital tools in their day-to-day practice. Of the 17 studies, 10 (59%) focused on clinical decision support tools, followed by 6 (35%) studies focusing on electronic health records and 1 (6%) on a remote patient-monitoring tool. We propose a theoretical framework for understanding the complex interplay between the use of digital tools, experience, and outcomes. We identified 6 constructs that encompass the positive and negative experiences of health care professionals when using digital tools, along with moderators and outcomes. Positive experiences included feeling confident, responsible, and satisfied, whereas negative experiences included frustration, feeling overwhelmed, and feeling frightened. Positive moderators that may reinforce the use of digital tools included sufficient training and adequate workflow integration, whereas negative moderators comprised unfavorable social structures and the lack of training. Positive outcomes included improved patient care and increased workflow efficiency, whereas negative outcomes included increased workload, increased safety risks, and issues with information quality. CONCLUSIONS: Although positive and negative outcomes and moderators that may affect the use of digital tools were commonly reported, the experiences of health care professionals, such as their thoughts and emotions, were less frequently discussed. On the basis of this finding, this study highlights the need for further research specifically targeting experiences as an important mediator of clinician well-being. It also emphasizes the importance of considering differences in the nature of specific tools as well as the profession and role of individual users. TRIAL REGISTRATION: PROSPERO CRD42023393883; https://tinyurl.com/2htpzzxj.

The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1.

Background: The Behaviour Change Technique Taxonomy v1 (BCTTv1) specifies the potentially active content of behaviour change interventions. Evaluation of BCTTv1 showed the need to extend it into a formal ontology, improve its labels and definitions, add BCTs and subdivide existing BCTs. We aimed to develop a Behaviour Change Technique Ontology (BCTO) that would meet these needs. Methods: The BCTO was developed by: (1) collating and synthesising feedback from multiple sources; (2) extracting information from published studies and classification systems; (3) multiple iterations of reviewing and refining entities, and their labels, definitions and relationships; (4) refining the ontology via expert stakeholder review of its comprehensiveness and clarity; (5) testing whether researchers could reliably apply the ontology to identify BCTs in intervention reports; and (6) making it available online and creating a machine-readable version. Results: Initially there were 282 proposed changes to BCTTv1. Following first-round review, 19 BCTs were split into two or more BCTs, 27 new BCTs were added and 26 BCTs were moved into a different group, giving 161 BCTs hierarchically organised into 12 logically defined higher-level groups in up to five hierarchical levels. Following expert stakeholder review, the refined ontology had 247 BCTs hierarchically organised into 20 higher-level groups. Independent annotations of intervention evaluation reports by researchers familiar and unfamiliar with the ontology resulted in good levels of inter-rater reliability (0.82 and 0.79, respectively). Following revision informed by this exercise, 34 BCTs were added, resulting in a final version of the BCTO containing 281 BCTs organised into 20 higher-level groups over five hierarchical levels. Discussion: The BCT Ontology provides a standard terminology and comprehensive classification system for the content of behaviour change interventions that can be reliably used to describe interventions.

Human Factors Influencing the Experience of Healthcare Professionals Using Digital Tools.

Digitalization in healthcare has the potential to offer numerous advantages to various stakeholders, however, healthcare professionals often encounter difficulties while using digital tools. We conducted a qualitative analysis of published studies to examine the experience of clinicians using digital tools. Our findings revealed that human factors influence clinicians' experiences and that integration of human factors into the design and development of healthcare technologies is of high importance to improve user experience and overall success.

New living evidence resource of human and non-human studies for early intervention and research prioritisation in anxiety, depression and psychosis.

In anxiety, depression and psychosis, there has been frustratingly slow progress in developing novel therapies that make a substantial difference in practice, as well as in predicting which treatments will work for whom and in what contexts. To intervene early in the process and deliver optimal care to patients, we need to understand the underlying mechanisms of mental health conditions, develop safe and effective interventions that target these mechanisms, and improve our capabilities in timely diagnosis and reliable prediction of symptom trajectories. Better synthesis of existing evidence is one way to reduce waste and improve efficiency in research towards these ends. Living systematic reviews produce rigorous, up-to-date and informative evidence summaries that are particularly important where research is emerging rapidly, current evidence is uncertain and new findings might change policy or practice. Global Alliance for Living Evidence on aNxiety, depressiOn and pSychosis (GALENOS) aims to tackle the challenges of mental health science research by cataloguing and evaluating the full spectrum of relevant scientific research including both human and preclinical studies. GALENOS will also allow the mental health community-including patients, carers, clinicians, researchers and funders-to better identify the research questions that most urgently need to be answered. By creating open-access datasets and outputs in a state-of-the-art online resource, GALENOS will help identify promising signals early in the research process. This will accelerate translation from discovery science into effective new interventions for anxiety, depression and psychosis, ready to be translated in clinical practice across the world.

Toward an ontology of identity-related constructs in addiction, with examples from nicotine and tobacco research.

BACKGROUND AND AIMS: We aimed to create a basic set of definitions and relationships for identity-related constructs, as part of the Addiction Ontology and E-Cigarette Ontology projects, that could be used by researchers with diverse theoretical positions and so facilitate evidence synthesis and interoperability. METHODS: We reviewed the use of identity-related constructs in psychological and social sciences and how these have been applied to addiction with a focus on nicotine and tobacco research. We, then, used an iterative process of adaptation and review to arrive at a basic set of identity-related classes with labels, definitions and relationships that could provide a common framework for research. RESULTS: We propose that 'identity' be used to refer to 'a cognitive representation by a person or group of themselves', with 'self-identity' referring to an individual's identity and 'group identity' referring to an identity held by a social group. Identities can then be classified at any level of granularity based on the content of the representations (e.g. 'tobacco smoker identity', 'cigarette smoker identity' and 'vaper identity'). We propose distinguishing identity from 'self-appraisal' to capture the distinction between the representation of oneself (e.g. as an 'ex-smoker') and (i) the importance and (ii) the positive or negative evaluation that we attach to what is represented. We label an identity that is appraised as enduring as a 'core identity', related to 'strong identity' because of the appraisal as important. Identities that are appraised positively or negatively involve 'positive self-appraisal' and 'negative self-appraisal' respectively. This allows us to create 'logically defined classes' of identity by combining them (e.g. 'positive core cigarette smoker identity' to refer to a cigarette smoker self-identity that is both positive and important). We refer to the totality of self-identities of a person as a 'composite self-identity'. CONCLUSIONS: An ontology of identity constructs may assist in improving clarity when discussing theories and evidence relating to this construct in addiction research.

Toward an ontology of tobacco, nicotine and vaping products.

BACKGROUND AND AIMS: Ontologies are ways of representing information that improve clarity and the ability to connect different data sources. This paper proposes an initial version of an ontology of tobacco, nicotine and vaping products with the aim of reducing ambiguity and confusion in the field. METHODS: Terms related to tobacco, nicotine and vaping products were identified in the research literature and their usage characterised. Basic Formal Ontology was used as a unifying upper-level ontology to describe the domain, and classes with definitions and labels were developed linking them to this ontology. Labels, definitions and properties were reviewed and revised in an iterative manner until a coherent set of classes was agreed by the authors. RESULTS: Overlapping, but distinct classes were developed: 'tobacco-containing product', 'nicotine-containing product' and 'vaping device'. Subclasses of tobacco-containing products are 'combustible tobacco-containing product', 'heated tobacco product' and 'smokeless tobacco-containing product'. Subclasses of combustible tobacco-containing product include 'cigar', 'cigarillo', 'bidi' and 'cigarette' with further subclasses including 'manufactured cigarette'. Manufactured cigarettes have properties that include 'machine-smoked nicotine yield' and 'machine-smoked tar yield'. Subclasses of smokeless tobacco product include 'nasal snuff', 'chewing tobacco product', and 'oral snuff' with its subclass 'snus'. Subclasses of nicotine-containing product include 'nicotine lozenge' and 'nicotine transdermal patch'. Subclasses of vaping device included 'electronic vaping device' with a further subclass, 'e-cigarette'. E-cigarettes have evolved with a complex range of properties including atomiser resistance, battery power, properties of consumables including e-liquid nicotine concentration and flavourings, and the ontology characterises classes of product accordingly. CONCLUSIONS: Use of an ontology of tobacco, nicotine and vaping products should help reduce ambiguity and confusion in tobacco control research and practice.

Specifying how intervention content is communicated: Development of a Style of Delivery Ontology.

Background: Investigating and enhancing the effectiveness of behaviour change interventions requires detailed and consistent specification of all aspects of interventions. We need to understand not only their content, that is the specific techniques, but also the source, mode, schedule, and style in which this content is delivered. Delivery style refers to the manner by which content is communicated to intervention participants. This paper reports the development of an ontology for specifying the style of delivery of interventions that depend on communication. This forms part of the Behaviour Change Intervention Ontology, which aims to cover all aspects of behaviour change intervention scenarios. Methods: The Style of Delivery Ontology was developed following methods for ontology development used in the Human Behaviour-Change Project, with seven key steps: 1) defining the scope of the ontology, 2) identifying key entities and developing their preliminary definitions by reviewing 100 behaviour change intervention evaluation reports and existing classification systems, 3) refining the ontology by piloting the ontology through annotations of 100 reports, 4) stakeholder review by eight behavioural science and public health experts, 5) inter-rater reliability testing through annotating 100 reports using the ontology, 6) specifying ontological relationships between entities, and 7) disseminating and maintaining the ontology. Results: The resulting ontology is a five-level hierarchical structure comprising 145 unique entities relevant to style of delivery. Key areas include communication processes, communication styles, and attributes of objects used in communication processes. Inter-rater reliability for annotating intervention evaluation reports was α=0.77 (good) for those familiar with the ontology and α=0.62 (acceptable) for those unfamiliar with it. Conclusions: The Style of Delivery Ontology can be used for both annotating and describing behaviour change interventions in a consistent and coherent manner, thereby improving evidence comparison, synthesis, replication, and implementation of effective interventions.

An ontology of mechanisms of action in behaviour change interventions.

Background: Behaviour change interventions influence behaviour through causal processes called "mechanisms of action" (MoAs). Reports of such interventions and their evaluations often use inconsistent or ambiguous terminology, creating problems for searching, evidence synthesis and theory development. This inconsistency includes the reporting of MoAs. An ontology can help address these challenges by serving as a classification system that labels and defines MoAs and their relationships. The aim of this study was to develop an ontology of MoAs of behaviour change interventions. Methods: To develop the MoA Ontology, we (1) defined the ontology's scope; (2) identified, labelled and defined the ontology's entities; (3) refined the ontology by annotating (i.e., coding) MoAs in intervention reports; (4) refined the ontology via stakeholder review of the ontology's comprehensiveness and clarity; (5) tested whether researchers could reliably apply the ontology to annotate MoAs in intervention evaluation reports; (6) refined the relationships between entities; (7) reviewed the alignment of the MoA Ontology with other relevant ontologies, (8) reviewed the ontology's alignment with the Theories and Techniques Tool; and (9) published a machine-readable version of the ontology. Results: An MoA was defined as "a process that is causally active in the relationship between a behaviour change intervention scenario and its outcome behaviour". We created an initial MoA Ontology with 261 entities through Steps 2-5. Inter-rater reliability for annotating study reports using these entities was α=0.68 ("acceptable") for researchers familiar with the ontology and α=0.47 for researchers unfamiliar with it. As a result of additional revisions (Steps 6-8), 23 further entities were added to the ontology resulting in 284 entities organised in seven hierarchical levels. Conclusions: The MoA Ontology extensively captures MoAs of behaviour change interventions. The ontology can serve as a controlled vocabulary for MoAs to consistently describe and synthesise evidence about MoAs across diverse sources.

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project.

Background  Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods  Researchers manually annotated 70 items of information ('entities') in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the 'FLAIR' framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results  The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05-0.88) compared with an average human annotator's score of 0.75 (SD=0.15, range 0.38-1.00). The algorithm for assigning entities to study arms ( e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions  While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper 3).