Large language models encode clinical knowledge.

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Healthcare interactions prior to first hospital admission with alcohol-related liver disease.

BACKGROUND AND AIMS: To examine the healthcare contacts of patients in the year before an index admission to hospital with alcohol-related liver disease (ArLD) to identify where opportunities for earlier identification of alcohol use disorders (AUD) and ArLD and intervention may occur. METHODS: A retrospective cohort study using the regional database encompassing NHS organisations across North West London (344 general practitioner [GP] practices, 4 acute hospital trusts and 2 mental health and community health trusts). Patients who had an index admission with ArLD were identified through healthcare coding and compared with a control cohort. Healthcare contacts, blood tests and AUD testing in the year preceding admission were measured. RESULTS: The ArLD cohort had 1494 participants with an index hospital admission with ArLD. The control cohort included 4462 participants. In the year preceding an index admission with ArLD, 91% of participants had at least one contact with primary care with an average of 2.97 (SD 2.45) contacts; 80% (n = 1199/1494) attended ED, 68% attended an outpatient clinic, and 42% (n = 628/1494) had at least one inpatient admission. Only 9% of the ArLD (137/1494) had formal testing for AUD. Abnormal bilirubin and platelets were more common in the ArLD than the control cohort 25% (138/560) and 28% (231/837), respectively, v 1% (12/1228) and 1% (20/1784). CONCLUSIONS: Prior to an index admission with ArLD patients have numerous interactions with all healthcare settings, indicating missed opportunities for early identification and treatment.

Conventional histological and cytological staining with simultaneous immunohistochemistry enabled by invisible chromogens.

Conventional histological stains, such as hematoxylin plus eosin (H&E), and immunohistochemistry (IHC) are mainstays of histology that provide complementary diagnostic information. H&E and IHC currently require separate slides, because the stains would otherwise obscure one another. This consumes small specimen, limiting the total amount of testing. Additionally, performing H&E and IHC on different slides does not permit comparison of staining at the single cell level, since the same cells are not present on each slide, and alignment of tissue features can be problematic due to changes in tissue landscape with sectioning. We have solved these problems by performing conventional staining and IHC on the same slide using invisible IHC chromogens, such that the chromogens are not visible when viewing the conventional stain and the conventional stain is excluded from images of the IHC. Covalently deposited chromogens provided a convenient route to invisible chromogen design and are stable to reagents used in conventional staining. A dual-camera brightfield microscope system was developed that permits simultaneous viewing of both visible conventional stains and invisible IHC chromogens. Simultaneous staining was demonstrated on several formalin-fixed paraffin-embedded tissue specimens using single and duplex IHC, with chromogens that absorb ultraviolet and near infrared light, followed by H&E staining. The concept was extended to other conventional stains, including mucicarmine special stain and Papanicoulou stain, and further extended to cytology specimens. In addition to interactive video review, images were recorded using multispectral imaging and image processing to provide flexible production of color composite images and enable quantitative analysis.

A large outbreak of COVID-19 in a UK prison, October 2020 to April 2021.

Prisons are susceptible to outbreaks. Control measures focusing on isolation and cohorting negatively affect wellbeing. We present an outbreak of coronavirus disease 2019 (COVID-19) in a large male prison in Wales, UK, October 2020 to April 2021, and discuss control measures.We gathered case-information, including demographics, staff-residence postcode, resident cell number, work areas/dates, test results, staff interview dates/notes and resident prison-transfer dates. Epidemiological curves were mapped by prison location. Control measures included isolation (exclusion from work or cell-isolation), cohorting (new admissions and work-area groups), asymptomatic testing (case-finding), removal of communal dining and movement restrictions. Facemask use and enhanced hygiene were already in place. Whole-genome sequencing (WGS) and interviews determined the genetic relationship between cases plausibility of transmission.Of 453 cases, 53% (n = 242) were staff, most aged 25-34 years (11.5% females, 27.15% males) and symptomatic (64%). Crude attack-rate was higher in staff (29%, 95% CI 26-64%) than in residents (12%, 95% CI 9-15%).Whole-genome sequencing can help differentiate multiple introductions from person-to-person transmission in prisons. It should be introduced alongside asymptomatic testing as soon as possible to control prison outbreaks. Timely epidemiological investigation, including data visualisation, allowed dynamic risk assessment and proportionate control measures, minimising the reduction in resident welfare.

SARS-CoV-2 sero-prevalence in the workforces of three large workplaces in South Wales: a sero-epidemiological study.

BACKGROUND: Sero-prevalence studies quantify the proportion of a population that has antibodies against SARS-CoV-2, and can be used to identify the extent of the COVID-19 pandemic at a population level. The aim of the study was to assess the sero-prevalence of SARS-CoV-2 antibodies in the workforce at three workplaces: a food factory, non-food factory and call-centre. METHODS: Nine hundred ninety-three participants were recruited from three workplaces in South Wales. Participants completed a questionnaire and had a lateral flow point-of-care SARS-CoV-2 antibody test administered by a healthcare professional. The data were analysed using multivariable logistic regression, both using complete records only and following multiple imputation. RESULTS: The sero-prevalence of SARS-CoV-2 antibodies ranged from 4% (n = 17/402) in the non-food factory to 10% (n = 28/281) in the food factory (OR 2.93; 95% CI 1.26 to 6.81). After taking account of confounding factors evidence of a difference remained (cOR comparing food factory to call centre (2.93; 95% CI 1.26 to 6.81) and non-food factory (3.99; 95% CI 1.97 to 8.08) respectively). The SARS-CoV-2 antibody prevalence also varied between roles within workplaces. People working in office based roles had a 2.23 times greater conditional odds (95% CI 1.02 to 4.87) of being positive for SARS-CoV-2 antibodies than those working on the factory floor. CONCLUSION: The sero-prevalence of SARS-CoV-2 antibodies varied by workplace and work role. Whilst it is not possible to state whether these differences are due to COVID-19 transmission within the workplaces, it highlights the importance of considering COVID-19 transmission in a range of workplaces and work roles.

Perioperative cytokine levels portend early death after pancreatectomy for ductal adenocarcinoma.

BACKGROUND: Soluble signaling molecules may play an important role in malignant pathogenesis. We hypothesize that perioperative cytokine levels are associated with outcomes in patients with pancreatic adenocarcinoma (PDAC) undergoing surgical resection. METHODS: One hundered and eighteen patients with benign or malignant pancreatic disease were enrolled in a prospective study through a protocol for banking biologic samples. Peripheral blood was drawn at time of operation, and a multiplex cytokine assay was performed. Statistical analysis was via χ2 and Kaplan Meier methods. RESULTS: Of 118 patients enrolled, 85 (72%) had a diagnosis of PDAC, and 60 (70%) ultimately underwent partial pancreatectomy. Cytokine levels were not associated with postoperative complications in this initial cohort. A plasma level of monocyte chemoattractant protein-1 (MCP-1) pg/mL ≤118 was associated with better overall survival (OS) (median survival 21 months vs 12.8 months, P = 0.023), as was non-detectable interleukin-8 (IL-8) (19 months) versus detectable IL-8 (12.8 months, P = 0.05). Patients with both MCP-1 >118 pg/mL and detectable IL-8 had a median survival of 10.6 months (P = 0.028). CONCLUSIONS: MCP-1 and IL-8 cytokine levels are associated with decreased survival following pancreatectomy for PDAC, and may be useful biomarkers. Measurement of these cytokine levels at different time points in future investigations will be important to validate these findings.

Information Architecture of Web-Based Interventions to Improve Health Outcomes: Systematic Review.

BACKGROUND: The rise in usage of and access to new technologies in recent years has led to a growth in digital health behavior change interventions. As the shift to digital platforms continues to grow, it is increasingly important to consider how the field of information architecture (IA) can inform the development of digital health interventions. IA is the way in which digital content is organized and displayed, which strongly impacts users' ability to find and use content. While many information architecture best practices exist, there is a lack of empirical evidence on the role it plays in influencing behavior change and health outcomes. OBJECTIVE: Our aim was to conduct a systematic review synthesizing the existing literature on website information architecture and its effect on health outcomes, behavioral outcomes, and website engagement. METHODS: To identify all existing information architecture and health behavior literature, we searched articles published in English in the following databases (no date restrictions imposed): ACM Digital Library, CINAHL, Cochrane Library, Google Scholar, Ebsco, and PubMed. The search terms used included information terms (eg, information architecture, interaction design, persuasive design), behavior terms (eg, health behavior, behavioral intervention, ehealth), and health terms (eg, smoking, physical activity, diabetes). The search results were reviewed to determine if they met the inclusion and exclusion criteria created to identify empirical research that studied the effect of IA on health outcomes, behavioral outcomes, or website engagement. Articles that met inclusion criteria were assessed for study quality. Then, data from the articles were extracted using a priori categories established by 3 reviewers. However, the limited health outcome data gathered from the studies precluded a meta-analysis. RESULTS: The initial literature search yielded 685 results, which was narrowed down to three publications that examined the effect of information architecture on health outcomes, behavioral outcomes, or website engagement. One publication studied the isolated impact of information architecture on outcomes of interest (ie, website use and engagement; health-related knowledge, attitudes, and beliefs; and health behaviors), while the other two publications studied the impact of information architecture, website features (eg, interactivity, email prompts, and forums), and tailored content on these outcomes. The paper that investigated IA exclusively found that a tunnel IA improved site engagement and behavior knowledge, but it decreased users' perceived efficiency. The first study that did not isolate IA found that the enhanced site condition improved site usage but not the amount of content viewed. The second study that did not isolate IA found that a tailored site condition improved site usage, behavior knowledge, and some behavior outcomes. CONCLUSIONS: No clear conclusion can be made about the relationship between IA and health outcomes, given limited evidence in the peer-reviewed literature connecting IA to behavioral outcomes and website engagement. Only one study reviewed solely manipulated IA, and we therefore recommend improving the scientific evidence base such that additional empirical studies investigate the impact of IA in isolation. Moreover, information from the gray literature and expert opinion might be identified and added to the evidence base, in order to lay the groundwork for hypothesis generation to improve empirical evidence on information architecture and health and behavior outcomes.

Personal discovery in diabetes self-management: Discovering cause and effect using self-monitoring data.

OBJECTIVE: To outline new design directions for informatics solutions that facilitate personal discovery with self-monitoring data. We investigate this question in the context of chronic disease self-management with the focus on type 2 diabetes. MATERIALS AND METHODS: We conducted an observational qualitative study of discovery with personal data among adults attending a diabetes self-management education (DSME) program that utilized a discovery-based curriculum. The study included observations of class sessions, and interviews and focus groups with the educator and attendees of the program (n = 14). RESULTS: The main discovery in diabetes self-management evolved around discovering patterns of association between characteristics of individuals' activities and changes in their blood glucose levels that the participants referred to as "cause and effect". This discovery empowered individuals to actively engage in self-management and provided a desired flexibility in selection of personalized self-management strategies. We show that discovery of cause and effect involves four essential phases: (1) feature selection, (2) hypothesis generation, (3) feature evaluation, and (4) goal specification. Further, we identify opportunities to support discovery at each stage with informatics and data visualization solutions by providing assistance with: (1) active manipulation of collected data (e.g., grouping, filtering and side-by-side inspection), (2) hypotheses formulation (e.g., using natural language statements or constructing visual queries), (3) inference evaluation (e.g., through aggregation and visual comparison, and statistical analysis of associations), and (4) translation of discoveries into actionable goals (e.g., tailored selection from computable knowledge sources of effective diabetes self-management behaviors). DISCUSSION: The study suggests that discovery of cause and effect in diabetes can be a powerful approach to helping individuals to improve their self-management strategies, and that self-monitoring data can serve as a driving engine for personal discovery that may lead to sustainable behavior changes. CONCLUSIONS: Enabling personal discovery is a promising new approach to enhancing chronic disease self-management with informatics interventions.

Analysing user-reported data for enhancement of SmokefreeTXT: a national text message smoking cessation intervention.

OBJECTIVE: This observational study highlights key insights related to participant engagement and cessation among adults who voluntarily subscribed to the nationwide US-based SmokefreeTXT program, a 42-day mobile phone text message smoking cessation program. METHODS: Point prevalence abstinence rates were calculated for subscribers who initiated treatment in the program (n=18 080). The primary outcomes for this study were treatment completion and point prevalence abstinence rate at the end of the 42-day treatment. Secondary outcomes were point prevalence abstinence rates at 7 days postquit, 3 months post-treatment and 6 months post-treatment, as well as response rates to point prevalence abstinence assessments. RESULTS: Over half the sample completed the 42-day treatment (n=9686). The end-of-treatment point prevalence abstinence for subscribers who initiated treatment was 7.2%. Among those who completed the entire 42 days of treatment, the end-of-treatment point prevalence abstinence was 12.9%. For subscribers who completed treatment, point prevalence abstinence results varied: 7 days postquit (23.7%), 3 months post-treatment (7.3%) and 6 months post-treatment (3.7%). Response rates for abstinence assessment messages ranged from 4.36% to 34.48%. CONCLUSIONS: Findings from this study illuminate the need to more deeply understand reasons for subscriber non-response and opt out and, in turn, improve program engagement and our ability to increase the likelihood for participants to stop smoking and measure long-term outcomes. Patterns of opt out for the program mirror the relapse curve generally observed for smoking cessation, thus highlighting time points at which to increase efforts to retain participants and provide additional support or incentives.

Behavior change interventions: the potential of ontologies for advancing science and practice.

A central goal of behavioral medicine is the creation of evidence-based interventions for promoting behavior change. Scientific knowledge about behavior change could be more effectively accumulated using "ontologies." In information science, an ontology is a systematic method for articulating a "controlled vocabulary" of agreed-upon terms and their inter-relationships. It involves three core elements: (1) a controlled vocabulary specifying and defining existing classes; (2) specification of the inter-relationships between classes; and (3) codification in a computer-readable format to enable knowledge generation, organization, reuse, integration, and analysis. This paper introduces ontologies, provides a review of current efforts to create ontologies related to behavior change interventions and suggests future work. This paper was written by behavioral medicine and information science experts and was developed in partnership between the Society of Behavioral Medicine's Technology Special Interest Group (SIG) and the Theories and Techniques of Behavior Change Interventions SIG. In recent years significant progress has been made in the foundational work needed to develop ontologies of behavior change. Ontologies of behavior change could facilitate a transformation of behavioral science from a field in which data from different experiments are siloed into one in which data across experiments could be compared and/or integrated. This could facilitate new approaches to hypothesis generation and knowledge discovery in behavioral science.

Trajectories of Depressive Symptoms Among Web-Based Health Risk Assessment Participants.

BACKGROUND: Health risk assessments (HRAs), which often screen for depressive symptoms, are administered to millions of employees and health plan members each year. HRA data provide an opportunity to examine longitudinal trends in depressive symptomatology, as researchers have done previously with other populations. OBJECTIVE: The primary research questions were: (1) Can we observe longitudinal trajectories in HRA populations like those observed in other study samples? (2) Do HRA variables, which primarily reflect modifiable health risks, help us to identify predictors associated with these trajectories? (3) Can we make meaningful recommendations for population health management, applicable to HRA participants, based on predictors we identify? METHODS: This study used growth mixture modeling (GMM) to examine longitudinal trends in depressive symptomatology among 22,963 participants in a Web-based HRA used by US employers and health plans. The HRA assessed modifiable health risks and variables such as stress, sleep, and quality of life. RESULTS: Five classes were identified: A "minimal depression" class (63.91%, 14,676/22,963) whose scores were consistently low across time, a "low risk" class (19.89%, 4568/22,963) whose condition remained subthreshold, a "deteriorating" class (3.15%, 705/22,963) who began at subthreshold but approached severe depression by the end of the study, a "chronic" class (4.71%, 1081/22,963) who remained highly depressed over time, and a "remitting" class (8.42%, 1933/22,963) who had moderate depression to start, but crossed into minimal depression by the end. Among those with subthreshold symptoms, individuals who were male (P<.001) and older (P=.01) were less likely to show symptom deterioration, whereas current depression treatment (P<.001) and surprisingly, higher sleep quality (P<.001) were associated with increased probability of membership in the "deteriorating" class as compared with "low risk." Among participants with greater symptomatology to start, those in the "severe" class tended to be younger than the "remitting" class (P<.001). Lower baseline sleep quality (P<.001), quality of life (P<.001), stress level (P<.001), and current treatment involvement (P<.001) were all predictive of membership in the "severe" class. CONCLUSIONS: The trajectories identified were consistent with trends in previous research. The results identified some key predictors: we discuss those that mirror prior studies and offer some hypotheses as to why others did not. The finding that 1 in 5 HRA participants with subthreshold symptoms deteriorated to the point of clinical distress during succeeding years underscores the need to learn more about such individuals. We offer additional recommendations for follow-up research, which should be designed to reflect changes in health plan demographics and HRA delivery platforms. In addition to utilizing additional variables such as cognitive style to refine predictive models, future research could also begin to test the impact of more aggressive outreach strategies aimed at participants who are likely to deteriorate or remain significantly depressed over time.

Case study - human body relay race.

This case presentation introduces the work of Heather Lewis, Graphic Designer from Birmingham Community Healthcare Foundation Trust, Clinical Illustration department. The graphic design team offer professional design solutions in a variety of formats such as scientific posters, banners, patient information booklets and promotional items. This particular project was requested by the Combined Community Dental Service, a Specialist Division in Birmingham.

Current Management of Perihilar Cholangiocarcinoma and Future Perspectives.

Perihilar cholangiocarcinoma is the most common type of biliary tract cancer and is associated with a high mortality, usually due to late presentation. High-resolution cross-sectional imaging modalities are necessary for diagnosis and preoperative planning. Although surgical resection with negative margins offers the only hope for cure, only a small subset of patients are amenable for surgery at the time of diagnosis. Portal vein embolization and biliary tract decompression are important in some patients prior to surgical resection. Liver transplantation in combination with neoadjuvant therapy has resulted in excellent 5-year recurrence-free survival rates in highly selected patients with inoperable disease. Gemcitabine plus cisplatin constitute the backbone of chemotherapy in patients with inoperable metastatic perihilar cholangiocarcinoma. Recent advances in understanding the molecular pathogenesis of CCA have created a growing interest in identifying novel therapies targeting key molecular pathways. Herein, we provide an overview of the most current principles of management of patients with perihilar cholangiocarcinoma.

Linking Nurse Leadership and Work Characteristics to Nurse Burnout and Engagement.

BACKGROUND: Burnout and engagement are critical conditions affecting patient safety and the functioning of healthcare organizations; the areas of worklife model suggest that work environment characteristics may impact employee burnout and general worklife quality. OBJECTIVES: The purpose was to present and test a conditional process model linking perceived transformational nurse leadership to nurse staff burnout and engagement via important work environment characteristics. METHODS: Working nurses (N = 120) provided perceptions of the core study variables via Internet- or paper-based survey. The hypothesized model was tested using the PROCESS analysis tool, which enables simultaneous testing of multiple, parallel, indirect effects within the SPSS statistical package. RESULTS: Findings support the areas of worklife model and suggest that transformational leadership is strongly associated with work environment characteristics that are further linked to nurse burnout and engagement. Interestingly, different work characteristics appear to be critical channels through which transformational leadership impacts nurse burnout and engagement. DISCUSSION: There are several methodological and practical implications of this work for researchers and practitioners interested in preventing burnout and promoting occupational health within healthcare organizations. These implications are tied to the connections observed between transformational leadership, specific work environment characteristics, and burnout and engagement outcomes.

Social Network Behavior and Engagement Within a Smoking Cessation Facebook Page.

BACKGROUND: Social media platforms are increasingly being used to support individuals in behavior change attempts, including smoking cessation. Examining the interactions of participants in health-related social media groups can help inform our understanding of how these groups can best be leveraged to facilitate behavior change. OBJECTIVE: The aim of this study was to analyze patterns of participation, self-reported smoking cessation length, and interactions within the National Cancer Institutes' Facebook community for smoking cessation support. METHODS: Our sample consisted of approximately 4243 individuals who interacted (eg, posted, commented) on the public Smokefree Women Facebook page during the time of data collection. In Phase 1, social network visualizations and centrality measures were used to evaluate network structure and engagement. In Phase 2, an inductive, thematic qualitative content analysis was conducted with a subsample of 500 individuals, and correlational analysis was used to determine how participant engagement was associated with self-reported session length. RESULTS: Between February 2013 and March 2014, there were 875 posts and 4088 comments from approximately 4243 participants. Social network visualizations revealed the moderator's role in keeping the community together and distributing the most active participants. Correlation analyses suggest that engagement in the network was significantly inversely associated with cessation status (Spearman correlation coefficient = -0.14, P=.03, N=243). The content analysis of 1698 posts from 500 randomly selected participants identified the most frequent interactions in the community as providing support (43%, n=721) and announcing number of days smoke free (41%, n=689). CONCLUSIONS: These findings highlight the importance of the moderator for network engagement and provide helpful insights into the patterns and types of interactions participants are engaging in. This study adds knowledge of how the social network of a smoking cessation community behaves within the confines of a Facebook group.

Mobile text messaging for health: a systematic review of reviews.

The aim of this systematic review of reviews is to identify mobile text-messaging interventions designed for health improvement and behavior change and to derive recommendations for practice. We have compiled and reviewed existing systematic research reviews and meta-analyses to organize and summarize the text-messaging intervention evidence base, identify best-practice recommendations based on findings from multiple reviews, and explore implications for future research. Our review found that the majority of published text-messaging interventions were effective when addressing diabetes self-management, weight loss, physical activity, smoking cessation, and medication adherence for antiretroviral therapy. However, we found limited evidence across the population of studies and reviews to inform recommended intervention characteristics. Although strong evidence supports the value of integrating text-messaging interventions into public health practice, additional research is needed to establish longer-term intervention effects, identify recommended intervention characteristics, and explore issues of cost-effectiveness.

Assessing Electronic Cigarette-Related Tweets for Sentiment and Content Using Supervised Machine Learning.

BACKGROUND: Electronic cigarettes (e-cigarettes) continue to be a growing topic among social media users, especially on Twitter. The ability to analyze conversations about e-cigarettes in real-time can provide important insight into trends in the public's knowledge, attitudes, and beliefs surrounding e-cigarettes, and subsequently guide public health interventions. OBJECTIVE: Our aim was to establish a supervised machine learning algorithm to build predictive classification models that assess Twitter data for a range of factors related to e-cigarettes. METHODS: Manual content analysis was conducted for 17,098 tweets. These tweets were coded for five categories: e-cigarette relevance, sentiment, user description, genre, and theme. Machine learning classification models were then built for each of these five categories, and word groupings (n-grams) were used to define the feature space for each classifier. RESULTS: Predictive performance scores for classification models indicated that the models correctly labeled the tweets with the appropriate variables between 68.40% and 99.34% of the time, and the percentage of maximum possible improvement over a random baseline that was achieved by the classification models ranged from 41.59% to 80.62%. Classifiers with the highest performance scores that also achieved the highest percentage of the maximum possible improvement over a random baseline were Policy/Government (performance: 0.94; % improvement: 80.62%), Relevance (performance: 0.94; % improvement: 75.26%), Ad or Promotion (performance: 0.89; % improvement: 72.69%), and Marketing (performance: 0.91; % improvement: 72.56%). The most appropriate word-grouping unit (n-gram) was 1 for the majority of classifiers. Performance continued to marginally increase with the size of the training dataset of manually annotated data, but eventually leveled off. Even at low dataset sizes of 4000 observations, performance characteristics were fairly sound. CONCLUSIONS: Social media outlets like Twitter can uncover real-time snapshots of personal sentiment, knowledge, attitudes, and behavior that are not as accessible, at this scale, through any other offline platform. Using the vast data available through social media presents an opportunity for social science and public health methodologies to utilize computational methodologies to enhance and extend research and practice. This study was successful in automating a complex five-category manual content analysis of e-cigarette-related content on Twitter using machine learning techniques. The study details machine learning model specifications that provided the best accuracy for data related to e-cigarettes, as well as a replicable methodology to allow extension of these methods to additional topics.

Social Listening: A Content Analysis of E-Cigarette Discussions on Twitter.

BACKGROUND: Electronic cigarette (e-cigarette) use has increased in the United States, leading to active debate in the public health sphere regarding e-cigarette use and regulation. To better understand trends in e-cigarette attitudes and behaviors, public health and communication professionals can turn to the dialogue taking place on popular social media platforms such as Twitter. OBJECTIVE: The objective of this study was to conduct a content analysis to identify key conversation trends and patterns over time using historical Twitter data. METHODS: A 5-category content analysis was conducted on a random sample of tweets chosen from all publicly available tweets sent between May 1, 2013, and April 30, 2014, that matched strategic keywords related to e-cigarettes. Relevant tweets were isolated from the random sample of approximately 10,000 tweets and classified according to sentiment, user description, genre, and theme. Descriptive analyses including univariate and bivariate associations, as well as correlation analyses were performed on all categories in order to identify patterns and trends. RESULTS: The analysis revealed an increase in e-cigarette-related tweets from May 2013 through April 2014, with tweets generally being positive; 71% of the sample tweets were classified as having a positive sentiment. The top two user categories were everyday people (65%) and individuals who are part of the e-cigarette community movement (16%). These two user groups were responsible for a majority of informational (79%) and news tweets (75%), compared to reputable news sources and foundations or organizations, which combined provided 5% of informational tweets and 12% of news tweets. Personal opinion (28%), marketing (21%), and first person e-cigarette use or intent (20%) were the three most common genres of tweets, which tended to have a positive sentiment. Marketing was the most common theme (26%), and policy and government was the second most common theme (20%), with 86% of these tweets coming from everyday people and the e-cigarette community movement combined, compared to 5% of policy and government tweets coming from government, reputable news sources, and foundations or organizations combined. CONCLUSIONS: Everyday people and the e-cigarette community are dominant forces across several genres and themes, warranting continued monitoring to understand trends and their implications regarding public opinion, e-cigarette use, and smoking cessation. Analyzing social media trends is a meaningful way to inform public health practitioners of current sentiments regarding e-cigarettes, and this study contributes a replicable methodology.