Sample Size Considerations for Fine-Tuning Large Language Models for Named Entity Recognition Tasks: Methodological Study.

BACKGROUND: Large language models (LLMs) have the potential to support promising new applications in health informatics. However, practical data on sample size considerations for fine-tuning LLMs to perform specific tasks in biomedical and health policy contexts are lacking. OBJECTIVE: This study aims to evaluate sample size and sample selection techniques for fine-tuning LLMs to support improved named entity recognition (NER) for a custom data set of conflicts of interest disclosure statements. METHODS: A random sample of 200 disclosure statements was prepared for annotation. All "PERSON" and "ORG" entities were identified by each of the 2 raters, and once appropriate agreement was established, the annotators independently annotated an additional 290 disclosure statements. From the 490 annotated documents, 2500 stratified random samples in different size ranges were drawn. The 2500 training set subsamples were used to fine-tune a selection of language models across 2 model architectures (Bidirectional Encoder Representations from Transformers [BERT] and Generative Pre-trained Transformer [GPT]) for improved NER, and multiple regression was used to assess the relationship between sample size (sentences), entity density (entities per sentence [EPS]), and trained model performance (F1-score). Additionally, single-predictor threshold regression models were used to evaluate the possibility of diminishing marginal returns from increased sample size or entity density. RESULTS: Fine-tuned models ranged in topline NER performance from F1-score=0.79 to F1-score=0.96 across architectures. Two-predictor multiple linear regression models were statistically significant with multiple R2 ranging from 0.6057 to 0.7896 (all P<.001). EPS and the number of sentences were significant predictors of F1-scores in all cases ( P<.001), except for the GPT-2_large model, where EPS was not a significant predictor (P=.184). Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large. Likewise, the threshold regression models indicate a diminishing marginal return for EPS with point estimates between 1.36 and 1.38. CONCLUSIONS: Relatively modest sample sizes can be used to fine-tune LLMs for NER tasks applied to biomedical text, and training data entity density should representatively approximate entity density in production data. Training data quality and a model architecture's intended use (text generation vs text processing or classification) may be as, or more, important as training data volume and model parameter size.

Political Party Collective Norms, Perceived Norms, and Mask Wearing Behavior: A Test of the Theory of Normative Social Behavior.

The theory of normative social behavior (TNSB) postulates that people are influenced by others' behaviors, which they observe from messages and experience. In addition to focusing on perceived (i.e., descriptive and injunctive) norms, the TNSB was expanded to include collective norms, which represent what people actually do. Testing this expanded theoretical model, the current study examined whether two types of collective norms - collective political norms and collective regional norms - interacted with descriptive norms to influence pandemic mask wearing behavior expectations among U.S. adults (N = 444). The interaction was statistically significant for collective political norms (ß = -.74, p = .009) but not collective regional norms (ß = -.16, p = .85). Specifically, descriptive norms were related to increased mask wearing expectation for all values of political party collective norms, but the effects were stronger when political party collective norms were low (i.e., low mask wearing behavior was normative). The findings support the inclusion of collective norms in the TNSB, clarify the relationships among different types of norms, and provide insights for norms-based interventions.

Signaling and meaning in organizational analytics: coping with Goodhart's Law in an era of digitization and datafication.

The future of work will be measured. The increasing and widespread adoption of analytics, the use of digital inputs and outputs to inform organizational decision making, makes the communication of data central to organizing. This article applies and extends signaling theory to provide a framework for the study of analytics as communication. We report three cases that offer examples of dubious, selective, and ambiguous signaling in the activities of workers seeking to shape the meaning of data within the practice of analytics. The analysis casts the future of work as a game of strategic moves between organizations, seeking to measure behaviors and quantify the performance of work, and workers, altering their behavioral signaling to meet situated goals. The framework developed offers a guide for future examinations of the asymmetric relationship between management and workers as organizations adopt metrics to monitor and evaluate work.

A Content Analysis of Self-Reported Financial Relationships in Biomedical Research.

INTRODUCTION: Financial conflicts of interest (fCOI) present well documented risks to the integrity of biomedical research. However, few studies differentiate among fCOI types in their analyses, and those that do tend to use preexisting taxonomies for fCOI identification. Research on fCOI would benefit from an empirically-derived taxonomy of self-reported fCOI and data on fCOI type and payor prevalence. METHODS: We conducted a content analysis of 6,165 individual self-reported relationships from COI statements distributed across 378 articles indexed with PubMed. Two coders used an iterative coding process to identify and classify individual fCOI types and payors. Inter-rater reliability was κ = 0.935 for fCOI type and κ = 0.884 for payor identification. RESULTS: Our analysis identified 21 fCOI types, 9 of which occurred at prevalences greater than 1%. These included research funding (24.8%), speaking fees (20.8%), consulting fees (18.8%), advisory relationships (11%), industry employment (7.6%), unspecified fees (4.8%), travel fees (3.2%), stock holdings (3.1%), and patent ownership (1%). Reported fCOI were held with 1,077 unique payors, 22 of which were present in more than 1% of financial relationships. The ten most common payors included Pfizer (4%), Novartis (3.9%), MSD (3.8%), Bristol Myers Squibb (3.2%), AstraZeneca (3.1%), GSK (3%), Boehringer Ingelheim (2.9%), Roche (2.8%), Eli LIlly (2.5%), and AbbVie (2.4%). CONCLUSIONS: These results provide novel multi-domain prevalence data on self-reported fCOI and payors in biomedical research. As such, they have the potential to catalyze future research that can assess the differential effects of various types of fCOI. Specifically, the data suggest that comparative analyses of the effects of different fCOI types are needed and that special attention should be paid to the diversity of payor types for research relationships.

Evidence for stratified conflicts of interest policies in research contexts: a methodological review.

OBJECTIVES: The purpose of this study was to conduct a methodological review of research on the effects of conflicts of interest (COIs) in research contexts. DESIGN: Methodological review. DATA SOURCES: Ovid. ELIGIBILITY CRITERIA: Studies published between 1986 and 2021 conducting quantitative assessments of relationships between industry funding or COI and four target outcomes: positive study results, methodological biases, reporting quality and results-conclusions concordance. DATA EXTRACTION AND SYNTHESIS: We assessed key facets of study design: our primary analysis identified whether studies stratified industry funding or COI variables by magnitude (ie, number of COI or disbursement amount), type (employment, travel fees, speaking fees) or if they assessed dichotomous variables (ie, conflict present or absent). Secondary analyses focused on target outcomes and available effects measures. RESULTS: Of the 167 articles included in this study, a substantial majority (98.2%) evaluated the effects of industry sponsorship. None evaluated associations between funding magnitude and outcomes of interest. Seven studies (4.3%) stratified industry funding based on the mechanism of disbursement or funder relationship to product (manufacturer or competitor). A fifth of the articles (19.8%) assessed the effects of author COI on target outcomes. None evaluated COI magnitude, and three studies (9.1%) stratified COI by disbursement type and/or reporting practices. Participation of an industry-employed author showed the most consistent effect on favourability of results across studies. CONCLUSIONS: Substantial evidence demonstrates that industry funding and COI can bias biomedical research. Evidence-based policies are essential for mitigating the risks associated with COI. Although most policies stratify guidelines for managing COI, differentiating COIs based on the type of relationship or monetary value, this review shows that the available research has generally not been designed to assess the differential risks of COI types or magnitudes. Targeted research is necessary to establish an evidence base that can effectively inform policy to manage COI.

Managing Uncertainty for and With Family: Communication Strategies and Motivations in Familial Uncertainty Management for Hereditary Cancer.

The management of uncertainty is integral to health and illness. Individuals manage uncertainty about their health through communication enmeshed in family systems, but existing theorizing focuses on individuals without accounting for family processes. An iterative qualitative analysis of 42 dyadic, family interviews (N = 84) revealed (a) moments in the context of hereditary cancer that involved individual-centered and familial uncertainty appraisal and management, (b) family members' communication strategies to prompt relatives to engage familial uncertainty, and (c) the communicative (re)creation and negotiation of family models for uncertainty management. The findings illuminate tensions that individuals encounter across their lifespan as they appraise and manage uncertainty about hereditary cancer risks. This study extends uncertainty management theory to encompass familial uncertainty management and contributes insights useful for the management of hereditary cancer.

A dashboard for exploring clinical trials sponsorship and potential virtual monopolies.

OBJECTIVE: To create a data visualization dashboard to advance research related to clinical trials sponsorship and monopolistic practices in the pharmaceuticals industry. MATERIALS AND METHODS: This R Shiny application aggregates data from ClinicialTrials.gov resulting from user's queries by terms. Returned data are visualized through an interactive dashboard. RESULTS: The Clinical Trials Sponsorship Network Dashboard (CTSND) uses force-directed network mapping algorithms to visualize clinical trials sponsorship data. Interpretation of network visualization is further supported with data on sponsor classes, sponsorship timelines, evaluated products, and target conditions. The source code for the CTSND is available at https://github.com/sscottgraham/ConflictMetrics. DISCUSSION: Monopolistic practices have been identified as a likely contributor to high drug prices in the United States. CTSND data and visualizations support the analysis of clinical trials sponsorship networks and may aid in identifying current and emerging monopolistic practices. CONCLUSIONS: CTSND data can support more robust deliberation about an understudied area of drug pricing.

Paperwork.

Healthcare today provides an especially rich context for the intertwined transformation of work and the technologies of work, which need to be understood in tandem. Advances in artificial intelligence, robotics, the internet of things, and computational science promise to transform healthcare. The slow speed of organizational and professional change compared to the rapid innovation of healthcare technology makes it a compelling context for engaged scholarship. Sorting through the promise, hype, and reality of the datafication and automation of health and healthcare presents challenges that communication scholarship can help address. In this essay, I share my own healthcare paperwork and information technology story and discuss implications for the study of health information technology, automation, and healthcare work.

"How do we rally around the one who was positive?" Familial uncertainty management in the context of men managing BRCA-related cancer risks.

RATIONALE: Men with BRCA-related cancer risks face increased disease risk as well as the prospect of passing on their risk to children. OBJECTIVE: This study investigates men's communicative appraisal and management of uncertainty related to BRCA-related cancer risks and decision-making. METHODS: Guided by uncertainty management theory (UMT), a directed content analysis approach was utilized to analyze interviews with 25 men who either carry a pathogenic BRCA variant or have a 50% chance of carrying a variant but have not yet been tested. RESULTS: Participants appraised their individual uncertainty as irrelevant or dangerous but appraised their familial uncertainty as dangerous. Men appraising their uncertainty as a danger exhibited more proactive information seeking healthcare behaviors-such as genetic testing and following recommended screenings-than men who appraised their uncertainty as irrelevant. Participants appraised familial uncertainty as a danger and were engaged in information management with family members, as well as encouraging family members to engage in proactive healthcare decision-making. CONCLUSIONS: Men with BRCA-related cancer risks lack understanding about their risks and how to manage them. Increased attention should be paid to the development of interventions tailored specifically to men. Further, interventions focusing on strategically developing proactive family communication behaviors would also be beneficial to men and their families.

"I'd Know That My Child Was Out There": Egg Donation, the Institutionalized "Ideal" Family, and Health Care Decision Making.

This study investigates how young women of egg-donating age perceive egg donation. Using institutional theory, this study demonstrates how participants frame a health care decision, such as egg donation, utilizing familial ideals. Results revealed that women expressed the importance of ownership over their genetic material and that familial ideals encourage an ideal way to create a family, which egg donation only fits as a last resort. Results show that familial ideals reach past the institution of family into broader decision making, such as that of health care. Further, results show that as more families are constructed through assisted reproductive technologies, attempts should be made to gradually alter the familial ideal to encompass novel medical technologies such as egg donation.

"Let's Sit Forward": Investigating Interprofessional Communication, Collaboration, Professional Roles, and Physical Space at EmergiCare.

Communication is key to hospital emergency department (ED) caregiving. Interventions in ED processes (and health care organizing in general) have struggled when they have ignored the professional role expectations that enable and constrain providers with patients and each other. Informed by a communication as design (CAD) approach, this study explored the intersections of professional roles, physical space, and communication at EmergiCare-an academic medical center and level-1 trauma center hospital. Based on an ethnographic analysis of field notes from 70 hours of shadowing at the EmergiCare ED, this study identified two specific communication patterns, "case talk" and "comfort talk," that reflect different logics for communication in health care organizing. The findings indicate (a) that case and comfort talk have different status and therefore different influence in EmergiCare ED interprofessional communication and (b) that the arrangement of physical space at EmergiCare ED reflects the requirements of case talk more so than comfort talk. These findings have important implications for theory and practice, including the importance of considering the macro-discursive construction of professional roles reified in the arrangement of work space.

Avoiding health information.

This study investigated why and how individuals avoid health information to support the development of models of uncertainty and information management and offer insights for those dealing with the information and uncertainty inherent to health and illness. Participants from student (n = 507) and community (n = 418) samples reported that they avoided health information to (a) maintain hope or deniability, (b) resist overexposure, (c) accept limits of action, (d) manage flawed information, (e) maintain boundaries, and (f) continue with life/activities. They also reported strategies for avoiding information, including removing or ignoring stimuli (e.g., avoiding people who might provide health advice) and controlling conversations (e.g., withholding information, changing the subject). Results suggest a link between previous experience with serious illness and health information avoidance. Building on uncertainty management theory, this study demonstrated that health information avoidance is situational, relatively common, not necessarily unhealthy, and may be used to accomplish multiple communication goals.