Automatic categorization of self-acknowledged limitations in randomized controlled trial publications.

OBJECTIVE: Acknowledging study limitations in a scientific publication is a crucial element in scientific transparency and progress. However, limitation reporting is often inadequate. Natural language processing (NLP) methods could support automated reporting checks, improving research transparency. In this study, our objective was to develop a dataset and NLP methods to detect and categorize self-acknowledged limitations (e.g., sample size, blinding) reported in randomized controlled trial (RCT) publications. METHODS: We created a data model of limitation types in RCT studies and annotated a corpus of 200 full-text RCT publications using this data model. We fine-tuned BERT-based sentence classification models to recognize the limitation sentences and their types. To address the small size of the annotated corpus, we experimented with data augmentation approaches, including Easy Data Augmentation (EDA) and Prompt-Based Data Augmentation (PromDA). We applied the best-performing model to a set of about 12K RCT publications to characterize self-acknowledged limitations at larger scale. RESULTS: Our data model consists of 15 categories and 24 sub-categories (e.g., Population and its sub-category DiagnosticCriteria). We annotated 1090 instances of limitation types in 952 sentences (4.8 limitation sentences and 5.5 limitation types per article). A fine-tuned PubMedBERT model for limitation sentence classification improved upon our earlier model by about 1.5 absolute percentage points in F1 score (0.821 vs. 0.8) with statistical significance (p<.001). Our best-performing limitation type classification model, PubMedBERT fine-tuning with PromDA (Output View), achieved an F1 score of 0.7, improving upon the vanilla PubMedBERT model by 2.7 percentage points, with statistical significance (p<.001). CONCLUSION: The model could support automated screening tools which can be used by journals to draw the authors' attention to reporting issues. Automatic extraction of limitations from RCT publications could benefit peer review and evidence synthesis, and support advanced methods to search and aggregate the evidence from the clinical trial literature.

CONSORT-TM: Text classification models for assessing the completeness of randomized controlled trial publications.

Objective: To develop text classification models for determining whether the checklist items in the CONSORT reporting guidelines are reported in randomized controlled trial publications. Materials and Methods: Using a corpus annotated at the sentence level with 37 fine-grained CONSORT items, we trained several sentence classification models (PubMedBERT fine-tuning, BioGPT fine-tuning, and in-context learning with GPT-4) and compared their performance. To address the problem of small training dataset, we used several data augmentation methods (EDA, UMLS-EDA, text generation and rephrasing with GPT-4) and assessed their impact on the fine-tuned PubMedBERT model. We also fine-tuned PubMedBERT models limited to checklist items associated with specific sections (e.g., Methods) to evaluate whether such models could improve performance compared to the single full model. We performed 5-fold cross-validation and report precision, recall, F1 score, and area under curve (AUC). Results: Fine-tuned PubMedBERT model that takes as input the sentence and the surrounding sentence representations and uses section headers yielded the best overall performance (0.71 micro-F1, 0.64 macro-F1). Data augmentation had limited positive effect, UMLS-EDA yielding slightly better results than data augmentation using GPT-4. BioGPT fine-tuning and GPT-4 in-context learning exhibited suboptimal results. Methods-specific model yielded higher performance for methodology items, other section-specific models did not have significant impact. Conclusion: Most CONSORT checklist items can be recognized reasonably well with the fine-tuned PubMedBERT model but there is room for improvement. Improved models can underpin the journal editorial workflows and CONSORT adherence checks and can help authors in improving the reporting quality and completeness of their manuscripts.

A structural equation model of the relationship between symptom burden,psychological resilience, coping styles, social support and psychological distress in elderly patients with acute exacerbation chronic obstructive pulmonary disease in China.

PURPOSE: The prevalence of psychological distress is frequently observed among old adults with acute exacerbation of chronic obstructive pulmonary disease (AECOPD). However, current researches are insufficient to clarify the correlation among these relevant factors. This study examined the effects of symptom burden, psychological resilience, coping styles and socialsupporton psychological distress METHODS: 255 elderly patients with AECOPD were conveniently selected in Taian, Shandong Province. The General Information Questionnaire, Distress Thermometer, The Revised Memorial Symptom Assessment Scale, Connor-Davidson Resilience Scale, Simplified Coping Style Questionnaire, Perceived Social Support Scale were used to investigate. The relationship among factors was estimated by using a structural equation model RESULTS: Psychological distress score of elderly patients with AECOPD was (5.25±1.01); coping styles, psychological resilience, symptom burden, social support directly affected psychological distress (the direct effects were -0.934, 0.174, 0.169 and -0.086); coping styles had the largest total effect on psychological distress (the total effect was -0.934); psychological resilience indirectly affected psychological distress through coping styles (the indirect effect was -0.743); symptom burden indirectly affected psychological distress through psychological resilience (the indirect effect was 0.254); social support indirectly affected psychological distress through symptom burden, psychological resilience, and coping styles (the indirect effect was -0.799) CONCLUSION: The psychological distress of elderly patients with AECOPD is at a moderate level; coping styles, psychological resilience and social support have positive effects on alleviating the psychological distress of elderly patients with AECOPD; symptom burden is negatively correlated with psychological distress. Healthcare professionals should pay more attention to elderly patients with AECOPD who are particularly prone to experiencing higher levels of psychological distress, especially in the presence of low coping style, limited psychological resilience, inadequate levels of social support, and high symptom burden.