Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses.

BACKGROUND: Systematic reviews are performed manually despite the exponential growth of scientific literature. OBJECTIVE: To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews. DESIGN: Diagnostic test accuracy study. SETTING: Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations. PARTICIPANTS: None. MEASUREMENTS: A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened. RESULTS: Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level. LIMITATIONS: Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time. CONCLUSION: The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level. PRIMARY FUNDING SOURCE: None.

Discordances in patients' and physicians' perspectives on which contextual factors should be accounted for during smoking cessation.

BACKGROUND: Smoking cessation interventions requires attending to the circumstances and needs of individual patients. We aimed at highlighting the discordances between patients' and physicians' perspectives on contextual factors that should be considered during smoking cessation. METHODS: We identified 36 contextual factors identified that should be considered during smoking cessation using PubMed and interviewing general practitioners. Physicians recruited through social networks campaigns and smoker or former smoker patients from the ComPaRe cohort selected the factors they considered most relevant in two online paired comparison experiment. Bradley Terry Luce models estimated the ability of each factor (i.e. the probability to be preferred). We calculated the Pearson's correlation and the intraclass correlation coefficients for the contextual factor from each perspective and compared the ranking of the 10 contextual factors with the highest abilities. RESULTS: Seven hundred and ninety-three patients' and 795 physicians' perspectives estimated the ability (i.e., importance) of the contextual factors in 11 963 paired comparisons. We found a high correlation between physicians' and patients' perspectives of the contextual factors to be considered for smoking cessation (r = 0.76, P < 0.0001). However, the agreement between the abilities of contextual factors was poor (ICC = 0.42 [-0.10; 0.75]; P = 0.09). Fine-grain analysis of participants' answers revealed many discrepancies. For example, 40% factors ranked in the top 10 most important for physicians were not in patients' top 10 ranking. CONCLUSION: Our results highlight the importance of patient-centered care, the need to engage discussions about patients' values, beyond what is thought to be important, to avoid overlooking their real context.

Psychometric properties and domains covered by patient-reported outcome measures used in trials assessing interventions for chronic pain.

OBJECTIVES: To identify the patient-reported outcome measures (PROMs) used in clinical trials assessing interventions for chronic pain, describe their psychometric properties, and the clinical domains they cover. STUDY DESIGN AND SETTING: We identified phase 3 or 4 interventional trials: on adult participants (aged >18 years), registered in clinicaltrials.gov between January 1, 2021 and December 31, 2022, and which provided "chronic pain" as a keyword condition. We excluded diagnostic studies and phase 1 or 2 trials. In each trial, one reviewer extracted all outcomes registered and identified those captured using PROMs. For each PROM used in more than 1% of identified trials, two reviewers assessed whether it covered the six important clinical domains from the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT): pain, emotional functioning, physical functioning, participant ratings of global improvement and satisfaction with treatment, symptoms and adverse events, and participant disposition (eg, adherence to medication). Second, reviewers searched PubMed for both the initial publication and latest review reporting the psychometric properties of each PROM and assessed their content validity, structural validity, internal consistency, reliability, measurement error, hypotheses testing, criterion validity, and responsiveness using published criteria from the literature. RESULTS: In total, 596 trials assessing 4843 outcomes were included in the study (median sample size 60, interquartile range 40-100). Trials evaluated behavioral (22%), device-based (21%), and drug-based (10%) interventions. Of 495 unique PROMs, 55 were used in more than 1% trials (16 were generic pain measures; 8 were pain measures for specific diseases; and 30 were measures of other symptoms or consequences of pain). About 50% PROMs had more than 50% of psychometric properties rated as sufficient. Scales often focused on a single clinical domain. Only 25% trials measured at least three clinical domains from IMMPACT. CONCLUSION: Half of PROMs used in trials assessing interventions for chronic pain had sufficient psychometric properties for more than 50% of criteria assessed. Few PROMs assessed more than one important clinical domain. Only 25% of trials measured more than 3/6 clinical domains considered important by IMMPACT.

Preferred study designs to support a comparative therapeutic strategy question in oncology: a vignette study.

INTRODUCTION: Some therapeutic strategy questions in oncology could be answered with studies using observational data. Target trial emulation is the application of design principles from randomized controlled trials to the analysis of observational data, to reduce design-induced biases. Our objective was to determine which type of study physicians would preferably plan to answer a comparative effectiveness question lacking evidence in oncology. METHODS: We launched an online survey among physicians specialized in oncology. We constructed a vignette-based inquiry where vignettes described study scenarios which could be conducted to answer the predefined question. We designed six vignettes described by study design (randomized controlled trial or observational study with a trial emulation framework), main study characteristics, probability of the study succeeding and anticipated delay before results availability. Participants randomly assessed five pair-wise comparisons of the vignettes and were asked which study they would preferably plan by using a Likert scale. The main outcome was the evaluation of clinicians' preferences for each pairwise comparison. Mean and median preference scores were calculated. RESULTS: 213 participants, specialized in many tumor types, assessed at least one comparison with 82% reporting France as their country of affiliation. The interquartile range was -4 to 4 across pairwise comparisons. The median preference score was in disfavor of the monocentric randomized controlled trial for the five comparisons where it appeared. The median preference score was strongly in favor of the multicentric national emulated trial when compared to the monocentric emulated trial 4 [IQ 2.5-4]. The mean preference score was the highest for the large European observational study 1.14 (SD 3.33), while the mean preference score was the lowest for the monocentric randomized controlled trial -1.86 (SD 2.93). CONCLUSION: No study design was strongly preferred, but the monocentric randomized controlled trial was the least favored study in pair-wise comparisons. The planification of the new research is a compromise between scientific soundness, feasibility, cost, and time before obtaining results. We need to have the right answers to the right questions at the right time.

Rationale and design of the FRENch CoHort of myocardial Infarction Evaluation (FRENCHIE) study.

BACKGROUND: Despite major advances in prevention and treatment, cardiovascular diseases - particularly acute myocardial infarction - remain a leading cause of death worldwide and in France. Collecting contemporary data about the characteristics, management and outcomes of patients with acute myocardial infarction in France is important. AIMS: The main objectives are to describe baseline characteristics, contemporary management, in-hospital and long-term outcomes of patients with acute myocardial infarction hospitalized in tertiary care centres in France; secondary objectives are to investigate determinants of prognosis (including periodontal disease and sleep-disordered breathing), to identify gaps between evidence-based recommendations and management and to assess medical care costs for the index hospitalization and during the follow-up period. METHODS: FRENCHIE (FRENch CoHort of myocardial Infarction Evaluation) is an ongoing prospective multicentre observational study (ClinicalTrials.gov Identifier: NCT04050956) enrolling more than 19,000 patients hospitalized for acute myocardial infarction with onset of symptoms within 48hours in 35 participating centres in France since March 2019. Main exclusion criteria are age<18 years, lack of health coverage and procedure-related myocardial infarction (types 4a and 5). Detailed information was collected prospectively, starting at admission, including demographic data, risk factors, medical history and treatments, initial management, with prehospital care pathways and medication doses, and outcomes until hospital discharge. The follow-up period (up to 20 years for each patient) is ensured by linking with the French national health database (Système national des données de santé), and includes information on death, hospital admissions, major clinical events, healthcare consumption (including drug reimbursement) and total healthcare costs. FRENCHIE is also used as a platform for cohort-nested studies - currently three randomized trials and two observational studies. CONCLUSIONS: This nationwide large contemporary cohort with very long-term follow-up will improve knowledge about acute myocardial infarction management and outcomes in France, and provide a useful platform for nested studies and trials.