Evaluating Intensity, Complexity, and Potential for Causal Inference in Social Needs Interventions: A Review of a Scoping Review.

Importance: Interventions that address needs such as low income, housing instability, and safety are increasingly appearing in the health care sector as part of multifaceted efforts to improve health and health equity, but evidence relevant to scaling these social needs interventions is limited. Objective: To summarize the intensity and complexity of social needs interventions included in randomized clinical trials (RCTs) and assess whether these RCTs were designed to measure the causal effects of intervention components on behavioral, health, or health care utilization outcomes. Evidence Review: This review of a scoping review was based on a Patient-Centered Outcomes Research Institute-funded evidence map of English-language US-based RCTs of social needs interventions published between January 1, 1995, and April 6, 2023. Studies were assessed for features related to intensity (defined using modal values as providing as-needed interaction, 8 participant contacts or more, contacts occurring every 2 weeks or more often, encounters of 30 minutes or longer, contacts over 6 months or longer, or home visits), complexity (defined as addressing multiple social needs, having dedicated staff, involving multiple intervention components or practitioners, aiming to change multiple participant behaviors [knowledge, action, or practice], requiring or providing resources or active assistance with resources, and permitting tailoring), and the ability to assess causal inferences of components (assessing interventions, comparators, and context). Findings: This review of a scoping review of social needs interventions identified 77 RCTs in 93 publications with a total of 135 690 participants. Most articles (68 RCTs [88%]) reported 1 or more features of high intensity. All studies reported 1 or more features indicative of high complexity. Because most studies compared usual care with multicomponent interventions that were moderately or highly dependent on context and individual factors, their designs permitted causal inferences about overall effectiveness but not about individual components. Conclusions and Relevance: Social needs interventions are complex, intense, and include multiple components. Our findings suggest that RCTs of these interventions address overall intervention effectiveness but are rarely designed to distinguish the causal effects of specific components despite being resource intensive. Future studies with hybrid effectiveness-implementation and sequential designs, and more standardized reporting of intervention intensity and complexity could help stakeholders assess the return on investment of these interventions.

Performance of two large language models for data extraction in evidence synthesis.

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT-4) for extracting pre-specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT-4. GPT-4 required use of a third-party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT-4 with the plug-in was lower (68.8%); however, most of the errors were due to the plug-in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT-4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open-source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.

Data extraction for evidence synthesis using a large language model: A proof-of-concept study.

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

Literature Review of Pathogen Agnostic Molecular Testing of Clinical Specimens From Difficult-to-Diagnose Patients: Implications for Public Health.

To better identify emerging or reemerging pathogens in patients with difficult-to-diagnose infections, it is important to improve access to advanced molecular testing methods. This is particularly relevant for cases where conventional microbiologic testing has been unable to detect the pathogen and the patient's specimens test negative. To assess the availability and utility of such testing for human clinical specimens, a literature review of published biomedical literature was conducted. From a corpus of more than 4,000 articles, a set of 34 reports was reviewed in detail for data on where the testing was being performed, types of clinical specimens tested, pathogen agnostic techniques and methods used, and results in terms of potential pathogens identified. This review assessed the frequency of advanced molecular testing, such as metagenomic next generation sequencing that has been applied to clinical specimens for supporting clinicians in caring for difficult-to-diagnose patients. Specimen types tested were from cerebrospinal fluid, respiratory secretions, and other body tissues and fluids. Publications included case reports and series, and there were several that involved clinical trials, surveillance studies, research programs, or outbreak situations. Testing identified both known human pathogens (sometimes in new sites) and previously unknown human pathogens. During this review, there were no apparent coordinated efforts identified to develop regional or national reports on emerging or reemerging pathogens. Therefore, development of a coordinated sentinel surveillance system that applies advanced molecular methods to clinical specimens which are negative by conventional microbiological diagnostic testing would provide a foundation for systematic characterization of emerging and underdiagnosed pathogens and contribute to national biodefense strategy goals.