A critical assessment of using ChatGPT for extracting structured data from clinical notes.

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT's capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral "prompt engineering" process, leveraging OpenAI's API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

A Novel Approach to Assessment of US Pediatric Trauma System Development.

Importance: Mature trauma systems are critical in building and maintaining national, state, and local resilience against all-hazard disasters. Currently, pediatric state trauma system plans are not standardized and thus are without concrete measures of potential effectiveness. Objective: To develop objective measures of pediatric trauma system capability at the state level, hypothesizing significant variation in capabilities between states, and to provide a contemporary report on the status of national pediatric trauma system planning and development. Design, Setting, and Participants: A national survey was deployed in 2018 to perform a gap analysis of state pediatric trauma system capabilities. Four officials from each state were asked to complete the survey regarding extensive pediatric-related or specific trauma system parameters. Using these parameters, a panel of 14 individuals representing national stakeholder sectors in pediatric trauma care convened to identify the essential components of the ideal pediatric trauma system using Delphi methodology. Data analysis was conducted from March 16, 2019, to February 23, 2020. Main Outcomes and Measures: Based on results from the national survey and consensus panel parameters, each state was given a composite score. The score was validated using US Centers for Disease Control and Prevention Wide-Ranging Online Data for Epidemiologic Research (CDC WONDER) fatal injury database. Results: The national survey had less than 10% missing data. The consensus panel reached agreement on 6 major domains of pediatric trauma systems (disaster, legislation/funding, access to care, injury prevention/recognition, quality improvement, pediatric readiness) and was used to develop the Pediatric Trauma System Assessment Score (PTSAS) based on 100 points. There was substantial variation across states, with state scores ranging from 48.5 to 100. Based on US CDC WONDER data, for every 1-point increase in PTSAS, there was a 0.12 per 100 000 decrease in mortality (95% CI, -0.22 to -0.02; P = .03). Conclusions and Relevance: Results of this cross-sectional study suggest that a more robust pediatric trauma system has a significant association with pediatric injury mortality. This study assessed the national landscape of capability and preparedness to provide pediatric trauma care at the state level. These parameters can tailor the maturation of children's interests within a state trauma system and assist with future state, regional, and national planning.