Detecting hallucinations in large language models using semantic entropy.

Large language model (LLM) systems, such as ChatGPT1 or Gemini2, can show impressive reasoning and question-answering capabilities but often 'hallucinate' false outputs and unsubstantiated answers3,4. Answering unreliably or without the necessary information prevents adoption in diverse fields, with problems including fabrication of legal precedents5 or untrue facts in news articles6 and even posing a risk to human life in medical domains such as radiology7. Encouraging truthfulness through supervision or reinforcement has been only partially successful8. Researchers need a general method for detecting hallucinations in LLMs that works even with new and unseen questions to which humans might not know the answer. Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations-confabulations-which are arbitrary and incorrect generations. Our method addresses the fact that one idea can be expressed in many ways by computing uncertainty at the level of meaning rather than specific sequences of words. Our method works across datasets and tasks without a priori knowledge of the task, requires no task-specific data and robustly generalizes to new tasks not seen before. By detecting when a prompt is likely to produce a confabulation, our method helps users understand when they must take extra care with LLMs and opens up new possibilities for using LLMs that are otherwise prevented by their unreliability.

Three-Dimensional Mapping of Scapular Body, Neck, and Glenoid Fractures.

OBJECTIVES: The purpose of this study was to report patterns of scapular fractures and define them with a contemporary methodology. METHODS: . DESIGN: Retrospective study, 2015-2021. SETTING: Single, academic, Level 1 trauma center. PATIENT SELECTION CRITERIA: Consecutive patients ≥18 years, presenting with unilateral scapula fracture, with thin-slice (≤0.5-mm) bilateral computed tomography (CT) scans of the entirety of both the injured and uninjured scapulae. OUTCOME MEASURES AND COMPARISONS: Thin-slice (0.5-mm) CT scans of injured and normal scapulae were obtained to create three-dimensional (3D) virtual models. 3D modeling software (Stryker Orthopedics Modeling and Analytics, Stryker Trauma GmbH, Kiel, Germany aka SOMA) was used to create a 3D map of fracture location and frequency. Fracture zones were delineated using anatomic landmarks to characterize fracture patterns. RESULTS: Eighty-seven patients were identified with 75 (86%) extra-articular and 12 (14%) intra-articular fractures. The dominant fracture pattern emanated from the superior lateral border (zone E) to an area inferior to the spinomedial angle (zone B) and was present in 80% of extra-articular fractures. A second-most common fracture line propagated from the primary (most-common) line toward the inferior medial scapular border with a frequency of 36%. Bare zones (with 1 or no fractures present) were identified in 4 unique areas. Furthermore, intra-articular fractures were found to be heterogenous. CONCLUSIONS: The 3D fracture map created in this study confirmed that extra-articular scapular fractures occur in certain patterns with a relatively high frequency. Results provide greater insight into scapular fracture locations and may help to study prognosis of injury and improve treatment strategy including operative approaches and surgical tactics.