Characterizing the Adoption and Experiences of Users of Artificial Intelligence-Generated Health Information in the United States: Cross-Sectional Questionnaire Study.

BACKGROUND: OpenAI's ChatGPT is a source of advanced online health information (OHI) that may be integrated into individuals' health information-seeking routines. However, concerns have been raised about its factual accuracy and impact on health outcomes. To forecast implications for medical practice and public health, more information is needed on who uses the tool, how often, and for what. OBJECTIVE: This study aims to characterize the reasons for and types of ChatGPT OHI use and describe the users most likely to engage with the platform. METHODS: In this cross-sectional survey, patients received invitations to participate via the ResearchMatch platform, a nonprofit affiliate of the National Institutes of Health. A web-based survey measured demographic characteristics, use of ChatGPT and other sources of OHI, experience characterization, and resultant health behaviors. Descriptive statistics were used to summarize the data. Both 2-tailed t tests and Pearson chi-square tests were used to compare users of ChatGPT OHI to nonusers. RESULTS: Of 2406 respondents, 21.5% (n=517) respondents reported using ChatGPT for OHI. ChatGPT users were younger than nonusers (32.8 vs 39.1 years, P<.001) with lower advanced degree attainment (BA or higher; 49.9% vs 67%, P<.001) and greater use of transient health care (ED and urgent care; P<.001). ChatGPT users were more avid consumers of general non-ChatGPT OHI (percentage of weekly or greater OHI seeking frequency in past 6 months, 28.2% vs 22.8%, P<.001). Around 39.3% (n=206) respondents endorsed using the platform for OHI 2-3 times weekly or more, and most sought the tool to determine if a consultation was required (47.4%, n=245) or to explore alternative treatment (46.2%, n=239). Use characterization was favorable as many believed ChatGPT to be just as or more useful than other OHIs (87.7%, n=429) and their doctor (81%, n=407). About one-third of respondents requested a referral (35.6%, n=184) or changed medications (31%, n=160) based on the information received from ChatGPT. As many users reported skepticism regarding the ChatGPT output (67.9%, n=336), most turned to their physicians (67.5%, n=349). CONCLUSIONS: This study underscores the significant role of AI-generated OHI in shaping health-seeking behaviors and the potential evolution of patient-provider interactions. Given the proclivity of these users to enact health behavior changes based on AI-generated content, there is an opportunity for physicians to guide ChatGPT OHI users on an informed and examined use of the technology.

Knowledge of HPV vaccination and associated HNC and treatment decision-making among minority populations.

OBJECTIVE: Understand vaccination knowledge and barriers to vaccination among minority adults. STUDY DESIGN: Cross-sectional survey. SETTING: Otolaryngology clinics at a safety net hospital and a tertiary academic center and a head and neck cancer screening event. METHODS: Survey was administered to consenting patients. Descriptive statistics and significance testing were used to characterize the data, with non-minority respondents as controls. Multivariate logistic regression was used to understand factors associated with vaccination. RESULTS: HPV vaccination among our 241 respondents (n = 41, 17.67 %) and their qualifying children (n = 52, 33.55 %) was low. Non-vaccinated minorities were significantly more likely to express interest in HPV vaccination (28.66 % vs 8.66 %, p = 0.016). Minority patients were significantly less knowledgeable about HPV causing cervical (88.64 % vs 72.45 %, p = 0.024) and head and neck (68.18 % vs 44.90 %, p = 0.005) cancer and were also less aware of HPV infection (95.45 % vs 81.12 %, p = 0.020) among non-women. Lack of knowledge about the HPV vaccine was the most cited reason why minority patients did not or were uninterested in vaccination for themselves or their children. In a multivariable logistic regression of factors associated with HPV vaccination, only increased age demonstrated a significant association with vaccination likelihood (OR = 0.91, 95 % CI = [0.88-0.95], p < 0.001). CONCLUSION: Reported HPV vaccination rates were low for both white and minority patients but did not significantly vary on univariate or multivariate analysis. However, minority respondents were significantly less knowledgeable about HPV and its manifestations; they most often cited inadequate knowledge as why did not receive or were uninterested in HPV vaccination. As such, HPV vaccination educational interventions may raise vaccination rates among minority populations.

Over-the-Counter Hearing Aids: A Nationwide Survey Study to Understand Perspectives in Primary Care.

OBJECTIVES: The expansion of over-the-counter (OTC) hearing aids has raised inquiries regarding primary care physicians' (PCP) knowledge, perspective, and perceived roles. We aimed to understand PCP perspectives on OTC hearing aids via nationwide online surveys. METHODS: RedCap survey was distributed to PCPs via online forums and public mailing lists. Outcomes included PCPs' attitudes toward, perceived role surrounding, confidence managing, and knowledge of OTC hearing aids. Regression analyses were performed to identify associated factors including demographics and practice characteristics. RESULTS: Cohort included 111 PCPs primarily working in non-rural (83.8%) outpatient academic medical centers (47.5%), with a mean (SD) of 16.9 (11.6) years practicing. Most reported unfamiliarity (61.3%) with OTC hearing aids but viewed them positively (91.9%). They often perceived themselves as poor sources of OTC hearing aid information (63.1%) but desired involvement (90.1%) and believed associated knowledge is important (98.2%). Rural practice environment was associated with less familiarity toward OTC hearing aids (ß = -0.72, [95% CI -1.40 to -0.04]). Respondents answered 5.0 (2.4) of 10 OTC hearing aid knowledge questions correctly. Using 5-point Likert scale, participants reported most confidence recognizing signs/symptoms of hearing loss 3.71 (0.84), but less confidence educating 1.68 (0.96) about and determining candidacy 1.72 (1.05) for OTC hearing aids. Participants reported continuing medical education courses and published guidelines would effectively improve their OTC hearing aid knowledge. CONCLUSION: PCPs displayed positive attitudes toward OTC hearing aids and valued involvement. Addressing unfamiliarity/knowledge gaps surrounding OTC hearing aids through courses and published guidelines may help clarify misconceptions and promote hearing health care. LEVEL OF EVIDENCE: NA Laryngoscope, 2024.

External Auditory Canal Exostosis and the California Surfer Population: A Cross-Sectional Study.

OBJECTIVES: California has the most surfers in the United States and a high prevalence of external auditory canal exostoses (EACEs) among them. We aimed to characterize their EACE knowledge, use of earplugs, and barriers toward earplug use. METHODS: A RedCap survey was distributed to online surfing forums and surfers at California beaches. Descriptive statistics and regression analyses were used to characterize responses. RESULTS: Our cohort included 334 primarily male (n = 269, 81.52%), college-educated (n = 237, 71.17%) surfers who were on average 30.79 years old (SD = 11.07). Two hundred and ninety-seven (90.00%) heard of EACE and 317 (96.06%) believed earplugs prevent EACE. However, 214 (64.85%) had never used earplugs. Multivariable logistic regression found increased age (OR = 1.04, 95% CI = [1.00-1.08], p = 0.03), higher EACE knowledge quiz scores (OR = 1.47, 95% CI = [1.19-1.80], p < 0.001), and primarily surfing in Southern California (OR = 2.19, 95% CI = [1.15-4.16], p = 0.02) increased the likelihood of earplug use. Common reasons against earplug use included reduced hearing, discomfort, and social hindrance. Two hundred and eighty-seven (86.45%) would wear earplugs following more EACE knowledge. They preferred learning from surf community members, doctors, and surf events. CONCLUSION: Low earplug use despite awareness of EACE preventability suggests a need for EACE education among California surfers and more accessible, user-friendly earplugs. Younger, less-skilled surfers who were more commonly unaware of EACE may represent a key intervention group. Education could be promoted through partnerships between health professionals and renowned surf organizations, as most participants indicated a willingness to use earplugs post-education. LEVEL OF EVIDENCE: N/A Laryngoscope, 2024.

TrachGPT: Appraisal of tracheostomy care recommendations from an artificial intelligent Chatbot.

Objective: Safe home tracheostomy care requires engagement and troubleshooting by patients, who may turn to online, AI-generated information sources. This study assessed the quality of ChatGPT responses to such queries. Methods: In this cross-sectional study, ChatGPT was prompted with 10 hypothetical tracheostomy care questions in three domains (complication management, self-care advice, and lifestyle adjustment). Responses were graded by four otolaryngologists for appropriateness, accuracy, and overall score. The readability of responses was evaluated using the Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKRGL). Descriptive statistics and ANOVA testing were performed with statistical significance set to p < .05. Results: On a scale of 1-5, with 5 representing the greatest appropriateness or overall score and a 4-point scale with 4 representing the highest accuracy, the responses exhibited moderately high appropriateness (mean = 4.10, SD = 0.90), high accuracy (mean = 3.55, SD = 0.50), and moderately high overall scores (mean = 4.02, SD = 0.86). Scoring between response categories (self-care recommendations, complication recommendations, lifestyle adjustments, and special device considerations) revealed no significant scoring differences. Suboptimal responses lacked nuance and contained incorrect information and recommendations. Readability indicated college and advanced levels for FRE (Mean = 39.5, SD = 7.17) and FKRGL (Mean = 13.1, SD = 1.47), higher than the sixth-grade level recommended for patient-targeted resources by the NIH. Conclusion: While ChatGPT-generated tracheostomy care responses may exhibit acceptable appropriateness, incomplete or misleading information may have dire clinical consequences. Further, inappropriately high reading levels may limit patient comprehension and accessibility. At this point in its technological infancy, AI-generated information should not be solely relied upon as a direct patient care resource.

Factors Associated With Head and Neck Polytrauma Presentation and Admissions at Emergency Departments of Varying Sizes.

Timely diagnosis of acute head and neck polytrauma presenting to emergency departments (EDs) optimizes outcomes. Since ED capacity influences triage and admission, the authors utilized the National Electronic Injury Surveillance System database to understand how ED size and trauma characteristics affect head and neck polytrauma presentation and admissions. Demographics and injury characteristics from the National Electronic Injury Surveillance System database from 2018 to 2021 were analyzed to delineate factors contributing to polytrauma presence and admission through multivariable logistic regressions. The authors' 207,951-patient cohort was primarily females (48.6%), non-Hispanic (62.4%), and white (51.4%) people who averaged 57.2 years old. Nonspecific head injuries were predominant (59.7%), followed by facial trauma (22.6%) with rare substance involvement (alcohol, 6.3%; drugs, 4.1%) presenting to high-volume EDs (48.5%). Of the patients, 20% were admitted, whereas 31.1% sustained polytrauma. Substance use [alcohol, odds ratio (OR) = 4.44; drugs, OR = 2.90] increased polytrauma likelihood; neck (OR = 1.35), face (OR = 1.14), and eye (OR = 1.26) associated with polytrauma more than head injuries. Burns (OR = 1.38) increased polytrauma likelihood more than internal organ injuries. Black patients sustained higher polytrauma when presented to non-small EDs (OR = 1.41-1.90) than white patients showed to small EDs. Admissions were higher for males (OR = 1.51). Relative to small EDs, large EDs demonstrated a higher increase in admissions (OR = 2.42). Neck traumas were more likely admitted than head traumas (OR = 1.71). Fractures (OR = 2.21) and burns (OR = 2.71) demonstrated an increased admission likelihood than internal organ injuries. Polytrauma presence and admissions likelihood are site, injury, and substance dependent. Understanding the impact of factors influencing polytrauma presence or admission will enhance triage to optimize outcomes.

Music to Define a Generation: Factors Associated With Hearing Protection Use at Music Venues.

OBJECTIVES: Excessive noise levels in music venues can negatively impact long-term hearing health. We investigated the prevalence, characteristics, and factors associated with hearing protection use among music event attendees. STUDY DESIGN: Cross-sectional cohort. METHODS: Adult participants (2,352) from online music forums completed a questionnaire assessing attendance at music events, hearing protection (HP) usage, type, and frequency, along with history of hearing-related symptoms and medical history. Primary outcome was defined as HP use at music venues. Data were characterized using descriptive statistics, paired t tests, and Pearson's chi-squared tests. Multivariable regression analysis elicited factors associated with any and regular HP use. RESULTS: Our 2,352 respondents were primarily male (61.3%) and on average 28.7 years old (SD = 7.0). Of the respondents, 60.6% reported using HP, and 33.2% reported regular use; high-fidelity earplugs were the most used HP type (57.5%). HP use was associated with significantly fewer reports of negative hearing-related symptoms. In a multivariable regression model, factors associated with less likelihood of HP use included being female (vs male, OR = 0.74, CI = [0.55-0.98]), less concern about effects on hearing (OR = 0.53, CI = [0.45-0.61]), and pop music event attendance (vs EDM, OR = 0.40, CI = [0.22-0.72]). Increased attendance frequency in the past year was associated with higher likelihood of regular HP use (OR = 1.07, CI = [1.04-1.09]). Decreased attendance duration was associated with decreased likelihood (OR = 0.09, CI = [0.03-0.33]). CONCLUSION: HP use prevalence was high, while frequent use remained low. HP use was significantly associated with reduced hearing-related symptoms. Demographic factors, otologic symp-tomology, and attendance-related characteristics were associated with HP use. These findings may guide interventions to increase adoption at music venues. LEVEL OF EVIDENCE: 4.

Evaluation of YouTube As A Source For Graves' Disease Information: Is High-Quality Guideline-Based Information Available?

Objective: To understand the quality of informational Graves' disease (GD) videos on YouTube for treatment decision-making quality and inclusion of American Thyroid Association (ATA) treatment guidelines. Study Design: Cross-sectional cohort. Setting: Informational YouTube videos with subject matter "Graves' Disease treatment." Method: The top 50 videos based on our query were assessed using the DISCERN instrument. This validated algorithm discretely rates treatment-related information from excellent (≥4.5) to very poor (<1.9). Videos were also screened for ATA guideline inclusion. Descriptive statistics were used for cohort characterization. Univariate and multivariate linear regressions characterized factors associated with DISCERN scores. Significance was set at P < .05. Results: The videos featured 57,513.43 views (SD = 162,579.25), 1054.70 likes (SD = 2329.77), and 168.80 comments (SD = 292.97). Most were patient education (52%) or patient experience (24%). A minority (40%) were made by thyroid specialists (endocrinologists, endocrine surgeons, or otolaryngologists). Under half did not mention all 3 treatment modalities (44%), and 54% did not mention any ATA recommendations. Overall, videos displayed poor reliability (mean = 2.26, SD = 0.67), treatment information quality (mean = 2.29, SD = 0.75), and overall video quality (mean = 2.47, SD = 1.07). Physician videos were associated with lower likes, views, and comments (P < .001) but higher DISCERN reliability (P = .015) and overall score (P = .019). Longer videos (P = .015), patient accounts (P = .013), and patient experience (P = .002) were associated with lower scores. Conclusion: The most available GD treatment content on YouTube varies significantly in the quality of medical information. This may contribute to suboptimal disease understanding, especially for patients highly engaged with online health information sources.

Associations Between Risk Perception and HPV Knowledge and Vaccine Uptake: Highlighting the LGBTQ+ Population.

INTRODUCTION: Gay and bisexual males and other LGBTQ+ communities are more frequently exposed to factors associated with an increased risk of human papillomavirus (HPV) acquisition. Vaccination is critical to protect against HPV+ head and neck cancer (HNC). We characterized the association of perceived level of risk of contraction with HPV knowledge, and vaccine decision-making. STUDY DESIGN: Cross-sectional cohort. SETTING: LGBTQ and general survey Reddit forums (control). METHODS: A survey was shared amongst the online forums. Descriptive statistics characterized the data. Multivariable logistic regression was used to understand factors associated with vaccination, self-perceived high risk, and knowledge of HPV + HNC. RESULTS: Of 718 respondents, most were female (41.09%), Caucasian (59.89%), college-educated (33.01%), and insured (77.15%) with a mean age of 30.75 years. Half were vaccinated (49.16%), with most unvaccinated endorsing interest (60.58%). Few dependents were vaccinated (25.91%), with interest in vaccination among parents of unvaccinated children (38.58%). Knowledge of HIV's association with HPV (62.95%), HPV causing HNC (55.57%), and the vaccine's efficacy against HNC (55.57%) was also moderate. Identifying female (P = .042), a self-perceived high-risk (P < .001), and having vaccinated children (P < .001) increased vaccination likelihood; transgender (P = .021), or lesbian or gay sexual identity (P < .001) decreased likelihood. Personal HNC diagnosis (P < .001), self-vaccination (P < .001), having vaccinated children (P < .001), having anal sex (P = .001) or no knowledge of past HPV status (P < .001) increased likelihood of high self-perceived risk. CONCLUSION: Efforts to improve public education regarding the association between HPV and HNC and vaccination efficacy are required to better inform vaccine decision-making among individuals at risk for HPV infection.

Assessing Laryngectomy Patient Education on YouTube: Investigating Quality and Reliability.

Objective: This study aimed to characterize the quality of laryngectomy-related patient education on YouTube and understand factors impacting video content quality. Study Design: Cross-sectional cohort analysis. Setting: Laryngectomy-related videos on YouTube. Methods: YouTube was anonymously queried for various laryngectomy procedure search terms. Video quality was evaluated using the validated DISCERN instrument which assesses treatment-related information quality. Descriptive statistics were used to characterize our cohort. Univariate and multivariable linear regression were used to assess factors associated with increased DISCERN score. Significance was set at P < .05. Results: Our 78-video cohort exhibited moderate levels of engagement, averaging 13,028.40 views (SD = 24,246.93), 69.79 likes (SD = 163.75), and 5.27 comments (SD = 18.81). Videos were most frequently uploaded to accounts belonging to physicians (43.59%) or health care groups (41.03%) and showcased operations (52.56%) or physician-led education (20.51%). Otolaryngologists were featured in most videos (85.90%), and most videos originated outside the United States (67.95%). Laryngectomy videos demonstrated poor reliability (mean = 2.35, SD = 0.77), quality of treatment information (mean = 1.92, SD = 0.86), and overall video quality (mean = 1.97, SD = 1.12). In multivariable linear regression, operative videos were associated with lower video quality relative to nonoperative videos (ß = -1.63, 95% confidence interval [CI] = [-2.03 to -1.24], P < .001); the opposite was true for videos from accounts with higher subscriber counts (ß = 0.02, 95% CI = [0.01-0.03], P = .005). Conclusion: The quality and quantity of YouTube's laryngectomy educational content is limited. There is an acute need to increase the quantity and quality of online laryngectomy-related content to better support patients and caregivers as they cope with their diagnosis, prepare for, and recover from surgery.

New Paradigm of Music Listening: Hearing Protection Perceptions and Treatment Decision-Making Among Music Venue Attendees.

OBJECTIVE: Investigate the prevalence of hearing protection (HP) use and behavioral motivations and barriers among adults attending music venues. STUDY DESIGN: Cross-sectional online survey study. SETTING: Noise exposure levels at popular social music venues often exceed national guidelines. METHODS: Surveys were distributed on online music communities. Respondents (n = 2352) were asked about demographics, HP use at music venues, knowledge about noise exposure impact, and perceptions of HP use. Data were characterized through descriptive statistics. Multivariable regression analysis explored differences in knowledge and perception between HP users and nonusers. RESULTS: In this cohort (mean age 29 ± 7 years, 61% male), HP users were significantly more aware of the impact of music venues on hearing (P < .01), believed their hearing ability had decreased after attending music venues (P < 0.01), and believed HP could protect from hearing loss (P < .01) than non-HP users. HP nonusers most frequently cited never considering HP (14.45%) and apathy about it affecting music quality (12.71%). Common sources of HP information were recommended by a friend/peer. Multivariable regression analysis accounting for demographics, medical history, and attendance characteristics found belief that HP use at music venues could protect from hearing loss (ß = 0.64, 95% confidence interval [CI] = [0.49-0.78]) and HP use (ß = 1.73, 95% CI = [1.47-1.98]) were significantly associated with increased subjective enjoyment while wearing HP. CONCLUSION: HP users and nonusers have significantly different perceptions of HP use and its impact. Our findings have implications for understanding motivations and barriers related to HP use and developing strategies to promote HP use at music venues.

Evaluation of Oropharyngeal Cancer Information from Revolutionary Artificial Intelligence Chatbot.

OBJECTIVE: With burgeoning popularity of artificial intelligence-based chatbots, oropharyngeal cancer patients now have access to a novel source of medical information. Because chatbot information is not reviewed by experts, we sought to evaluate an artificial intelligence-based chatbot's oropharyngeal cancer-related information for accuracy. METHODS: Fifteen oropharyngeal cancer-related questions were developed and input into ChatGPT version 3.5. Four physician-graders independently assessed accuracy, comprehensiveness, and similarity to a physician response using 5-point Likert scales. Responses graded lower than three were then critiqued by physician-graders. Critiques were analyzed using inductive thematic analysis. Readability of responses was assessed using Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKRGL) scales. RESULTS: Average accuracy, comprehensiveness, and similarity to a physician response scores were 3.88 (SD = 0.99), 3.80 (SD = 1.14), and 3.67 (SD = 1.08), respectively. Posttreatment-related questions were most accurate, comprehensive, and similar to a physician response, followed by treatment-related, then diagnosis-related questions. Posttreatment-related questions scored significantly higher than diagnosis-related questions in all three domains (p < 0.01). Two themes of the physician critiques were identified: suboptimal education value and potential to misinform patients. The mean FRE and FKRGL scores both indicated greater than an 11th grade readability level-higher than the 6th grade level recommended for patients. CONCLUSION: ChatGPT responses may not educate patients to an appropriate degree, could outright misinform them, and read at a more difficult grade level than is recommended for patient material. As oropharyngeal cancer patients represent a vulnerable population facing complex, life-altering diagnoses, and treatments, they should be cautious when consuming chatbot-generated medical information. LEVEL OF EVIDENCE: NA Laryngoscope, 134:2252-2257, 2024.

Evaluation of the quality of thyroidectomy-related posts on a video-based social media platform.

Objective: To evaluate the quality of thyroidectomy-related posts on TikTok, the fastest-growing social media platform worldwide. Methods: Videos posted from April 2020 to September 2022 were queried on TikTok using the search terms "thyroidsurgery," "thyroidectomy," and "thyroidremoval." Two reviewers recorded thematic, demographic, and performance data of these posts. The DISCERN instrument was used to evaluate the quality and reliability of the information contained in the videos. Descriptive statistics were used to characterize post-submitter demographics and video content. Simple and multiple linear regression analyses were used to evaluate the association between DISCERN scores and video characteristics. Univariate analysis of variance was performed to compare DISCERN scores between author types. Results: In this study, 228 TikTok videos were included which totaled over 23 million views. On average, each video accumulated more than 6000 "likes," 300 comments, and 70 shares. The average total DISCERN score was 27.46, which is deemed to be of poor overall quality. Upon multiple linear regression, video duration (ß = 4.66, p < .001) and educational subject type (ß = 3.97, p < .001) significantly positively predicted aggregate DISCERN scores, while journey subject type (ß = -3.19, p = .006), and reassurance subject type (ß = -2.52, p = .035) significantly negatively predicted aggregate DISCERN scores. Aggregate DISCERN scores varied significantly (p < .05) between author types. Conclusion: Social media posts on TikTok about thyroidectomy are mostly of poor quality and reliability but vary by authorship, subject type, and video characteristics. Given its widespread popularity, TikTok videos may have an increasing role in shaping patient perception of thyroidectomy and may represent an opportunity to provide education. Lay summary: TikTok posts about thyroidectomy are mostly of poor quality but vary by authorship, subject, and video characteristics. Given its popularity, TikTok videos may have a role in shaping the patient perception of thyroidectomy and may represent an opportunity to provide education. Level of evidence: Level 4.

Bridging the Gap Between Urological Research and Patient Understanding: The Role of Large Language Models in Automated Generation of Layperson's Summaries.

INTRODUCTION: This study assessed ChatGPT's ability to generate readable, accurate, and clear layperson summaries of urological studies, and compared the performance of ChatGPT-generated summaries with original abstracts and author-written patient summaries to determine its effectiveness as a potential solution for creating accessible medical literature for the public. METHODS: Articles from the top 5 ranked urology journals were selected. A ChatGPT prompt was developed following guidelines to maximize readability, accuracy, and clarity, minimizing variability. Readability scores and grade-level indicators were calculated for the ChatGPT summaries, original abstracts, and patient summaries. Two MD physicians independently rated the accuracy and clarity of the ChatGPT-generated layperson summaries. Statistical analyses were conducted to compare readability scores. Cohen's κ coefficient was used to assess interrater reliability for correctness and clarity evaluations. RESULTS: A total of 256 journal articles were included. The ChatGPT-generated summaries were created with an average time of 17.5 (SD 15.0) seconds. The readability scores of the ChatGPT-generated summaries were significantly better than the original abstracts, with Global Readability Score 54.8 (12.3) vs 29.8 (18.5), Flesch Kincade Reading Ease 54.8 (12.3) vs 29.8 (18.5), Flesch Kincaid Grade Level 10.4 (2.2) vs 13.5 (4.0), Gunning Fog Score 12.9 (2.6) vs 16.6 (4.1), Smog Index 9.1 (2.0) vs 12.0 (3.0), Coleman Liau Index 12.9 (2.1) vs 14.9 (3.7), and Automated Readability Index 11.1 (2.5) vs 12.0 (5.7; P < .0001 for all except Automated Readability Index, which was P = .037). The correctness rate of ChatGPT outputs was >85% across all categories assessed, with interrater agreement (Cohen's κ) between 2 independent physician reviewers ranging from 0.76-0.95. CONCLUSIONS: ChatGPT can create accurate summaries of scientific abstracts for patients, with well-crafted prompts enhancing user-friendliness. Although the summaries are satisfactory, expert verification is necessary for improved accuracy.

Evaluating the Effectiveness of Artificial Intelligence-powered Large Language Models Application in Disseminating Appropriate and Readable Health Information in Urology.

PURPOSE: The Internet is a ubiquitous source of medical information, and natural language processors are gaining popularity as alternatives to traditional search engines. However, suitability of their generated content for patients is not well understood. We aimed to evaluate the appropriateness and readability of natural language processor-generated responses to urology-related medical inquiries. MATERIALS AND METHODS: Eighteen patient questions were developed based on Google Trends and were used as inputs in ChatGPT. Three categories were assessed: oncologic, benign, and emergency. Questions in each category were either treatment or sign/symptom-related questions. Three native English-speaking Board-Certified urologists independently assessed appropriateness of ChatGPT outputs for patient counseling using accuracy, comprehensiveness, and clarity as proxies for appropriateness. Readability was assessed using the Flesch Reading Ease and Flesh-Kincaid Reading Grade Level formulas. Additional measures were created based on validated tools and assessed by 3 independent reviewers. RESULTS: Fourteen of 18 (77.8%) responses were deemed appropriate, with clarity having the most 4 and 5 scores (P = .01). There was no significant difference in appropriateness of the responses between treatments and symptoms or between different categories of conditions. The most common reason from urologists for low scores was responses lacking information-sometimes vital information. The mean (SD) Flesch Reading Ease score was 35.5 (SD=10.2) and the mean Flesh-Kincaid Reading Grade Level score was 13.5 (1.74). Additional quality assessment scores showed no significant differences between different categories of conditions. CONCLUSIONS: Despite impressive capabilities, natural language processors have limitations as sources of medical information. Refinement is crucial before adoption for this purpose.