Text and Audio Simplification: Human vs. ChatGPT.

Text and audio simplification to increase information comprehension are important in healthcare. With the introduction of ChatGPT, evaluation of its simplification performance is needed. We provide a systematic comparison of human and ChatGPT simplified texts using fourteen metrics indicative of text difficulty. We briefly introduce our online editor where these simplification tools, including ChatGPT, are available. We scored twelve corpora using our metrics: six text, one audio, and five ChatGPT simplified corpora (using five different prompts). We then compare these corpora with texts simplified and verified in a prior user study. Finally, a medical domain expert evaluated the user study texts and five, new ChatGPT simplified versions. We found that simple corpora show higher similarity with the human simplified texts. ChatGPT simplification moves metrics in the right direction. The medical domain expert's evaluation showed a preference for the ChatGPT style, but the text itself was rated lower for content retention.

Effects of Added Emphasis and Pause in Audio Delivery of Health Information.

Health literacy is crucial to supporting good health and is a major national goal. Audio delivery of information is becoming more popular for informing oneself. In this study, we evaluate the effect of audio enhancements in the form of information emphasis and pauses with health texts of varying difficulty and we measure health information comprehension and retention. We produced audio snippets from difficult and easy text and conducted the study on Amazon Mechanical Turk (AMT). Our findings suggest that emphasis matters for both information comprehension and retention. When there is no added pause, emphasizing significant information can lower the perceived difficulty for difficult and easy texts. Comprehension is higher (54%) with correctly placed emphasis for the difficult texts compared to not adding emphasis (50%). Adding a pause lowers perceived difficulty and can improve retention but adversely affects information comprehension.

A Comparison of Google and ChatGPT for Automatic Generation of Health-related Multiple-choice Questions.

Critical to producing accessible content is an understanding of what characteristics affect understanding and comprehension. To answer this question, we are producing a large corpus of health-related texts with associated questions that can be read or listened to by study participants to measure the difficulty of the underlying content, which can later be used to better understand text difficulty and user comprehension. In this paper, we examine methods for automatically generating multiple-choice questions using Google's related questions and ChatGPT. Overall, we find both algorithms generate reasonable questions that are complementary; ChatGPT questions are more similar to the snippet while Google related-search questions have more lexical variation.

Audio delivery of health information: An NLP study of information difficulty and bias in listeners.

Health literacy is the ability to understand, process, and obtain health information and make suitable decisions about health care [3]. Traditionally, text has been the main medium for delivering health information. However, virtual assistants are gaining popularity in this digital era; and people increasingly rely on audio and smart speakers for health information. We aim to identify audio/text features that contribute to the difficulty of the information delivered over audio. We are creating a health-related audio corpus. We selected text snippets and calculated seven text features. Then, we converted the text snippets to audio snippets. In a pilot study with Amazon Mechanical Turk (AMT) workers, we measured the perceived and actual difficulty of the audio using the response of multiple choice and free recall questions. We collected demographic information as well as bias about doctors' gender, task preference, and health information preference. Thirteen workers completed thirty audio snippets and related questions. We found a strong correlation between text features lexical chain, and the dependent variables, and multiple choice response, percentage of matching word, percentage of similar word, cosine similarity, and time taken (in seconds). In addition, doctors were generally perceived to be more competent than warm. How warm workers perceive male doctors correlated significantly with perceived difficulty.

Improving the Quality of Suggestions for Medical Text Simplification Tools.

Text continues to be an important medium for communicating health-related information. We have built a text simplification tool that gives concrete suggestions on how to simplify health and medical texts. An important component of the tool identifies difficult words and suggests simpler synonyms based on pre-existing resources (WordNet and UMLS). These candidate substitutions are not always appropriate in all contexts. In this paper, we introduce a filtering algorithm that utilizes semantic similarity based on word embeddings to determine if the candidate substitution is appropriate in the context of the text. We provide an analysis of our approach on a new dataset of 788 labeled substitution examples. The filtering algorithm is particularly helpful at removing obvious examples and can improve the precision by 3% at a recall level of 95%.

Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty.

Objective: Simplifying healthcare text to improve understanding is difficult but critical to improve health literacy. Unfortunately, few tools exist that have been shown objectively to improve text and understanding. We developed an online editor that integrates simplification algorithms that suggest concrete simplifications, all of which have been shown individually to affect text difficulty. Materials and Methods: The editor was used by a health educator at a local community health center to simplify 4 texts. A controlled experiment was conducted with community center members to measure perceived and actual difficulty of the original and simplified texts. Perceived difficulty was measured using a Likert scale; actual difficulty with multiple-choice questions and with free recall of information evaluated by the educator and 2 sets of automated metrics. Results: The results show that perceived difficulty improved with simplification. Several multiple-choice questions, measuring actual difficulty, were answered more correctly with the simplified text. Free recall of information showed no improvement based on the educator evaluation but was better for simplified texts when measured with automated metrics. Two follow-up analyses showed that self-reported education level and the amount of English spoken at home positively correlated with question accuracy for original texts and the effect disappears with simplified text. Discussion: Simplifying text is difficult and the results are subtle. However, using a variety of different metrics helps quantify the effects of changes. Conclusion: Text simplification can be supported by algorithmic tools. Without requiring tool training or linguistic knowledge, our simplification editor helped simplify healthcare related texts.

Incidence and Impact of Missing Functional Elements on Information Comprehension using Audio and Text.

Audio is increasingly used to communicate health information. Initial evaluations have shown it to be an effective means with many features that can be optimized. This study focuses on missing functional elements: words that relate concepts in a sentence but are often excluded for brevity. They are not easily recognizable without linguistics expertise but can be detected algorithmically. Two studies showed that they are common and affect comprehension. A corpus statistics study with medical (Cochrane sentences, N=44,488) and general text (English and Simple English Wikipedia sentences, N=318,056 each) showed that functional elements were missing in 20-30% of sentences. A user study with Cochrane (N=50) and Wikipedia (N=50) paragraphs in text and audio format showed that more missing functional elements increased perceived difficulty of reading text, with the effect less pronounced with audio, and increased actual difficulty of both written and audio information with less information recalled with more missing elements.

A comparison of text versus audio for information comprehension with future uses for smart speakers.

OBJECTIVE: Audio is increasingly used to access information on the Internet through virtual assistants and smart speakers. Our objective is to evaluate the distribution of health information through audio. MATERIALS AND METHODS: We conducted 2 studies to compare comprehension after reading or listening to information using a new corpus containing short text snippets from Cochrane (N = 50) and Wikipedia (N = 50). In study 1, the snippets were first presented as audio or text followed by a multiple-choice question. Then, the same information was presented as text and the question was repeated in addition to questions about perceived difficulty, severity and the likelihood of encountering the disease. In study 2, the first multiple-choice question was replaced with a free recall question. RESULTS: Study 1 showed that information comprehension is very similar in both presentation modes (53% accuracy for text and 55% for audio). Study 2 showed that information retention is higher with text, but similar comprehension. Both studies show improvement in performance with repeated information presentation. DISCUSSION: Audio presentation of information is effective and the format novel. Performance was slightly lower with audio when asked to repeat information, but comparable to text for answering questions. Additional studies are needed with different types of information and presentation combinations. CONCLUSION: The use of audio to provide health information is a promising field and will become increasingly important with the popularity of smart speakers and virtual assistants, particularly for consumers who do not use computers, for example minority groups, or those with limited sight or motor control.

Algorithmic Generation of Grammar Simplification Rules Using Large Corpora.

There is often a discontinuity between patients' literacy level and educational materials. In response, we are developing an online medical text simplification editor. In this paper, we describe generating grammar simplification rules from a large parallel corpus (N=141,500) containing original sentences and their simplified variants. We algorithmically identified grammatical transformations between sentences (N=26,600) and used distributional characteristics in two corpora to select transformations with the broadest application and the least ambiguity. This resulted in a top set of 146 rules. Two experts evaluated 20 representative rules reflecting 4 characteristics (long/short and weak/strong) each with 5 example sentences. Generally, we found that the rules are helpful for guiding simplification. Using a 5-point Likert scale (5=best), stronger rules scored higher for ease of applying (4.11), overall helpfulness (4.40) and usefulness of examples (4.05). Rule length did not affect the expert scores. The grammar simplification rules are being integrated in our text editor.

Predicting Transition Words Between Sentence for English and Spanish Medical Text.

Transition words add important information and are useful for increasing text comprehension for readers. Our goal is to automatically detect transition words in the medical domain. We introduce a new dataset for identifying transition words categorized into 16 different types with occurrences in adjacent sentence pairs in medical texts from English and Spanish Wikipedia (70K and 27K examples, respectively). We provide classification results using a feedforward neural network with word embedding features. Overall, we detect the need for a transition word with 78% accuracy in English and 84% in Spanish. For individual transition word categories, performance varies widely and is not related to either the number of training examples or the number of transition words in the category. The best accuracy in English was for Examplification words (82%) and in Spanish for Contrast words (96%).

Using Lexical Chains to Identify Text Difficulty: A Corpus Statistics and Classification Study.

Our goal is data-driven discovery of features for text simplification. In this paper, we investigate three types of lexical chains: exact, synonymous, and semantic. A lexical chain links semantically related words in a document. We examine their potential with a document-level corpus statistics study (914 texts) to estimate their overall capacity to differentiate between easy and difficult text and a classification task (11 000 sentences) to determine usefulness of features at sentence-level for simplification. For the corpus statistics study we tested five document-level features for each chain type: total number of chains, average chain length, average chain span, number of crossing chains, and the number of chains longer than half the document length. We found significant differences between easy and difficult text for average chain length and the average number of cross chains. For the sentence classification study, we compared the lexical chain features to standard bag-of-words features on a range of classifiers: logistic regression, naïve Bayes, decision trees, linear and RBF kernel SVM, and random forest. The lexical chain features performed significantly better than the bag-of-words baseline across all classifiers with the best classifier achieving an accuracy of ∼90% (compared to 78% for bag-of-words). Overall, we find several lexical chain features provide specific information useful for identifying difficult sentences of text, beyond what is available from standard lexical features.

Improving Consumer Understanding of Medical Text: Development and Validation of a New SubSimplify Algorithm to Automatically Generate Term Explanations in English and Spanish.

BACKGROUND: While health literacy is important for people to maintain good health and manage diseases, medical educational texts are often written beyond the reading level of the average individual. To mitigate this disconnect, text simplification research provides methods to increase readability and, therefore, comprehension. One method of text simplification is to isolate particularly difficult terms within a document and replace them with easier synonyms (lexical simplification) or an explanation in plain language (semantic simplification). Unfortunately, existing dictionaries are seldom complete, and consequently, resources for many difficult terms are unavailable. This is the case for English and Spanish resources. OBJECTIVE: Our objective was to automatically generate explanations for difficult terms in both English and Spanish when they are not covered by existing resources. The system we present combines existing resources for explanation generation using a novel algorithm (SubSimplify) to create additional explanations. METHODS: SubSimplify uses word-level parsing techniques and specialized medical affix dictionaries to identify the morphological units of a term and then source their definitions. While the underlying resources are different, SubSimplify applies the same principles in both languages. To evaluate our approach, we used term familiarity to identify difficult terms in English and Spanish and then generated explanations for them. For each language, we extracted 400 difficult terms from two different article types (General and Medical topics) balanced for frequency. For English terms, we compared SubSimplify's explanation with the explanations from the Consumer Health Vocabulary, WordNet Synonyms and Summaries, as well as Word Embedding Vector (WEV) synonyms. For Spanish terms, we compared the explanation to WordNet Summaries and WEV Embedding synonyms. We evaluated quality, coverage, and usefulness for the simplification provided for each term. Quality is the average score from two subject experts on a 1-4 Likert scale (two per language) for the synonyms or explanations provided by the source. Coverage is the number of terms for which a source could provide an explanation. Usefulness is the same expert score, however, with a 0 assigned when no explanations or synonyms were available for a term. RESULTS: SubSimplify resulted in quality scores of 1.64 for English (P<.001) and 1.49 for Spanish (P<.001), which were lower than those of existing resources (Consumer Health Vocabulary [CHV]=2.81). However, in coverage, SubSimplify outperforms all existing written resources, increasing the coverage from 53.0% to 80.5% in English and from 20.8% to 90.8% in Spanish (P<.001). This result means that the usefulness score of SubSimplify (1.32; P<.001) is greater than that of most existing resources (eg, CHV=0.169). CONCLUSIONS: Our approach is intended as an additional resource to existing, manually created resources. It greatly increases the number of difficult terms for which an easier alternative can be made available, resulting in greater actual usefulness.

Measuring Text Difficulty Using Parse-Tree Frequency.

Text simplification often relies on dated, unproven readability formulas. As an alternative and motivated by the success of term familiarity, we test a complementary measure: grammar familiarity. Grammar familiarity is measured as the frequency of the 3rd level sentence parse tree and is useful for evaluating individual sentences. We created a database of 140K unique 3rd level parse structures by parsing and binning all 5.4M sentences in English Wikipedia. We then calculated the grammar frequencies across the corpus and created 11 frequency bins. We evaluate the measure with a user study and corpus analysis. For the user study, we selected 20 sentences randomly from each bin, controlling for sentence length and term frequency, and recruited 30 readers per sentence (N=6,600) on Amazon Mechanical Turk. We measured actual difficulty (comprehension) using a Cloze test, perceived difficulty using a 5-point Likert scale, and time taken. Sentences with more frequent grammatical structures, even with very different surface presentations, were easier to understand, perceived as easier and took less time to read. Outcomes from readability formulas correlated with perceived but not with actual difficulty. Our corpus analysis shows how the metric can be used to understand grammar regularity in a broad range of corpora.

NegAIT: A new parser for medical text simplification using morphological, sentential and double negation.

Many different text features influence text readability and content comprehension. Negation is commonly suggested as one such feature, but few general-purpose tools exist to discover negation and studies of the impact of negation on text readability are rare. In this paper, we introduce a new negation parser (NegAIT) for detecting morphological, sentential, and double negation. We evaluated the parser using a human annotated gold standard containing 500 Wikipedia sentences and achieved 95%, 89% and 67% precision with 100%, 80%, and 67% recall, respectively. We also investigate two applications of this new negation parser. First, we performed a corpus statistics study to demonstrate different negation usage in easy and difficult text. Negation usage was compared in six corpora: patient blogs (4K sentences), Cochrane reviews (91K sentences), PubMed abstracts (20K sentences), clinical trial texts (48K sentences), and English and Simple English Wikipedia articles for different medical topics (60K and 6K sentences). The most difficult text contained the least negation. However, when comparing negation types, difficult texts (i.e., Cochrane, PubMed, English Wikipedia and clinical trials) contained significantly (p<0.01) more morphological negations. Second, we conducted a predictive analytics study to show the importance of negation in distinguishing between easy and difficulty text. Five binary classifiers (Naïve Bayes, SVM, decision tree, logistic regression and linear regression) were trained using only negation information. All classifiers achieved better performance than the majority baseline. The Naïve Bayes' classifier achieved the highest accuracy at 77% (9% higher than the majority baseline).

When synonyms are not enough: Optimal parenthetical insertion for text simplification.

As more patients use the Internet to answer health-related queries, simplifying medical information is becoming increasingly important. To simplify medical terms when synonyms are unavailable, we must add multi-word explanations. Following a data-driven approach, we conducted two user studies to determine the best formulation for adding explanatory content as parenthetical expressions. Study 1 focused on text with a single difficult term (N=260). We examined the effects of different types of text, types of content in parentheses, difficulty of the explanatory content, and position of the term in the sentence on actual difficulty, perceived difficulty, and reading time. We found significant support that enclosing the difficult term in parentheses is best for difficult text and enclosing the explanation in parentheses is best for simple text. Study 2 (N=116) focused on lists with multiple difficult terms. The same interaction is present although statistically insignificant, but parenthetical insertion can still significantly simplify text.

The Role of Surface, Semantic and Grammatical Features on Simplification of Spanish Medical Texts: A User Study.

Simplifying medical texts facilitates readability and comprehension. While most simplification work focuses on English, we investigate whether features important for simplifying English text are similarly helpful for simplifying Spanish text. We conducted a user study on 15 Spanish medical texts using Amazon Mechanical Turk and measured perceived and actual difficulty. Using the median of the difficulty scores, we split the texts into easy and difficult groups and extracted 10 surface, 2 semantic and 4 grammatical features. Using t-tests, we identified those features that significantly distinguish easy text from difficult text in Spanish and compare with prior work in English. We found that easy Spanish texts use more repeated words and adverbs, less negations and more familiar words, similar to English. Also like English, difficult Spanish texts use more nouns and adjectives. However in contrast to English, easier Spanish texts contained longer sentences and used grammatical structures that were more varied.

Moving Beyond Readability Metrics for Health-Related Text Simplification.

Limited health literacy is a barrier to understanding health information. Simplifying text can reduce this barrier and possibly other known disparities in health. Unfortunately, few tools exist to simplify text with demonstrated impact on comprehension. By leveraging modern data sources integrated with natural language processing algorithms, we are developing the first semi-automated text simplification tool. We present two main contributions. First, we introduce our evidence-based development strategy for designing effective text simplification software and summarize initial, promising results. Second, we present a new study examining existing readability formulas, which are the most commonly used tools for text simplification in healthcare. We compare syllable count, the proxy for word difficulty used by most readability formulas, with our new metric 'term familiarity' and find that syllable count measures how difficult words 'appear' to be, but not their actual difficulty. In contrast, term familiarity can be used to measure actual difficulty.

Effects on Text Simplification: Evaluation of Splitting Up Noun Phrases.

To help increase health literacy, we are developing a text simplification tool that creates more accessible patient education materials. Tool development is guided by a data-driven feature analysis comparing simple and difficult text. In the present study, we focus on the common advice to split long noun phrases. Our previous corpus analysis showed that easier texts contained shorter noun phrases. Subsequently, we conducted a user study to measure the difficulty of sentences containing noun phrases of different lengths (2-gram, 3-gram, and 4-gram); noun phrases of different conditions (split or not); and, to simulate unknown terms, pseudowords (present or not). We gathered 35 evaluations for 30 sentences in each condition (3 × 2 × 2 conditions) on Amazon's Mechanical Turk (N = 12,600). We conducted a 3-way analysis of variance for perceived and actual difficulty. Splitting noun phrases had a positive effect on perceived difficulty but a negative effect on actual difficulty. The presence of pseudowords increased perceived and actual difficulty. Without pseudowords, longer noun phrases led to increased perceived and actual difficulty. A follow-up study using the phrases (N = 1,350) showed that measuring awkwardness may indicate when to split noun phrases. We conclude that splitting noun phrases benefits perceived difficulty but hurts actual difficulty when the phrasing becomes less natural.

The effect of word familiarity on actual and perceived text difficulty.

There is little evidence that readability formula outcomes relate to text understanding. The potential cause may lie in their strong reliance on word and sentence length. We evaluated word familiarity rather than word length as a stand-in for word difficulty. Word familiarity represents how well known a word is, and is estimated using word frequency in a large text corpus, in this work the Google web corpus. We conducted a study with 239 people, who provided 50 evaluations for each of 275 words. Our study is the first study to focus on actual difficulty, measured with a multiple-choice task, in addition to perceived difficulty, measured with a Likert scale. Actual difficulty was correlated with word familiarity (r=0.219, p<0.001) but not with word length (r=-0.075, p=0.107). Perceived difficulty was correlated with both word familiarity (r=-0.397, p<0.001) and word length (r=0.254, p<0.001).

User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention.

BACKGROUND: Adequate health literacy is important for people to maintain good health and manage diseases and injuries. Educational text, either retrieved from the Internet or provided by a doctor's office, is a popular method to communicate health-related information. Unfortunately, it is difficult to write text that is easy to understand, and existing approaches, mostly the application of readability formulas, have not convincingly been shown to reduce the difficulty of text. OBJECTIVE: To develop an evidence-based writer support tool to improve perceived and actual text difficulty. To this end, we are developing and testing algorithms that automatically identify difficult sections in text and provide appropriate, easier alternatives; algorithms that effectively reduce text difficulty will be included in the support tool. This work describes the user evaluation with an independent writer of an automated simplification algorithm using term familiarity. METHODS: Term familiarity indicates how easy words are for readers and is estimated using term frequencies in the Google Web Corpus. Unfamiliar words are algorithmically identified and tagged for potential replacement. Easier alternatives consisting of synonyms, hypernyms, definitions, and semantic types are extracted from WordNet, the Unified Medical Language System (UMLS), and Wiktionary and ranked for a writer to choose from to simplify the text. We conducted a controlled user study with a representative writer who used our simplification algorithm to simplify texts. We tested the impact with representative consumers. The key independent variable of our study is lexical simplification, and we measured its effect on both perceived and actual text difficulty. Participants were recruited from Amazon's Mechanical Turk website. Perceived difficulty was measured with 1 metric, a 5-point Likert scale. Actual difficulty was measured with 3 metrics: 5 multiple-choice questions alongside each text to measure understanding, 7 multiple-choice questions without the text for learning, and 2 free recall questions for information retention. RESULTS: Ninety-nine participants completed the study. We found strong beneficial effects on both perceived and actual difficulty. After simplification, the text was perceived as simpler (P<.001) with simplified text scoring 2.3 and original text 3.2 on the 5-point Likert scale (score 1: easiest). It also led to better understanding of the text (P<.001) with 11% more correct answers with simplified text (63% correct) compared to the original (52% correct). There was more learning with 18% more correct answers after reading simplified text compared to 9% more correct answers after reading the original text (P=.003). There was no significant effect on free recall. CONCLUSIONS: Term familiarity is a valuable feature in simplifying text. Although the topic of the text influences the effect size, the results were convincing and consistent.