The Tool for Automatic Measurement of Morphological Information (TAMMI).

This study documents and assesses the Tool for Automatic Measurement of Morphological Information (TAMMI), which calculates measures related to basic morpheme counts, morphological variety, morphological complexity, morpheme type-token counts, and variables found in the MorphoLex database (Sánchez-Gutiérrez et al., 2018) including morpheme frequency/length, morpheme family size counts and frequency, and morpheme hapax counts. These measures are assessed in two studies that include a word frequency measure as a control variable. The first study examined links between morphological variables and judgements of reading ease in a corpus of ~ 5000 reading excerpts, finding that variables related to derivational variety, word frequency, affix frequency, and morpheme counts explained 40% of the variance in the reading scores. The second examined links between morphological variables and human assessments of vocabulary proficiency in a corpus of ~ 7000 essays written by English-language learners (ELLs), finding that the number of morphemes, morpheme variety, and the number of roots explained 21% of the variance in the human assessments.

Lexical and phraseological differences between second language written and spoken opinion responses.

This study examines differences in lexical and phraseological complexity features between second language (L2) written and spoken opinion responses via classification analysis. The study further examines the characteristics of L2 written and spoken responses that were misclassified in terms of lexical and phraseological differences, L2 learners' vocabulary knowledge, and raters' judgments of L2 use. The goal is to more thoroughly explore potential differences in lexical and phraseological production based on modality. The results indicated that L2 written responses tended to elicit greater lexical and phraseological complexity. The results also indicated that crossing the boundaries from L2 spoken to written (i.e., the use of less lexical and phraseological complexity) was related to lower levels of L2 vocabulary knowledge and tended to be penalized by raters in terms of L2 use. In contrast, crossing the boundaries from L2 written output to spoken (i.e., the use of greater lexical and phraseological complexity) was acceptable in terms of L2 use. Overall, this study highlights lexical and phraseological differences and the importance of the use of greater lexical and phraseological complexity in a modality-insensitive manner in L2 opinion-giving responses.

Do Struggling Adult Readers Monitor Their Reading? Understanding the Role of Online and Offline Comprehension Monitoring Processes During Reading.

Comprehension monitoring is a meta-cognitive skill that is defined as the ability to self-evaluate one's comprehension of text. Although it is known that struggling adult readers are poor at monitoring their comprehension, additional research is needed to understand the mechanisms underlying comprehension monitoring and their role in reading comprehension in this population. This study used a comprehension monitoring task with struggling adult readers, which included online eye movements (reread and regression path durations) and an offline verbal protocol (oral explanations of key information). We examined whether eye movements predicted accuracy on the passages' reading comprehension questions, a norm-referenced reading assessment, and an offline verbal protocol after controlling for age and traditional component skills (i.e., decoding, oral language, working memory). Regression path duration uniquely predicted accuracy on the questions; however, decoding and oral vocabulary were the most salient predictors of the norm-referenced reading comprehension measure. Regression path duration also predicted the offline verbal protocol, such that those who exhibited longer regression path duration were also better at explaining key information. These results contribute to the literature regarding struggling adults' reading component skills, eye movement behaviors involved in processing connected text, and future considerations in assessing comprehension monitoring.

The persuasive essays for rating, selecting, and understanding argumentative and discourse elements (PERSUADE) corpus 1.0.

This paper introduces the Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements (PERSUADE) corpus.The PERSUADE corpus is large-scale corpus of writing with annotated discourse elements. The goal of the corpus is to spur the development of new, open-source scoring algorithms that identify discourse elements in argumentative writing to open new avenues for the development of automatic writing evaluation systems that focus more specifically on the semantic and organizational elements of student writing.

Precision communication: Physicians' linguistic adaptation to patients' health literacy.

Little quantitative research has explored which clinician skills and behaviors facilitate communication. Mutual understanding is especially challenging when patients have limited health literacy (HL). Two strategies hypothesized to improve communication include matching the complexity of language to patients' HL ("universal tailoring"); or always using simple language ("universal precautions"). Through computational linguistic analysis of 237,126 email exchanges between dyads of 1094 physicians and 4331 English-speaking patients, we assessed matching (concordance/discordance) between physicians' linguistic complexity and patients' HL, and classified physicians' communication strategies. Among low HL patients, discordance was associated with poor understanding (P = 0.046). Physicians' "universal tailoring" strategy was associated with better understanding for all patients (P = 0.01), while "universal precautions" was not. There was an interaction between concordance and communication strategy (P = 0.021): The combination of dyadic concordance and "universal tailoring" eliminated HL-related disparities. Physicians' ability to adapt communication to match their patients' HL promotes shared understanding and equity. The 'Precision Medicine' construct should be expanded to include the domain of 'Precision Communication.'

Developing and Testing Automatic Models of Patient Communicative Health Literacy Using Linguistic Features: Findings from the ECLIPPSE study.

Patients with diabetes and limited health literacy (HL) may have suboptimal communication exchange with their health care providers and be at elevated risk of adverse health outcomes. These difficulties are generally attributed to patients' reduced ability to both communicate and understand health-related ideas as well as physicians' lack of skill in identifying those with limited HL. Understanding and identifying patients with barriers posed by lower HL to improve healthcare delivery and outcomes is an important research avenue. However, doing so using traditional methods has proven difficult and infeasible to scale. This study using corpus analyses, expert human ratings of HL, and natural language processing (NLP) approaches to estimate HL at the individual patient level. The goal of the study is to better understand HL from a linguistic perspective and to open new research areas to enhance population management and individualized care. Specifically, this study examines HL as a function of patients' demonstrated ability to communicate health-related information to their providers via secure messages. The study develops an NLP-based HL model and validates the model by predicting patient-related events such as medical outcomes and hospitalizations. Results indicate that the developed model predicts human ratings of HL with ~80% accuracy. Validation indicates that lower HL patients are more likely to be nonwhite and have lower educational attainment. In addition, patients with lower HL suffered more negative health outcomes and had higher healthcare service utilization.

Employing computational linguistics techniques to identify limited patient health literacy: Findings from the ECLIPPSE study.

OBJECTIVE: To develop novel, scalable, and valid literacy profiles for identifying limited health literacy patients by harnessing natural language processing. DATA SOURCE: With respect to the linguistic content, we analyzed 283 216 secure messages sent by 6941 diabetes patients to physicians within an integrated system's electronic portal. Sociodemographic, clinical, and utilization data were obtained via questionnaire and electronic health records. STUDY DESIGN: Retrospective study used natural language processing and machine learning to generate five unique "Literacy Profiles" by employing various sets of linguistic indices: Flesch-Kincaid (LP_FK); basic indices of writing complexity, including lexical diversity (LP_LD) and writing quality (LP_WQ); and advanced indices related to syntactic complexity, lexical sophistication, and diversity, modeled from self-reported (LP_SR), and expert-rated (LP_Exp) health literacy. We first determined the performance of each literacy profile relative to self-reported and expert-rated health literacy to discriminate between high and low health literacy and then assessed Literacy Profiles' relationships with known correlates of health literacy, such as patient sociodemographics and a range of health-related outcomes, including ratings of physician communication, medication adherence, diabetes control, comorbidities, and utilization. PRINCIPAL FINDINGS: LP_SR and LP_Exp performed best in discriminating between high and low self-reported (C-statistics: 0.86 and 0.58, respectively) and expert-rated health literacy (C-statistics: 0.71 and 0.87, respectively) and were significantly associated with educational attainment, race/ethnicity, Consumer Assessment of Provider and Systems (CAHPS) scores, adherence, glycemia, comorbidities, and emergency department visits. CONCLUSIONS: Since health literacy is a potentially remediable explanatory factor in health care disparities, the development of automated health literacy indicators represents a significant accomplishment with broad clinical and population health applications. Health systems could apply literacy profiles to efficiently determine whether quality of care and outcomes vary by patient health literacy; identify at-risk populations for targeting tailored health communications and self-management support interventions; and inform clinicians to promote improvements in individual-level care.

Predicting the readability of physicians' secure messages to improve health communication using novel linguistic features: Findings from the ECLIPPSE study.

BACKGROUND: Low literacy skills impact important aspects of communication, including health-related information exchanges. Unsuccessful communication on the part of physician or patient contributes to lower quality of care, is associated with poorer chronic disease control, jeopardizes patient safety and can lead to unfavorable healthcare utilization patterns. To date, very little research has focused on digital communication between physicians and patients, such as secure messages sent via electronic patient portals. METHOD: The purpose of the current study is to develop an automated readability formula to better understand what elements of physicians' digital messages make them more or less difficult to understand. The formula is developed using advanced natural language processing (NLP) to predict human ratings of physician text difficulty. RESULTS: The results indicate that NLP indices that capture a diverse set of linguistic features predict the difficulty of physician messages better than classic readability tools such as Flesch Kincaid Grade Level. Our results also provide information about the textual features that best explain text readability. CONCLUSION: Implications for how the readability formula could provide feedback to physicians to improve digital health communication by promoting linguistic concordance between physician and patient are discussed.

Using natural language processing and machine learning to classify health literacy from secure messages: The ECLIPPSE study.

Limited health literacy is a barrier to optimal healthcare delivery and outcomes. Current measures requiring patients to self-report limitations are time-consuming and may be considered intrusive by some. This makes widespread classification of patient health literacy challenging. The objective of this study was to develop and validate "literacy profiles" as automated indicators of patients' health literacy to facilitate a non-intrusive, economic and more comprehensive characterization of health literacy among a health care delivery system's membership. To this end, three literacy profiles were generated based on natural language processing (combining computational linguistics and machine learning) using a sample of 283,216 secure messages sent from 6,941 patients to their primary care physicians. All patients were participants in Kaiser Permanente Northern California's DISTANCE Study. Performance of the three literacy profiles were compared against a gold standard of patient self-reported health literacy. Associations were analyzed between each literacy profile and patient demographics, health outcomes and healthcare utilization. T-tests were used for numeric data such as A1C, Charlson comorbidity index and healthcare utilization rates, and chi-square tests for categorical data such as sex, race, poor adherence and severe hypoglycemia. Literacy profiles varied in their test characteristics, with C-statistics ranging from 0.61-0.74. Relations between literacy profiles and health outcomes revealed patterns consistent with previous health literacy research: patients identified via literacy profiles indicative of limited health literacy: (a) were older and more likely of minority status; (b) had poorer medication adherence and glycemic control; and (c) exhibited higher rates of hypoglycemia, comorbidities and healthcare utilization. This represents the first successful attempt to employ natural language processing to estimate health literacy. Literacy profiles can offer an automated and economical way to identify patients with limited health literacy and greater vulnerability to poor health outcomes.

The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap.

This article introduces the second version of the Tool for the Automatic Analysis of Cohesion (TAACO 2.0). Like its predecessor, TAACO 2.0 is a freely available text analysis tool that works on the Windows, Mac, and Linux operating systems; is housed on a user's hard drive; is easy to use; and allows for batch processing of text files. TAACO 2.0 includes all the original indices reported for TAACO 1.0, but it adds a number of new indices related to local and global cohesion at the semantic level, reported by latent semantic analysis, latent Dirichlet allocation, and word2vec. The tool also includes a source overlap feature, which calculates lexical and semantic overlap between a source and a response text (i.e., cohesion between the two texts based measures of text relatedness). In the first study in this article, we examined the effects that cohesion features, prompt, essay elaboration, and enhanced cohesion had on expert ratings of text coherence, finding that global semantic similarity as reported by word2vec was an important predictor of coherence ratings. A second study was conducted to examine the source and response indices. In this study we examined whether source overlap between the speaking samples found in the TOEFL-iBT integrated speaking tasks and the responses produced by test-takers was predictive of human ratings of speaking proficiency. The results indicated that the percentage of keywords found in both the source and response and the similarity between the source document and the response, as reported by word2vec, were significant predictors of speaking quality. Combined, these findings help validate the new indices reported for TAACO 2.0.

Sentiment Analysis and Social Cognition Engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis.

This study introduces the Sentiment Analysis and Cognition Engine (SEANCE), a freely available text analysis tool that is easy to use, works on most operating systems (Windows, Mac, Linux), is housed on a user's hard drive (as compared to being accessed via an Internet interface), allows for batch processing of text files, includes negation and part-of-speech (POS) features, and reports on thousands of lexical categories and 20 component scores related to sentiment, social cognition, and social order. In the study, we validated SEANCE by investigating whether its indices and related component scores can be used to classify positive and negative reviews in two well-known sentiment analysis test corpora. We contrasted the results of SEANCE with those from Linguistic Inquiry and Word Count (LIWC), a similar tool that is popular in sentiment analysis, but is pay-to-use and does not include negation or POS features. The results demonstrated that both the SEANCE indices and component scores outperformed LIWC on the categorization tasks.

The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion.

This study introduces the Tool for the Automatic Analysis of Cohesion (TAACO), a freely available text analysis tool that is easy to use, works on most operating systems (Windows, Mac, and Linux), is housed on a user's hard drive (rather than having an Internet interface), allows for the batch processing of text files, and incorporates over 150 classic and recently developed indices related to text cohesion. The study validates TAACO by investigating how its indices related to local, global, and overall text cohesion can predict expert judgments of text coherence and essay quality. The findings of this study provide predictive validation of TAACO and support the notion that expert judgments of text coherence and quality are either negatively correlated or not predicted by local and overall text cohesion indices, but are positively predicted by global indices of cohesion. Combined, these findings provide supporting evidence that coherence for expert raters is a property of global cohesion and not of local cohesion, and that expert ratings of text quality are positively related to global cohesion.

Natural language processing in an intelligent writing strategy tutoring system.

The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider a broad array of linguistic, rhetorical, and contextual features. This study assesses the potential for computational indices to predict human ratings of essay quality. Past studies have demonstrated that linguistic indices related to lexical diversity, word frequency, and syntactic complexity are significant predictors of human judgments of essay quality but that indices of cohesion are not. The present study extends prior work by including a larger data sample and an expanded set of indices to assess new lexical, syntactic, cohesion, rhetorical, and reading ease indices. Three models were assessed. The model reported by McNamara, Crossley, and McCarthy (Written Communication 27:57-86, 2010) including three indices of lexical diversity, word frequency, and syntactic complexity accounted for only 6% of the variance in the larger data set. A regression model including the full set of indices examined in prior studies of writing predicted 38% of the variance in human scores of essay quality with 91% adjacent accuracy (i.e., within 1 point). A regression model that also included new indices related to rhetoric and cohesion predicted 44% of the variance with 94% adjacent accuracy. The new indices increased accuracy but, more importantly, afford the means to provide more meaningful feedback in the context of a writing tutoring system.