Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks.

Natural language understanding (NLU) has made massive progress driven by large benchmarks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained models on the long tail? We conceptualize the long tail using macro-level dimensions (underrepresented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning research for NLU. Our analysis asks three questions: (i) Which long tail dimensions do transfer learning studies target? (ii) Which properties of adaptation methods help improve performance on the long tail? (iii) Which methodological gaps have greatest negative impact on long tail performance? Our answers highlight major avenues for future research in transfer learning for the long tail. Lastly, using our meta-analysis framework, we perform a case study comparing the performance of various adaptation methods on clinical narratives, which provides interesting insights that may enable us to make progress along these future avenues.

Improving broad-coverage medical entity linking with semantic type prediction and large-scale datasets.

OBJECTIVES: Biomedical natural language processing tools are increasingly being applied for broad-coverage information extraction-extracting medical information of all types in a scientific document or a clinical note. In such broad-coverage settings, linking mentions of medical concepts to standardized vocabularies requires choosing the best candidate concepts from large inventories covering dozens of types. This study presents a novel semantic type prediction module for biomedical NLP pipelines and two automatically-constructed, large-scale datasets with broad coverage of semantic types. METHODS: We experiment with five off-the-shelf biomedical NLP toolkits on four benchmark datasets for medical information extraction from scientific literature and clinical notes. All toolkits adopt a staged approach of mention detection followed by two stages of medical entity linking: (1) generating a list of candidate concepts, and (2) picking the best concept among them. We introduce a semantic type prediction module to alleviate the problem of overgeneration of candidate concepts by filtering out irrelevant candidate concepts based on the predicted semantic type of a mention. We present MedType, a fully modular semantic type prediction model which we integrate into the existing NLP toolkits. To address the dearth of broad-coverage training data for medical information extraction, we further present WikiMed and PubMedDS, two large-scale datasets for medical entity linking. RESULTS: Semantic type filtering improves medical entity linking performance across all toolkits and datasets, often by several percentage points of F-1. Further, pretraining MedType on our novel datasets achieves state-of-the-art performance for semantic type prediction in biomedical text. CONCLUSIONS: Semantic type prediction is a key part of building accurate NLP pipelines for broad-coverage information extraction from biomedical text. We make our source code and novel datasets publicly available to foster reproducible research.

Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research.

Natural language processing (NLP) research combines the study of universal principles, through basic science, with applied science targeting specific use cases and settings. However, the process of exchange between basic NLP and applications is often assumed to emerge naturally, resulting in many innovations going unapplied and many important questions left unstudied. We describe a new paradigm of Translational NLP, which aims to structure and facilitate the processes by which basic and applied NLP research inform one another. Translational NLP thus presents a third research paradigm, focused on understanding the challenges posed by application needs and how these challenges can drive innovation in basic science and technology design. We show that many significant advances in NLP research have emerged from the intersection of basic principles with application needs, and present a conceptual framework outlining the stakeholders and key questions in translational research. Our framework provides a roadmap for developing Translational NLP as a dedicated research area, and identifies general translational principles to facilitate exchange between basic and applied research.

Equine-assisted psychotherapy with traumatized couples-Improvement of relationship quality and psychological symptoms.

Many traumatised individuals suffering from deployment related PTSD report severe problems in their relationships. Up until now, the therapeutic interventions used by the German Armed Forces have rarely targeted these problems through the integration of partners. For this reason, a Program designed specifically for couples was developed. In this prospective study equine-assisted psychotherapy was applied to soldiers and their spouses. The study population consisted of n = 36 couples, divided in n = 20 therapy group with a inpatient equine-assisted intervention and a 16-couples control group. After the intervention, numerous significant improvements occurred in the therapy group in the areas of current, somatic and communication problems, depressive symptoms and partnership quality but not in the control group. PTSD was reduced significantly on the sub-scale associated with negative thoughts. These results show that the intervention is an effective way to improve partnership quality and reduce the stressors that the partners of afflicted service members face.

Robust Knowledge Graph Completion with Stacked Convolutions and a Student Re-Ranking Network.

Knowledge Graph (KG) completion research usually focuses on densely connected benchmark datasets that are not representative of real KGs. We curate two KG datasets that include biomedical and encyclopedic knowledge and use an existing commonsense KG dataset to explore KG completion in the more realistic setting where dense connectivity is not guaranteed. We develop a deep convolutional network that utilizes textual entity representations and demonstrate that our model outperforms recent KG completion methods in this challenging setting. We find that our model's performance improvements stem primarily from its robustness to sparsity. We then distill the knowledge from the convolutional network into a student network that re-ranks promising candidate entities. This re-ranking stage leads to further improvements in performance and demonstrates the effectiveness of entity re-ranking for KG completion.

Ambiguity in medical concept normalization: An analysis of types and coverage in electronic health record datasets.

OBJECTIVES: Normalizing mentions of medical concepts to standardized vocabularies is a fundamental component of clinical text analysis. Ambiguity-words or phrases that may refer to different concepts-has been extensively researched as part of information extraction from biomedical literature, but less is known about the types and frequency of ambiguity in clinical text. This study characterizes the distribution and distinct types of ambiguity exhibited by benchmark clinical concept normalization datasets, in order to identify directions for advancing medical concept normalization research. MATERIALS AND METHODS: We identified ambiguous strings in datasets derived from the 2 available clinical corpora for concept normalization and categorized the distinct types of ambiguity they exhibited. We then compared observed string ambiguity in the datasets with potential ambiguity in the Unified Medical Language System (UMLS) to assess how representative available datasets are of ambiguity in clinical language. RESULTS: We found that <15% of strings were ambiguous within the datasets, while over 50% were ambiguous in the UMLS, indicating only partial coverage of clinical ambiguity. The percentage of strings in common between any pair of datasets ranged from 2% to only 36%; of these, 40% were annotated with different sets of concepts, severely limiting generalization. Finally, we observed 12 distinct types of ambiguity, distributed unequally across the available datasets, reflecting diverse linguistic and medical phenomena. DISCUSSION: Existing datasets are not sufficient to cover the diversity of clinical concept ambiguity, limiting both training and evaluation of normalization methods for clinical text. Additionally, the UMLS offers important semantic information for building and evaluating normalization methods. CONCLUSIONS: Our findings identify 3 opportunities for concept normalization research, including a need for ambiguity-specific clinical datasets and leveraging the rich semantics of the UMLS in new methods and evaluation measures for normalization.

Disparities in COVID-19 Incidence, Hospitalizations, and Testing, by Area-Level Deprivation - Utah, March 3-July 9, 2020.

Coronavirus disease 2019 (COVID-19) has had a substantial impact on racial and ethnic minority populations and essential workers in the United States, but the role of geographic social and economic inequities (i.e., deprivation) in these disparities has not been examined (1,2). As of July 9, 2020, Utah had reported 27,356 confirmed COVID-19 cases. To better understand how area-level deprivation might reinforce ethnic, racial, and workplace-based COVID-19 inequities (3), the Utah Department of Health (UDOH) analyzed confirmed cases of infection with SARS-CoV-2 (the virus that causes COVID-19), COVID-19 hospitalizations, and SARS-CoV-2 testing rates in relation to deprivation as measured by Utah's Health Improvement Index (HII) (4). Age-weighted odds ratios (weighted ORs) were calculated by weighting rates for four age groups (≤24, 25-44, 45-64, and ≥65 years) to a 2000 U.S. Census age-standardized population. Odds of infection increased with level of deprivation and were two times greater in high-deprivation areas (weighted OR = 2.08; 95% confidence interval [CI] = 1.99-2.17) and three times greater (weighted OR = 3.11; 95% CI = 2.98-3.24) in very high-deprivation areas, compared with those in very low-deprivation areas. Odds of hospitalization and testing also increased with deprivation, but to a lesser extent. Local jurisdictions should use measures of deprivation and other social determinants of health to enhance transmission reduction strategies (e.g., increasing availability and accessibility of SARS-CoV-2 testing and distributing prevention guidance) to areas with greatest need. These strategies might include increasing availability and accessibility of SARS-CoV-2 testing, contact tracing, isolation options, preventive care, disease management, and prevention guidance to facilities (e.g., clinics, community centers, and businesses) in areas with high levels of deprivation.

Racial and Ethnic Disparities Among COVID-19 Cases in Workplace Outbreaks by Industry Sector - Utah, March 6-June 5, 2020.

Improved understanding of the overall distribution of workplace coronavirus disease 2019 (COVID-19) outbreaks by industry sector could help direct targeted public health action; however, this has not been described. The Utah Department of Health (UDOH) analyzed COVID-19 surveillance data to describe workplace outbreaks by industry sectors. In this report, workplaces refer to non-health care, noncongregate-living, and noneducational settings. As of June 5, 2020, UDOH reported 277 COVID-19 outbreaks, 210 (76%) of which occurred in workplaces. Approximately 12% (1,389 of 11,448) of confirmed COVID-19 cases in Utah were associated with workplace outbreaks. The 210 workplace outbreaks occurred in 15 of 20 industry sectors;* nearly one half of all workplace outbreaks occurred in three sectors: Manufacturing (43; 20%), Construction (32; 15%) and Wholesale Trade (29; 14%); 58% (806 of 1,389) of workplace outbreak-associated cases occurred in these three sectors. Although 24% of Utah's workforce in all 15 affected sectors identified as Hispanic or Latino (Hispanic) or a race other than non-Hispanic white (nonwhite†) (1), 73% (970 of 1,335) of workplace outbreak-associated COVID-19 cases were in persons who identified as Hispanic or nonwhite. Systemic social inequities have resulted in the overrepresentation of Hispanic and nonwhite workers in frontline occupations where exposure to SARS-CoV-2, the virus that causes COVID-19, might be higher (2); extra vigilance in these sectors is needed to ensure prevention and mitigation strategies are applied equitably and effectively to workers of racial and ethnic groups disproportionately affected by COVID-19. Health departments can adapt workplace guidance to each industry sector affected by COVID-19 to account for different production processes and working conditions.

Factors Affecting Health Care Engagement of Patients With End-Stage Heart Failure: An Exploratory Survey Study.

Background. The decision to receive a permanent left ventricular assist device (LVAD) to treat end-stage heart failure (HF) involves understanding and weighing the risks and benefits of a highly invasive treatment. The goal of this study was to characterize end-stage HF patients across parameters that may affect their decision making and to inform the development of an LVAD decision support tool. Methods. A survey of 35 end-stage HF patients at an LVAD implant hospital was performed to characterize their information-seeking habits, interaction with physicians, technology use, numeracy, and concerns about their health. Survey responses were analyzed using descriptive statistics, grounded theory method, and Bayesian network learning. Results. Most patients indicated an interest in using some type of decision support tool (roadmap of health progression: 46%, n = 16; personal prognosis: 51%, n = 18; short videos of patients telling stories of their experiences with an LVAD: 57%, n = 20). Information patients desired in a hypothetical decision support tool fell into the following topics: prognoses for health outcomes, technical information seeking, expressing emotions, and treatment decisions. Desire for understanding their condition was closely related to whether they had difficult interpreting their electronic medical record in the past. Conclusions. Most patients reported interest in engaging in their health care decision making and seeing their prognosis and electronic health record information. Patients who were less interested in their own treatment decisions were characterized by having less success understanding their health information. Design of a decision support tool for potential LVAD patients should consider a spectrum of health literacy and include information beyond the technical specifications of LVAD support.

Attitudes towards disclosing a mental illness among German soldiers and their comrades.

Many soldiers with mental illness (SWMIs) struggle with the decision whether to disclose their condition in or outside the military. This study therefore explored views on (self-)labeling as 'mentally ill', experiences of discrimination and coping, risks and benefits of (non-)disclosure, service use, disclosure decisions and consequences of disclosing. Active-duty SWMIs as well as soldiers without mental illness (commanding officers; enlisted ranks) and military social workers participated in focus groups. Transcripts were analyzed using qualitative content analysis. SWMIs perceived negative stereotypes about their group (weakness, incompetence, blame, malingering) and saw stigma as a barrier to help-seeking. Being labeled 'mentally ill' was seen as harmful for one's career. Self-labeling led to poor self-esteem, greater need for help and feelings of weakness. Many SWMIs had experienced discrimination, such as gossip or inappropriate comments. Social isolation was a disadvantage of secrecy. Most SWMIs preferred selective disclosure and many did not disclose to their family. Military staff without mental illness expressed partly different views and described organizational challenges posed by SWMIs. Our findings suggest that disclosure decisions are personal and difficult and that stigma remains a barrier to re-integration and recovery of SWMIs in the military. Implications for interventions to support SWMIs are discussed.

Modeling Dialogue Acts with Content Word Filtering and Speaker Preferences.

We present an unsupervised model of dialogue act sequences in conversation. By modeling topical themes as transitioning more slowly than dialogue acts in conversation, our model de-emphasizes content-related words in order to focus on conversational function words that signal dialogue acts. We also incorporate speaker tendencies to use some acts more than others as an additional predictor of dialogue act prevalence beyond temporal dependencies. According to the evaluation presented on two dissimilar corpora, the CNET forum and NPS Chat corpus, the effectiveness of each modeling assumption is found to vary depending on characteristics of the data. De-emphasizing content-related words yields improvement on the CNET corpus, while utilizing speaker tendencies is advantageous on the NPS corpus. The components of our model complement one another to achieve robust performance on both corpora and outperform state-of-the-art baseline models.

Data mining and education.

An emerging field of educational data mining (EDM) is building on and contributing to a wide variety of disciplines through analysis of data coming from various educational technologies. EDM researchers are addressing questions of cognition, metacognition, motivation, affect, language, social discourse, etc. using data from intelligent tutoring systems, massive open online courses, educational games and simulations, and discussion forums. The data include detailed action and timing logs of student interactions in user interfaces such as graded responses to questions or essays, steps in rich problem solving environments, games or simulations, discussion forum posts, or chat dialogs. They might also include external sensors such as eye tracking, facial expression, body movement, etc. We review how EDM has addressed the research questions that surround the psychology of learning with an emphasis on assessment, transfer of learning and model discovery, the role of affect, motivation and metacognition on learning, and analysis of language data and collaborative learning. For example, we discuss (1) how different statistical assessment methods were used in a data mining competition to improve prediction of student responses to intelligent tutor tasks, (2) how better cognitive models can be discovered from data and used to improve instruction, (3) how data-driven models of student affect can be used to focus discussion in a dialog-based tutoring system, and (4) how machine learning techniques applied to discussion data can be used to produce automated agents that support student learning as they collaborate in a chat room or a discussion board.

Automating annotation of information-giving for analysis of clinical conversation.

OBJECTIVE: Coding of clinical communication for fine-grained features such as speech acts has produced a substantial literature. However, annotation by humans is laborious and expensive, limiting application of these methods. We aimed to show that through machine learning, computers could code certain categories of speech acts with sufficient reliability to make useful distinctions among clinical encounters. MATERIALS AND METHODS: The data were transcripts of 415 routine outpatient visits of HIV patients which had previously been coded for speech acts using the Generalized Medical Interaction Analysis System (GMIAS); 50 had also been coded for larger scale features using the Comprehensive Analysis of the Structure of Encounters System (CASES). We aggregated selected speech acts into information-giving and requesting, then trained the machine to automatically annotate using logistic regression classification. We evaluated reliability by per-speech act accuracy. We used multiple regression to predict patient reports of communication quality from post-visit surveys using the patient and provider information-giving to information-requesting ratio (briefly, information-giving ratio) and patient gender. RESULTS: Automated coding produces moderate reliability with human coding (accuracy 71.2%, κ=0.57), with high correlation between machine and human prediction of the information-giving ratio (r=0.96). The regression significantly predicted four of five patient-reported measures of communication quality (r=0.263-0.344). DISCUSSION: The information-giving ratio is a useful and intuitive measure for predicting patient perception of provider-patient communication quality. These predictions can be made with automated annotation, which is a practical option for studying large collections of clinical encounters with objectivity, consistency, and low cost, providing greater opportunity for training and reflection for care providers.

A Classification Approach for Risk Prognosis of Patients on Mechanical Ventricular Assistance.

The identification of optimal candidates for ventricular assist device (VAD) therapy is of great importance for future widespread application of this life-saving technology. During recent years, numerous traditional statistical models have been developed for this task. In this study, we compared three different supervised machine learning techniques for risk prognosis of patients on VAD: Decision Tree, Support Vector Machine (SVM) and Bayesian Tree-Augmented Network, to facilitate the candidate identification. A predictive (C4.5) decision tree model was ultimately developed based on 6 features identified by SVM with assistance of recursive feature elimination. This model performed better compared to the popular risk score of Lietz et al. with respect to identification of high-risk patients and earlier survival differentiation between high- and low- risk candidates.

When are tutorial dialogues more effective than reading?

It is often assumed that engaging in a one-on-one dialogue with a tutor is more effective than listening to a lecture or reading a text. Although earlier experiments have not always supported this hypothesis, this may be due in part to allowing the tutors to cover different content than the noninteractive instruction. In 7 experiments, we tested the interaction hypothesis under the constraint that (a) all students covered the same content during instruction, (b) the task domain was qualitative physics, (c) the instruction was in natural language as opposed to mathematical or other formal languages, and (d) the instruction conformed with a widely observed pattern in human tutoring: Graesser, Person, and Magliano's 5-step frame. In the experiments, we compared 2 kinds of human tutoring (spoken and computer mediated) with 2 kinds of natural-language-based computer tutoring (Why2-Atlas and Why2-AutoTutor) and 3 control conditions that involved studying texts. The results depended on whether the students' preparation matched the content of the instruction. When novices (students who had not taken college physics) studied content that was written for intermediates (students who had taken college physics), then tutorial dialogue was reliably more beneficial than less interactive instruction, with large effect sizes. When novices studied material written for novices or intermediates studied material written for intermediates, then tutorial dialogue was not reliably more effective than the text-based control conditions.