Emergent linguistic structure in artificial neural networks trained by self-supervision.

This paper explores the knowledge of linguistic structure learned by large artificial neural networks, trained via self-supervision, whereby the model simply tries to predict a masked word in a given context. Human language communication is via sequences of words, but language understanding requires constructing rich hierarchical structures that are never observed explicitly. The mechanisms for this have been a prime mystery of human language acquisition, while engineering work has mainly proceeded by supervised learning on treebanks of sentences hand labeled for this latent structure. However, we demonstrate that modern deep contextual language models learn major aspects of this structure, without any explicit supervision. We develop methods for identifying linguistic hierarchical structure emergent in artificial neural networks and demonstrate that components in these models focus on syntactic grammatical relationships and anaphoric coreference. Indeed, we show that a linear transformation of learned embeddings in these models captures parse tree distances to a surprising degree, allowing approximate reconstruction of the sentence tree structures normally assumed by linguists. These results help explain why these models have brought such large improvements across many language-understanding tasks.

Translating Akkadian to English with neural machine translation.

Cuneiform is one of the earliest writing systems in recorded human history (ca. 3,400 BCE-75 CE). Hundreds of thousands of such texts were found over the last two centuries, most of which are written in Sumerian and Akkadian. We show the high potential in assisting scholars and interested laypeople alike, by using natural language processing (NLP) methods such as convolutional neural networks (CNN), to automatically translate Akkadian from cuneiform Unicode glyphs directly to English (C2E) and from transliteration to English (T2E). We show that high-quality translations can be obtained when translating directly from cuneiform to English, as we get 36.52 and 37.47 Best Bilingual Evaluation Understudy 4 (BLEU4) scores for C2E and T2E, respectively. For C2E, our model is better than the translation memory baseline in 9.43, and for T2E, the difference is even higher and stands at 13.96. The model achieves best results in short- and medium-length sentences (c. 118 or less characters). As the number of digitized texts grows, the model can be improved by further training as part of a human-in-the-loop system which corrects the results.

Shared computational principles for language processing in humans and deep language models.

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Targeting the actin nucleation promoting factor WASp provides a therapeutic approach for hematopoietic malignancies.

Cancer cells depend on actin cytoskeleton rearrangement to carry out hallmark malignant functions including activation, proliferation, migration and invasiveness. Wiskott-Aldrich Syndrome protein (WASp) is an actin nucleation-promoting factor and is a key regulator of actin polymerization in hematopoietic cells. The involvement of WASp in malignancies is incompletely understood. Since WASp is exclusively expressed in hematopoietic cells, we performed in silico screening to identify small molecule compounds (SMCs) that bind WASp and promote its degradation. We describe here one such identified molecule; this WASp-targeting SMC inhibits key WASp-dependent actin processes in several types of hematopoietic malignancies in vitro and in vivo without affecting naïve healthy cells. This small molecule demonstrates limited toxicity and immunogenic effects, and thus, might serve as an effective strategy to treat specific hematopoietic malignancies in a safe and precisely targeted manner.

A conformational change within the WAVE2 complex regulates its degradation following cellular activation.

WASp family Verprolin-homologous protein-2 (WAVE2), a member of the Wiskott-Aldrich syndrome protein (WASp) family of actin nucleation promoting factors, is a central regulator of actin cytoskeleton polymerization and dynamics. Multiple signaling pathways operate via WAVE2 to promote the actin-nucleating activity of the actin-related protein 2/3 (Arp2/3) complex. WAVE2 exists as a part of a pentameric protein complex known as the WAVE regulatory complex (WRC), which is unstable in the absence of its individual proteins. While the involvement of WAVE2 in actin polymerization has been well documented, its negative regulation mechanism is poorly characterized to date. Here, we demonstrate that WAVE2 undergoes ubiquitylation in a T-cell activation dependent manner, followed by proteasomal degradation. The WAVE2 ubiquitylation site was mapped to lysine 45, located at the N-terminus where WAVE2 binds to the WRC. Using Förster resonance energy transfer (FRET), we reveal that the autoinhibitory conformation of the WRC maintains the stability of WAVE2 in resting cells; the release of autoinhibition following T-cell activation facilitates the exposure of WAVE2 to ubiquitylation, leading to its degradation. The dynamic conformational structures of WAVE2 during cellular activation dictate its degradation.