No influence of regular rhythmic priming on grammaticality judgment and sentence comprehension in English-speaking children.

A growing body of research has demonstrated the association between music and language, particularly between rhythm and grammar skills in children. A compelling piece of evidence for the influence of music on language comes from findings that a brief exposure to regular musical rhythm improved subsequent syntactic language performance in children. Nevertheless, those observations were made on one particular task, i.e., grammaticality judgment, mostly with French-speaking children. Here, we sought to corroborate and extend the rhythmic priming effect with English-speaking children aged 7 to 12 years who underwent two different syntactic tasks on spoken sentences: one involving judgment on morphosyntactic well-formedness (grammaticality judgment) and the other requiring noun-verb relation analysis (sentence comprehension), both following either regular or irregular rhythmic priming. Half of the children were administered synthetic speech stimuli (Experiment 1), and the other half were presented with natural speech (Experiment 2). Across the two experiments, we did not find any rhythmic priming effect; children's performance on both the grammaticality judgment and sentence comprehension tasks was comparable irrespective of the regularity in prior rhythms. These results imply that the positive influence of regular rhythmic priming on syntactic processing may be confined to specific language or age populations, warranting further investigation.

Children are eager to take credit for prosocial acts, and cost affects this tendency.

We report two experiments on children's tendency to enhance their reputations through communicative acts. In the experiments, 4-year-olds (N = 120) had the opportunity to inform a social partner that they had helped him in his absence. In a first experiment, we pitted a prosocial act ("Let's help clean up for Doggie!") against an instrumental act ("Let's move these out of our way"). Children in the prosocial condition were quicker to inform their partner of the act and more likely to protest when another individual was given credit for it. In a second experiment, we replicated the prosocial condition but with a new manipulation: high-cost versus low-cost helping. We manipulated both the language surrounding cost (i.e., "This will be pretty tough to clean up" vs. "It will be really easy to clean this up") and how difficult the task itself was. As predicted, children in the high-cost condition were quicker to inform their partner of the act and more likely to take back credit for it. These results suggest that even 4-year-old children make active attempts to elicit positive reputational judgments for their prosocial acts, with cost as a moderating factor.

SGABU computational platform for multiscale modeling: Bridging the gap between education and research.

BACKGROUND AND OBJECTIVE: In accordance with the latest aspirations in the field of bioengineering, there is a need to create a web accessible, but powerful cloud computational platform that combines datasets and multiscale models related to bone modeling, cancer, cardiovascular diseases and tissue engineering. The SGABU platform may become a powerful information system for research and education that can integrate data, extract information, and facilitate knowledge exchange with the goal of creating and developing appropriate computing pipelines to provide accurate and comprehensive biological information from the molecular to organ level. METHODS: The datasets integrated into the platform are obtained from experimental and/or clinical studies and are mainly in tabular or image file format, including metadata. The implementation of multiscale models, is an ambitious effort of the platform to capture phenomena at different length scales, described using partial and ordinary differential equations, which are solved numerically on complex geometries with the use of the finite element method. The majority of the SGABU platform's simulation pipelines are provided as Common Workflow Language (CWL) workflows. Each of them requires creating a CWL implementation on the backend and a user-friendly interface using standard web technologies. Platform is available at https://sgabu-test.unic.kg.ac.rs/login. RESULTS: The main dashboard of the SGABU platform is divided into sections for each field of research, each one of which includes a subsection of datasets and multiscale models. The datasets can be presented in a simple form as tabular data, or using technologies such as Plotly.js for 2D plot interactivity, Kitware Paraview Glance for 3D view. Regarding the models, the usage of Docker containerization for packing the individual tools and CWL orchestration for describing inputs with validation forms and outputs with tabular views for output visualization, interactive diagrams, 3D views and animations. CONCLUSIONS: In practice, the structure of SGABU platform means that any of the integrated workflows can work equally well on any other bioengineering platform. The key advantage of the SGABU platform over similar efforts is its versatility offered with the use of modern, modular, and extensible technology for various levels of architecture.

"Dividing the labor": Lexical verbs and the linguistic encoding of physical support in 2- to 4.5-year-old children.

Infants reason about support configurations (e.g., teddy bear on table) and young children talk about a variety of support relations, including support-from-below (e.g., apple on table) and many other types (e.g., Band-Aid on leg, picture on wall). Given this wide variation in support types, we asked whether early differentiation of the semantic space of support may play a key role in helping children to learn spatial language in this domain. Previous research has shown such differentiation with 20-month-olds mapping the basic locative construction (BE on) to support-from-below (cube on top of box), but not to a mechanical support configuration (cube on side of box via adhesion). Older children and adults show the same differentiation, with preferential mapping of BE on to support-from-below and lexical verbs to mechanical support. We further explored the development of this differentiation by testing how children aged 2 to 4.5 years map lexical verbs to a wide variety of support configurations. In Experiment 1, using an intermodal preferential pointing paradigm, we found that 2- to 3.5-year-olds map a lexical verb phrase ("sticks to") to mechanical support via adhesion. In Experiments 2 and 3, we expanded the range of mechanical support relations and used production and forced-choice tasks to ask whether 2- to 4.5-year-olds also encode mechanical relations using lexical verbs. We found that they do. These findings suggest continuity between infancy and childhood in the way that children use spatial language to differentially map to support-from-below versus mechanical support and raise new questions about how mechanical support language develops.

The relation of bilingual cognitive skills to the second language writing performance of primary grade students.

The purposes of the current study were to examine the effects of cognitive and reading skills (i.e., working memory [WM], oral language development [OLD], and reading skills) on second language (L2) writing performance as well as the changes in these relationships across different grades among Spanish-speaking children learning English. A battery of measures assessing English and Spanish WM, OLD, reading skills, and English writing were administered to 494 English learners in Grades 1 to 3. Path analysis was conducted for each grade separately in both English and Spanish models. The findings indicated that the relationships between English writing performance and English cognitive and reading skills became stronger as the grades increased. However, the relationships between English writing and the Spanish cognitive and reading determinants were mixed, indicating a statistically significant relationship with Spanish WM and reading skills for Grade 2 and 3 students but not with OLD across all grades. Implications for L2 writing development are discussed.

Brain bases of morphological awareness and longitudinal word reading outcomes.

Children's spoken language skills are essential to the development of the "reading brain," or the neurocognitive systems that underlie successful literacy. Morphological awareness, or sensitivity to the smallest units of meaning, is a language skill that facilitates fluent recognition of meaning in print. Yet despite the growing evidence that morphology is integral to literacy success, associations among morphological awareness, literacy acquisition, and brain development remain largely unexplored. To address this gap, we conducted a longitudinal investigation with 75 elementary school children (5-11 years of age) who completed an auditory morphological awareness neuroimaging task at Time 1 as well as literacy assessments at both Time 1 and Time 2 (1.5 years later). Findings reveal longitudinal brain-behavior associations between morphological processing at Time 1 and reading outcomes at Time 2. First, activation in superior temporal brain regions involved in word segmentation was associated with both future reading skill and steeper reading gains over time. Second, a wider array of brain regions across the language network were associated with polymorphemic word reading as compared with broader word reading skill (reading both simple and complex words). Together, these findings reinforce the importance of word segmentation skills in learning to read and highlight the importance of considering complex word reading skills in building comprehensive neurocognitive models of literacy. This study fills a gap in our knowledge of how processing meaningful units in speech may help to explain differences in children's reading development over time and informs ongoing theoretical questions about the role of morphology in learning to read.

Developmental trajectories of spoken language comprehension and functional communication in children with cerebral palsy: A prospective cohort study.

AIM: To investigate spoken language comprehension (SLC), single-word comprehension (SWC), functional communication development, and their determinants, in children with cerebral palsy. METHOD: This was a prospective cohort study in the Netherlands spanning 2 years 6 months. The main outcomes were SLC and SWC, assessed by the Computer-Based instrument for Low motor Language Testing (C-BiLLT) and the Peabody Picture Vocabulary Test-III-NL (PPVT-III-NL) respectively; and functional communication, measured by a subscale of the Focus on the Outcomes of Communication Under Six-34 (FOCUS-34). Linear mixed models were used to determine developmental trajectories, which were compared with norm and reference data. Potential determinants, for example intellectual functions, speech production, functional communication level (classified with the Communication Function Classification System, CFCS), and functional mobility, were added to assess their effects. RESULTS: Children with cerebral palsy (n = 188; mean age 59 months, range 17-110) were monitored for 2 years 6 months. Developmental trajectories for SLC (C-BiLLT) and SWC (PPVT-III-NL) were nonlinear; those for functional communication (FOCUS-34) were linear. Compared with norm and reference groups, significantly delayed SLC, SWC, and functional communication development were found. Determinants for SLC and SWC were intellectual functions and functional communication level (CFCS); and for functional communication development (FOCUS-34), speech production and arm-hand functioning. INTERPRETATION: Children with cerebral palsy showed delayed SLC, SWC, and functional communication development compared with norm and reference groups. Remarkably, functional mobility was not associated with the development of SLC, SWC, or functional communication. WHAT THIS PAPER ADDS: Children with cerebral palsy have delayed spoken language comprehension (SLC), single-word comprehension (SWC), and functional communication development, compared to norm and reference data. Determinants for SLC and SWC development are intellectual functions and functional communication level. Determinants for functional communication development are speech production and arm-hand functioning. Functional mobility is not associated with SLC, SWC, or functional communication.

Large language models and the future of rheumatology: assessing impact and emerging opportunities.

PURPOSE OF REVIEW: Large language models (LLMs) have grown rapidly in size and capabilities as more training data and compute power has become available. Since the release of ChatGPT in late 2022, there has been growing interest and exploration around potential applications of LLM technology. Numerous examples and pilot studies demonstrating the capabilities of these tools have emerged across several domains. For rheumatology professionals and patients, LLMs have the potential to transform current practices in medicine. RECENT FINDINGS: Recent studies have begun exploring capabilities of LLMs that can assist rheumatologists in clinical practice, research, and medical education, though applications are still emerging. In clinical settings, LLMs have shown promise in assist healthcare professionals enabling more personalized medicine or generating routine documentation like notes and letters. Challenges remain around integrating LLMs into clinical workflows, accuracy of the LLMs and ensuring patient data confidentiality. In research, early experiments demonstrate LLMs can offer analysis of datasets, with quality control as a critical piece. Lastly, LLMs could supplement medical education by providing personalized learning experiences and integration into established curriculums. SUMMARY: As these powerful tools continue evolving at a rapid pace, rheumatology professionals should stay informed on how they may impact the field.

Preschoolers retain more details from event sequences 1 week following an in-group demonstration.

Previous research highlights that the learning processes of preschool-aged children are influenced by the cultural group membership of the information sources. As of yet, however, no study has aimed to explore the influence of cultural group membership on the long-term retention of novel information. In the current study, 4-year-old children observed three event sequences that were demonstrated by either an adult speaking their native language or a foreign language speaker. In Experiment 1, children (N = 56) were allowed to imitate the events immediately. Results showed that the average number of accurately reproduced details (native = 3.26; foreign = 3.11) and the order of event elements (native = 1.69; foreign = 1.49) did not significantly differ in the two conditions. In Experiment 2, children (N = 56) were allowed to imitate only following a 1-week delay. In this case, children retained more details (native = 2.6; foreign = 2.2) and reproduced the order in the event sequences more accurately (native = 1.18; foreign = 0.87) following a native demonstration. The behavior of children in all conditions differed from a baseline group without any instruction (n = 15). These findings show that preschoolers retain more information in the long term when it was demonstrated to them by a member of their own culture. Importantly, they also learn from people belonging to different cultures-as evidenced by both the lack of difference in Experiment 1 and the difference between the out-group condition of Experiment 2 and the baseline.

The development of rhythmic categories as revealed through an iterative production task.

Both humans and non-humans (e.g. birds and primates) preferentially produce and perceive auditory rhythms with simple integer ratios. In addition, these preferences (biases) tend to reflect specific integer-ratio rhythms that are common to one's cultural listening experience. To better understand the developmental trajectory of these biases, we estimated children's rhythm biases across the entire rhythm production space of simple (e.g., ratios of 1, 2, and 3) three-interval rhythms. North American children aged 6-11 years completed an iterative rhythm production task, in which they attempted to tap in synchrony with repeating three-interval rhythms chosen randomly from the space. For each rhythm, the child's produced rhythm was presented back to them as the stimulus, and over the course of 5 such iterations we used their final reproductions to estimate their rhythmic biases or priors. Results suggest that regardless of the initial rhythm, after 5 iterations, children's tapping converged on rhythms with (nearly) simple integer ratios, indicating that, like adults, their rhythmic priors consist of rhythms with simple-integer ratios. Furthermore, the relative weights (or prominence of different rhythmic priors) observed in children were highly correlated with those of adults. However, we also observed some age-related changes, especially for the ratio types that vary most across cultures. In an additional rhythm perception task, children were better at detecting rhythmic disruptions to a culturally familiar rhythm (in 4/4 m with 2:1:1 ratio pattern) than to a culturally unfamiliar rhythm (7/8 m with 3:2:2 ratios), and performance in this task was correlated with tapping variability in the iterative task. Taken together, our findings provide evidence that children as young as 6-years-old exhibit simple integer-ratio categorical rhythm priors in their rhythm production that closely resemble those of adults in the same culture.

Discourse coherence modulates use of predictive processing during sentence comprehension.

Context has been shown to be vitally important for comprehension. Lexical processing is facilitated when words are highly predictable given their local sentence context, suggesting that people pre-activate likely upcoming words to aid comprehension. However, this facilitation is affected by knowledge about the global context in which comprehension takes place: people predict less when in an environment where expectations are frequently violated. The current study investigated whether discourse coherence is an additional cue that comprehenders use to modulate lexical prediction. In a series of online, self-paced reading experiments, participants read target sentences preceded by short contextual preambles. Local facilitation effects were manipulated through the cloze probability of a critical word within the target sentence and discourse coherence was manipulated by varying the degree to which the target sentence was consistent with the information presented in the preamble. In the first two experiments, target sentences were read more slowly when they occurred in less coherent discourses, but no local facilitation effects were observed. In the third experiment, we strengthened the predictability manipulation by using semantically anomalous critical words. In this experiment, predictable words were processed more quickly and anomalous words more slowly when they occurred in highly coherent discourse. Our results suggest that comprehenders are sensitive to shifts in the topic of discourse and that they downregulate predictive processing when they encounter incoherence in the discourse. This is consistent with recent theoretical accounts suggesting that comprehenders flexibly engage in predictive processing, pre-activating semantic and lexical information less when their expectations are less likely to be reliable.

Does bilingual experience influence statistical language learning?

Statistical language learning (SL) tasks measure different aspects of foreign language learning. Studies have used SL tasks to investigate whether bilingual experience confers advantages in acquiring additional languages through implicit processes. However, the results have been inconsistent, which may be related to bilingualism-related features (e.g., degree of dissimilarity between the specific language pair) and other variables such as specific processes that are targeted by the SL task. In the present study, we compared the performance of one Spanish monolingual and two bilingual (Spanish-Basque and Spanish-English) groups across three well-established SL tasks. Each task targeted a different aspect of foreign language learning; specifically, word segmentation, morphological rule generalization, and word-referent learning. In Experiment 1, we manipulated sub-lexical phonotactic patterns to vary the difficulty of three SL tasks, with the results showing no differences between the groups in word segmentation. In Experiment 2, we included non-adjacent dependencies to target affixal morphology rule learning, but again no group-related differences were found. In Experiment 3, we addressed word learning using an audio-visual SL task combining exclusive and multiple word-referent mappings, and found that bilinguals outperformed monolinguals, suggesting that bilingualism may exert influences on SL at the lexical level. This advantage might have been mediated by the high working memory demands required to perform the task. Summarizing, this study shows no evidence for a general bilingual advantage in SL, although bilinguals may outperform monolinguals under specific experimental conditions such as SL tasks that place high demands on working memory processes. In addition, the similar performance of Spanish-Basque and Spanish-English bilinguals across all three SL tasks suggests that the degree of dissimilarity between pairs of spoken languages does not modulate SL skills.

Universality in eye movements and reading: A replication with increased power.

Liversedge, Drieghe, Li, Yan, Bai and Hyönä (2016) reported an eye movement study that investigated reading in Chinese, Finnish and English (languages with markedly different orthographic characteristics). Analyses of the eye movement records showed robust differences in fine grained characteristics of eye movements between languages, however, overall sentence reading times did not differ. Liversedge et al. interpreted the entire set of results across languages as reflecting universal aspects of processing in reading. However, the study has been criticized as being statistically underpowered (Brysbaert, 2019) given that only 19-21 subjects were tested in each language. Also, given current best practice, the original statistical analyses can be considered to be somewhat weak (e.g., no inclusion of random slopes and no formal comparison of performance between the three languages). Finally, the original study did not include any formal statistical model to assess effects across all three languages simultaneously. To address these (and some other) concerns, we tested at least 80 new subjects in each language and conducted formal statistical modeling of our data across all three languages. To do this, we included an index that captured variability in visual complexity in each language. Unlike the original findings, the new analyses showed shorter total sentence reading times for Chinese relative to Finnish and English readers. The other main findings reported in the original study were consistent. We suggest that the faster reading times for Chinese subjects occurred due to cultural changes that have taken place in the decade or so that lapsed between when the original and current subjects were tested. We maintain our view that the results can be taken to reflect universality in aspects of reading and we evaluate the claims regarding a lack of statistical power that were levelled against the original article.

Of words and whistles: Statistical learning operates similarly for identical sounds perceived as speech and non-speech.

Statistical learning is an ability that allows individuals to effortlessly extract patterns from the environment, such as sound patterns in speech. Some prior evidence suggests that statistical learning operates more robustly for speech compared to non-speech stimuli, supporting the idea that humans are predisposed to learn language. However, any apparent statistical learning advantage for speech could be driven by signal acoustics, rather than the subjective perception per se of sounds as speech. To resolve this issue, the current study assessed whether there is a statistical learning advantage for ambiguous sounds that are subjectively perceived as speech-like compared to the same sounds perceived as non-speech, thereby controlling for acoustic features. We first induced participants to perceive sine-wave speech (SWS)-a degraded form of speech not immediately perceptible as speech-as either speech or non-speech. After this induction phase, participants were exposed to a continuous stream of repeating trisyllabic nonsense words, composed of SWS syllables, and then completed an explicit familiarity rating task and an implicit target detection task to assess learning. Critically, participants showed robust and equivalent performance on both measures, regardless of their subjective speech perception. In contrast, participants who perceived the SWS syllables as more speech-like showed better detection of individual syllables embedded in speech streams. These results suggest that speech perception facilitates processing of individual sounds, but not the ability to extract patterns across sounds. Our findings suggest that statistical learning is not influenced by the perceived linguistic relevance of sounds, and that it may be conceptualized largely as an automatic, stimulus-driven mechanism.

Conditionals in context: Brain signatures of prediction in discourse processing.

Comprehenders are known to generate expectations about upcoming linguistic input at the sentence and discourse level. However, most previous studies on prediction focused mainly on word-induced brain activity rather than examining neural activity preceding a critical stimulus in discourse processing, where prediction actually takes place. In this EEG study, participants were presented with multiple sentences resembling a discourse including conditional sentences with either only if or if, which are characterized by different semantics, triggering stronger or weaker predictions about the possible continuation of the presented discourses, respectively. Results revealed that discourses including only if, as compared to discourses with bare if, triggered an increased predictive neural activity before the expected critical word, resembling the readiness potential. Moreover, word-induced P300 brain responses were found to be enhanced by unpredictable discourse continuations and reduced in predictable discourse continuations. Intriguingly, brain responses preceding and following the critical word were found to be correlated, which yields evidence for predictive activity modulating word-induced processing on the discourse level. These findings shed light on the predictive nature of neural processes at the discourse level, critically advancing our understanding of the functional interconnection between discourse understanding and prediction processes in brain and mind.

A novel task and methods to evaluate inter-individual variation in audio-visual associative learning.

Learning audio-visual associations is foundational to a number of real-world skills, such as reading acquisition or social communication. Characterizing individual differences in such learning has therefore been of interest to researchers in the field. Here, we present a novel audio-visual associative learning task designed to efficiently capture inter-individual differences in learning, with the added feature of using non-linguistic stimuli, so as to unconfound language and reading proficiency of the learner from their more domain-general learning capability. By fitting trial-by-trial performance in our novel learning task using simple-to-use statistical tools, we demonstrate the expected inter-individual variability in learning rate as well as high precision in its estimation. We further demonstrate that such measured learning rate is linked to working memory performance in Italian-speaking (N = 58) and French-speaking (N = 51) adults. Finally, we investigate the extent to which learning rate in our task, which measures cross-modal audio-visual associations while mitigating familiarity confounds, predicts reading ability across participants with different linguistic backgrounds. The present work thus introduces a novel non-linguistic audio-visual associative learning task that can be used across languages. In doing so, it brings a new tool to researchers in the various domains that rely on multi-sensory integration from reading to social cognition or socio-emotional learning.

Translation, cross-cultural adaptation, and validation of health and self-management in diabetes questionnaire (HASMID-10) into Brazilian Portuguese.

BACKGROUND: Considering the ability of the health and self-management in diabetes questionnaire (HASMID-10) to verify the impact of self-management on diabetes, we highlight its relevance to scientific research and clinical applicability. However, to date, no study has been conducted to scientifically support its use in other languages. OBJECTIVE: To translate, cross-culturally adapt, and validate the HASMID-10 into the Brazilian Portuguese. DESIGN AND SETTING: A translation, cross-cultural adaptation, and validation study conducted at Ceuma University. METHODS: Study was conducted in accordance with the Guidelines for the Process of Cross-Cultural Adaptation of Self-Report Measures and Consensus-based Standards for the Selection of Health Measurement Instruments. We included participants of both sexes diagnosed with diabetes, aged between 18 and 64 years, and without cognitive deficits or any other limitations that would prevent them from answering the questionnaire. We assessed participants using the problem areas in diabetes (PAID) scale and HASMID-10. We assessed reliability using a test-retest model with a 7-day interval between assessments. We used intraclass correlation coefficient (ICC), 95% confidence interval (CI), standard error of measurement (SEM), minimum detectable difference (MDD), Spearman correlation coefficient, and floor and ceiling effects. RESULTS: Sample comprised 116 participants, most of whom were women, overweight, non-practitioners of physical activity, and nonsmokers. We observed significant correlations (P = 0.006; rho = -0.256) between the HASMID-10 and PAID, adequate reliability (ICC = 0.780) and internal consistency (Cronbach's alpha = 0.796). No ceiling or floor effects were observed. CONCLUSION: HASMID-10 has adequate measurement properties and may be used for Brazilians.

pLM4ACE: A protein language model based predictor for antihypertensive peptide screening.

Angiotensin-I converting enzyme (ACE) regulates the renin-angiotensin system and is a drug target in clinical treatment for hypertension. This study aims to develop a protein language model (pLM) with evolutionary scale modeling (ESM-2) embeddings that is trained on experimental data to screen peptides with strong ACE inhibitory activity. Twelve conventional peptide embedding approaches and five machine learning (ML) modeling methods were also tested for performance comparison. Among the 65 classifiers tested, logistic regression with ESM-2 embeddings showed the best performance, with balanced accuracy (BACC), Matthews correlation coefficient (MCC), and area under the curve of 0.883 ± 0.017, 0.77 ± 0.032, and 0.96 ± 0.009, respectively. Multilayer perceptron and support vector machine also exhibited great compatibility with ESM-2 embeddings. The ESM-2 embeddings showed superior performance in enhancing the prediction model compared to the 12 traditional embedding methods. A user-friendly webserver (https://sqzujiduce.us-east-1.awsapprunner.com) with the top three models is now freely available.

Contra assertions, feedback improves word recognition: How feedback and lateral inhibition sharpen signals over noise.

Whether top-down feedback modulates perception has deep implications for cognitive theories. Debate has been vigorous in the domain of spoken word recognition, where competing computational models and agreement on at least one diagnostic experimental paradigm suggest that the debate may eventually be resolvable. Norris and Cutler (2021) revisit arguments against lexical feedback in spoken word recognition models. They also incorrectly claim that recent computational demonstrations that feedback promotes accuracy and speed under noise (Magnuson et al., 2018) were due to the use of the Luce choice rule rather than adding noise to inputs (noise was in fact added directly to inputs). They also claim that feedback cannot improve word recognition because feedback cannot distinguish signal from noise. We have two goals in this paper. First, we correct the record about the simulations of Magnuson et al. (2018). Second, we explain how interactive activation models selectively sharpen signals via joint effects of feedback and lateral inhibition that boost lexically-coherent sublexical patterns over noise. We also review a growing body of behavioral and neural results consistent with feedback and inconsistent with autonomous (non-feedback) architectures, and conclude that parsimony supports feedback. We close by discussing the potential for synergy between autonomous and interactive approaches.

Multimodal fine-tuning of clinical language models for predicting COVID-19 outcomes.

Clinical prediction models tend only to incorporate structured healthcare data, ignoring information recorded in other data modalities, including free-text clinical notes. Here, we demonstrate how multimodal models that effectively leverage both structured and unstructured data can be developed for predicting COVID-19 outcomes. The models are trained end-to-end using a technique we refer to as multimodal fine-tuning, whereby a pre-trained language model is updated based on both structured and unstructured data. The multimodal models are trained and evaluated using a multicenter cohort of COVID-19 patients encompassing all encounters at the emergency department of six hospitals. Experimental results show that multimodal models, leveraging the notion of multimodal fine-tuning and trained to predict (i) 30-day mortality, (ii) safe discharge and (iii) readmission, outperform unimodal models trained using only structured or unstructured healthcare data on all three outcomes. Sensitivity analyses are performed to better understand how well the multimodal models perform on different patient groups, while an ablation study is conducted to investigate the impact of different types of clinical notes on model performance. We argue that multimodal models that make effective use of routinely collected healthcare data to predict COVID-19 outcomes may facilitate patient management and contribute to the effective use of limited healthcare resources.