Virtual Experience Toolkit: An End-to-End Automated 3D Scene Virtualization Framework Implementing Computer Vision Techniques.

Virtualization plays a critical role in enriching the user experience in Virtual Reality (VR) by offering heightened realism, increased immersion, safer navigation, and newly achievable levels of interaction and personalization, specifically in indoor environments. Traditionally, the creation of virtual content has fallen under one of two broad categories: manual methods crafted by graphic designers, which are labor-intensive and sometimes lack precision; traditional Computer Vision (CV) and Deep Learning (DL) frameworks that frequently result in semi-automatic and complex solutions, lacking a unified framework for both 3D reconstruction and scene understanding, often missing a fully interactive representation of the objects and neglecting their appearance. To address these diverse challenges and limitations, we introduce the Virtual Experience Toolkit (VET), an automated and user-friendly framework that utilizes DL and advanced CV techniques to efficiently and accurately virtualize real-world indoor scenarios. The key features of VET are the use of ScanNotate, a retrieval and alignment tool that enhances the precision and efficiency of its precursor, supported by upgrades such as a preprocessing step to make it fully automatic and a preselection of a reduced list of CAD to speed up the process, and the implementation in a user-friendly and fully automatic Unity3D application that guides the users through the whole pipeline and concludes in a fully interactive and customizable 3D scene. The efficacy of VET is demonstrated using a diversified dataset of virtualized 3D indoor scenarios, supplementing the ScanNet dataset.

Biosignal comparison for autism assessment using machine learning models and virtual reality.

Clinical assessment procedures encounter challenges in terms of objectivity because they rely on subjective data. Computational psychiatry proposes overcoming this limitation by introducing biosignal-based assessments able to detect clinical biomarkers, while virtual reality (VR) can offer ecological settings for measurement. Autism spectrum disorder (ASD) is a neurodevelopmental disorder where many biosignals have been tested to improve assessment procedures. However, in ASD research there is a lack of studies systematically comparing biosignals for the automatic classification of ASD when recorded simultaneously in ecological settings, and comparisons among previous studies are challenging due to methodological inconsistencies. In this study, we examined a VR screening tool consisting of four virtual scenes, and we compared machine learning models based on implicit (motor skills and eye movements) and explicit (behavioral responses) biosignals. Machine learning models were developed for each biosignal within the virtual scenes and then combined into a final model per biosignal. A linear support vector classifier with recursive feature elimination was used and tested using nested cross-validation. The final model based on motor skills exhibited the highest robustness in identifying ASD, achieving an AUC of 0.89 (SD = 0.08). The best behavioral model showed an AUC of 0.80, while further research is needed for the eye-movement models due to limitations with the eye-tracking glasses. These findings highlight the potential of motor skills in enhancing objectivity and reliability in the early assessment of ASD compared to other biosignals.

[Virtual reality and digital biomarkers: a clinical tool for early autism diagnosis]. / Realidad virtual y biomarcadores digitales: una herramienta clínica para el diagnóstico del autismo.

INTRODUCTION: Autism Spectrum Disorder (ASD) is a neurodevelopmental condition which traditional assessment procedures encounter certain limitations. The current ASD research field is exploring and endorsing innovative methods to assess the disorder early on, based on the automatic detection of biomarkers. However, many of these procedures lack ecological validity in their measurements. In this context, virtual reality (VR) shows promise for objectively recording biosignals while users experience ecological situations. METHODS: This study outlines a novel and playful VR procedure for the early assessment of ASD, relying on multimodal biosignal recording. During a VR experience featuring 12 virtual scenes, eye gaze, motor skills, electrodermal activity and behavioural performance were measured in 39 children with ASD and 42 control peers. Machine learning models were developed to identify digital biomarkers and classify autism. RESULTS: Biosignals reported varied performance in detecting ASD, while the combined model resulting from the combination of specific-biosignal models demonstrated the ability to identify ASD with an accuracy of 83% (SD = 3%) and an AUC of 0.91 (SD = 0.04). DISCUSSION: This screening tool may support ASD diagnosis by reinforcing the outcomes of traditional assessment procedures.

Introducción: El Trastorno del Espectro Autista (TEA) es un trastorno del neurodesarrollo, y sus procedimientos tradicionales de evaluación encuentran ciertas limitaciones. El actual campo de investigación sobre TEA está explorando y respaldando métodos innovadores para evaluar el trastorno tempranamente, basándose en la detección automática de biomarcadores. Sin embargo, muchos de estos procedimientos carecen de validez ecológica en sus mediciones. En este contexto, la realidad virtual (RV) presenta un prometedor potencial para registrar objetivamente bioseñales mientras los usuarios experimentan situaciones ecológicas. Métodos: Este estudio describe un novedoso y lúdico procedimiento de RV para la evaluación temprana del TEA, basado en la grabación multimodal de bioseñales. Durante una experiencia de RV con 12 escenas virtuales, se midieron la mirada, las habilidades motoras, la actividad electrodermal y el rendimiento conductual en 39 niños con TEA y 42 compañeros de control. Se desarrollaron modelos de aprendizaje automático para identificar biomarcadores digitales y clasificar el autismo. Resultados: Las bioseñales reportaron un rendimiento variado en la detección del TEA, mientras que el modelo resultante de la combinación de los modelos de las bioseñales demostró la capacidad de identificar el TEA con una precisión del 83% (DE = 3%) y un AUC de 0.91 (DE = 0.04). Discusión: Esta herramienta de detección puede respaldar el diagnóstico del TEA al reforzar los resultados de los procedimientos tradicionales de evaluación.

Beyond Reality: Exploring the effect of different virtual reality environments on visual assessment of cakes.

Visual assessment triggers physiological, emotional, and cognitive responses in consumer behavior. This confluence of signals can be influenced by context, which plays a crucial role in eating behavior. The strategies used to evoke scenarios that enhance ecological validity in sensory experiences have evolved in the last years to include immersive technologies and virtual reality (VR) to simulate the complexity of the real world and predict consumer preferences. This study explored VR's effect on visual liking and hedonic responses of five virtual cakes in two virtual contexts designed with advanced 3D modeling and photogrammetry techniques to ensure high realism and immersion. Although the virtual contexts themselves did not impact liking ratings, the variables "context-cake," "age," and "subjective hunger" had a significant effect on the visual liking of cakes. A Check-All-That-Apply (CATA) questionnaire showed significant differences in responses for various terms related to the intrinsic and extrinsic characteristics of the five cakes. Finally, the internal preference map separated two consumer patterns of visual liking: traditional versus innovative.

Social Signal Processing in Affective Virtual Reality: Human-Shaped Agents Increase Electrodermal Activity in an Elicited Negative Environment.

Prior research on affect elicitation indicates that stimuli with social content (pictures or videos) are more arousing than nonsocial stimuli. In particular, they elicit stronger physiological arousal as measured by electrodermal activity (EDA; i.e., social EDA effect). However, it is unclear how this effect applies to virtual reality (VR), which enables an enhanced sense of presence (SoP) and ecological validity. The study here approached this question from a social-emotional VR framework. A sample of N = 72 participants (55 percent women) experienced a set of six virtual environments (VEs) in the form of emotional parks specifically designed to elicit positive, negative, or neutral affectivity. Half of these VEs included human-shaped agents (social context) and the other half omitted these agents (nonsocial context). The results supported the social EDA effect, which in addition was amplified by the reported SoP. Importantly, the VE featuring a social negative content qualified this observed social EDA effect. The finding is discussed in the light of a negativity bias reported in affect literature, through which negative stimuli typically mobilize attention and bodily activation as a mechanism linked to stress responses. The study's implications extend to the use of VR in both research and practical applications, emphasizing the role of social content in influencing affective and physiological responses.

Realidad virtual y biomarcadores digitales: una herramienta clínica para el diagnóstico del autismo / Virtual reality and digital biomarkers: a clinical tool for early autism diagnosis

Resumen Introducción : El Trastorno del Espectro Autista (TEA) es un trastorno del neurodesarrollo, y sus procedimien tos tradicionales de evaluación encuentran ciertas li mitaciones. El actual campo de investigación sobre TEA está explorando y respaldando métodos innovadores para evaluar el trastorno tempranamente, basándose en la detección automática de biomarcadores. Sin embargo, muchos de estos procedimientos carecen de validez ecológica en sus mediciones. En este contexto, la reali dad virtual (RV) presenta un prometedor potencial para registrar objetivamente bioseñales mientras los usuarios experimentan situaciones ecológicas. Métodos : Este estudio describe un novedoso y lúdi co procedimiento de RV para la evaluación temprana del TEA, basado en la grabación multimodal de bio señales. Durante una experiencia de RV con 12 esce nas virtuales, se midieron la mirada, las habilidades motoras, la actividad electrodermal y el rendimiento conductual en 39 niños con TEA y 42 compañeros de control. Se desarrollaron modelos de aprendizaje automático para identificar biomarcadores digitales y clasificar el autismo. Resultados : Las bioseñales reportaron un rendimien to variado en la detección del TEA, mientras que el modelo resultante de la combinación de los modelos de las bioseñales demostró la capacidad de identificar el TEA con una precisión del 83% (DE = 3%) y un AUC de 0.91 (DE = 0.04). Discusión : Esta herramienta de detección pue de respaldar el diagnóstico del TEA al reforzar los resultados de los procedimientos tradicionales de evaluación.

Abstract Introduction : Autism Spectrum Disorder (ASD) is a neurodevelopmental condition which traditional as sessment procedures encounter certain limitations. The current ASD research field is exploring and endorsing innovative methods to assess the disorder early on, based on the automatic detection of biomarkers. How ever, many of these procedures lack ecological validity in their measurements. In this context, virtual reality (VR) shows promise for objectively recording biosignals while users experience ecological situations. Methods : This study outlines a novel and playful VR procedure for the early assessment of ASD, relying on multimodal biosignal recording. During a VR experience featuring 12 virtual scenes, eye gaze, motor skills, elec trodermal activity and behavioural performance were measured in 39 children with ASD and 42 control peers. Machine learning models were developed to identify digital biomarkers and classify autism. Results : Biosignals reported varied performance in detecting ASD, while the combined model resulting from the combination of specific-biosignal models demon strated the ability to identify ASD with an accuracy of 83% (SD = 3%) and an AUC of 0.91 (SD = 0.04). Discussion : This screening tool may support ASD diagnosis by reinforcing the outcomes of traditional assessment procedures.

Virtual humans for ASD intervention: a brief scoping review / Humanos virtuales para la intervención en TEA: una breve revisión del alcance

Abstract Individuals with autism spectrum disorder may present social-communicative and behavioral deficits. Recently, research on treatment and diagnosis has shifted its focus to the application of new tech nologies. Among them is virtual reality, which guarantees a high sense of realism to the experience and allows the implementation of a virtual agent that facilitates the use of the application. In social skills interventions, it has been mostly chosen to implement a virtual agent with a human appearance. Virtual humans guide the user-system interaction through the use of verbal and nonverbal language. They can be equipped with responsiveness: the ability to provide responses to the user based on data recorded during the use of the technology. Responsiveness is functional when the goal is to create an interaction similar to that of everyday life, as it allows for behavioral responses and, at a more sophisticated level, vocal responses. Considering virtual agents capable of holding a conversation with the user, to date three different methods have been implemented that make communication more or less realistic. This brief review proposes a synopsis of relevant virtual humans' features and highlights some key ASD research areas wherein virtual humans are implemented for diagnosis and treatment. A total of 11 studies were selected and their analysis was summarized into 7 main categories. Finally, the clinical and technological implications of the results found were discussed.

Resumen Los individuos con trastorno del espectro autista pueden presentar déficits socio-comunicativos y conductuales. Recientemente, la investigación sobre el tratamiento y el diagnóstico se ha centrado en la aplicación de nuevas tecnologías. Entre ellas se encuentra la realidad virtual, que garantiza un alto sentido de realismo a la experiencia y permite la implementación de un agente virtual que facilite el uso de la aplicación. En las intervenciones de habilidades sociales, se ha optado mayoritariamente por implementar un agente virtual con apariencia humana. Los humanos virtuales guían la interacción usuario-sistema mediante el uso de lenguaje verbal y no verbal. Estos pueden estar dotados de responsividad: la capacidad de proporcionar respuestas al usuario basadas en los datos registrados durante el uso de la tecnología. La responsividad es funcional cuando el objetivo es crear una interacción similar a la de la vida cotidiana, ya que permite dar respuestas conductuales y, a un nivel más sofisticado, respuestas vocales. Considerando los agentes virtuales capaces de mantener una conversación con el usuario, hasta la fecha se han implementado tres métodos diferentes que hacen que la comunicación sea más o menos realista. Esta breve revisión propone una sinopsis de las características de los humanos virtuales relevantes y destaca algunas áreas de investigación clave del TEA en las que se implemen tan humanos virtuales para el diagnóstico y el tratamiento. Se seleccionó un total de 11 estudios y su análisis se resumió en 7 categorías principales. Por último, se discuten las implicaciones clínicas y tecnológicas de los resultados encontrados.

Intervenciones tecnológicas adaptativas para el trastorno del espectro autista: una revisión bibliográfica / Adaptive technological interventions for autism spectrum disorder: a literature review

Resumen Los individuos con trastornos del espectro autista suelen describirse con deficiencias comunicativas, sociales, emocionales y de comportamiento. A menudo están aislados y son pasivos, con pocas oportunidades de interacción positiva y constructiva con el mundo exterior. Por otra parte, pueden mostrar comportamientos retraídos, estereotipados y disruptivos. Estas condiciones pueden dificultar seriamente sus habilidades adaptativas al ambiente, con consecuencias negativas en su calidad de vida. La heterogeneidad fenotípica y la manifestación, así como la gravedad de los síntomas, pueden variar considerablemente según el niño. Esos determinan la necesidad de intervenciones personalizadas y adaptivas según las necesidades específicas, incluyendo factores como la edad, las habilidades intelectuales y las áreas afectadas específicas. Una intervención temprana promovería las habilidades adaptativas, la autodeterminación y la autonomía hacia el entorno. No obstante, los tiempos de esperas y los costes no permiten una evaluación temprana y como consecuencia los tiempos de intervención se retrasan afectando la cualidad de vida de los niños y de los pa dres. Además, los programas tradicionales de intervención dependen de la experiencia de los terapeutas. Una posible forma de superar este problema es el uso de tecnología adaptativa objetiva según las necesidades. El objetivo de este artículo es proporcionar una visión general de las pruebas empíricas disponible en los últimos siete años. En total, se seleccionaron 8 estudios, con 132 participantes que utilizaron 4 sistemas tecnológicos. Por último, se discuten las implicaciones tecnológicas, clínicas, psicológicas y rehabilitadoras de los hallazgos. Se esbozaron directrices prácticas dentro de esta área temática como perspectivas de investigación futuras.

Abstract Individuals with autistic spectrum disorder are often described as having communication, social, emo tional, and behavioral impairments. They are often isolated and passive, with few opportunities for positive and constructive interaction with the outside world. Moreover, they may exhibit withdrawn, stereotyped and disruptive behaviors. The aforementioned conditions can seriously hamper their ability to adapt to the environment, with negative consequences on their quality of life. Phenotypic heterogeneity and manifestation, as well as symptom severity, can vary greatly from child to child. These determine the need for individualized and adaptive interventions according to specific needs, including factors such as age, intellectual ability, and specific affected areas. Early intervention would promote adaptive skills, self-determination, and autonomy towards the environment. However, wait times and costs do not allow for early assessment, and therefore intervention times are delayed, affecting the quality of life of children and parents. In addition, traditional intervention programs depend on the expertise of the therapists. One possible way to overcome this problem is by using objective adaptive technologies based on needs. This article aims to provide an overview of the empirical evidence available over the past seven years. Overall, 8 studies were selected, with 132 participants using 4 technological systems. Finally, the technological, clinical, psychological, and rehabilitative implications of the findings are discussed. Practical guidelines within this topic area are outlined as future research perspectives.

Biomarcadores del trastorno del especto autista basados en bioseñales, realidad virtual e inteligencia artificial / Autism spectrum disorder biomarkers based on biosignals, virtual reality and artificial intelligence

Se ha observado que la estratificación de trastornos del espectro autista (TEA) generada por las escalas actuales no es efectiva para la personalización de tratamientos tempranos. La evaluación clínica de TEA requiere su consideración como un continuo de déficits, y existe la necesidad de identificar parámetros biológicamente significativos (biomarcadores) que tengan el poder de caracterizar automáticamente a cada individuo en diferentes etapas del desarrollo neurológico. El incipiente campo de la psiquiatría computacional (CP) intenta satisfacer las necesidades de diagnóstico de precisión mediante el desarrollo de potentes técnicas computacionales y matemáticas. Una creciente actividad científica propone el uso de medidas implícitas basadas en bioseñales para la clasificación de ASD. Las tecnologías de realidad virtual (VR) han demostrado potencial para las intervenciones de TEA, pero la mayoría de los trabajos han utilizado la realidad virtual para el aprendizaje / objetivo de las intervenciones. Muy pocos estudios han utilizado señales biológicas para el registro y el análisis detallado de las respuestas conductuales que se pueden utilizar para monitorear o producir cambios a lo largo del tiempo. En el presente trabajo se introduce el concepto de biomarcadores conductuales basados en VR o VRBB. Los VRBB van a permitir la clasificación de TEA utilizando un paradigma de psiquiatría computacional basado en procesos cerebrales implícitos medidos a través de señales psicofisiológicas y el comportamiento de sujetos expuestos a complejas réplicas de condiciones sociales utilizando interfaces de realidad virtual.

It has been observed that the stratification of Autism Spectrum Disorders (ASD) generated by the current scales is not effective for the personalization of early treatments. The clinical evaluation of ASD requires its consideration as a continuum of deficits, and there is a need to identify biologically significant parameters (biomarkers) that have the power to automatically characterize each individual at different stages of neurological development. The emerging field of computational psychiatry (CP) attempts to meet the needs of precision diagnosis by developing powerful computational and mathematical techniques. A growing scientific activity proposes the use of implicit measures based on biosignals for the classification of ASD. Virtual reality (VR) technologies have demonstrated potential for ASD interventions, but most of the work has used virtual reality for the learning / objective of interventions. Very few studies have used biological signals for recording and detailed analysis of behavioral responses that can be used to monitor or produce changes over time. In this paper the concept of behavioral biomarkers based on VR or VRBB is introduced. VRBB will allow the classification of ASD using a paradigm of computational psychiatry based on implicit brain processes measured through psychophysiological signals and the behavior of subjects exposed to complex replicas of social conditions using virtual reality interfaces.

Uso de entornos virtuales para trastornos del neurodesarrollo: una revisión del estado del arte y agenda futura / Use of virtual reality for neurodevelopmental disorders. A review of the state of the art and future agenda

Hasta ahora las herramientas diagnósticas de los trastornos del espectro autista (TEA) se basan mayoritariamente en criterios cualitativos de información observacional en contextos con baja validez ecológica. Una creciente actividad científica propone medidas implícitas para la evaluación y diagnóstico del TEA. Dichas medidas se basan en procesos de carácter biológico e inconsciente, subyacentes a la capacidad de cognición humana y son obtenidas a través de la adquisición y tratamiento de respuestas cerebrales, fisiológicas y comportamentales, con el objetivo de obtener la estructura comportamental del paciente TEA ante un estímulo. La compleja relación existente entre respuestas fisiológicas y la estructura comportamental del paciente TEA ante un estímulo, obliga a utilizar técnicas avanzadas de tratamiento de la señal basadas en computación cognitiva. Las técnicas de inteligencia artificial, tales como aprendizaje automático (machine learning) y neurocomputación aplicadas al análisis de señales psicofisiológicas, han demostra do su robustez para la clasificación de complejos constructos cognitivos. La realidad virtual (RV) es una herramienta que permite recrear situaciones de la vida real con una alta fidelidad sensorial, pero al mismo tiempo controlar individualmente cada una de las situaciones y estímulos que influyen en el comportamiento humano. También permite la medición en tiempo real de las reacciones humanas ante tales estímulos. Este documento analiza los últimos avances científicos y tecnológicos relevantes para sus aplicaciones en el diagnóstico del TEA. Afirmamos que la RV es una herramienta muy valiosa para la investigación del TEA, especialmente para la evaluación y diagnóstico de habilidades y competencias complejas.

To date, the diagnostic tools for autism spectrum disorder (ASD) have been mostly based on qualitative criteria from observational information in contexts with low ecological validity. We are witnessing a growing scientific activity that proposes the use of implicit measures for the evaluation and diagnosis of ASD. These measures are based on processes of a biological and unconscious nature, underlying the capacity of human cognition, and are obtained through the acquisition and treatment of brain, physiological and behavioral responses in order to obtain the behavioral structure of the ASD patient facing a stimulus. The complex relationship between physiological responses and the behavioral structure of the ASD patient requires the use of advanced techniques of signal processing based on cognitive computation. Artificial intelligence (AI) techniques, such as machine learning and neurocomputing applied to the analysis of psychophysiological signals, have demonstrated their robustness for the classification of complex cognitive constructs. Virtual reality (VR) is a tool that allows recreating real-life situations with high sensory fidelity, but at the same time individually controlling each of the situations and stimuli that influence human behavior. It also allows the measurement in real time of human reactions to such stimuli. This document analyzes the latest scientific and technological advances relevant to its applications in the diagnosis of ASD. We conclude that VR is a very valuable tool for ASD research, especially for the evaluation and diagnosis of complex skills and competencies.