Recent Advances in Transparent Electrodes and Their Multimodal Sensing Applications.

This review examines the recent advancements in transparent electrodes and their crucial role in multimodal sensing technologies. Transparent electrodes, notable for their optical transparency and electrical conductivity, are revolutionizing sensors by enabling the simultaneous detection of diverse physical, chemical, and biological signals. Materials like graphene, carbon nanotubes, and conductive polymers, which offer a balance between optical transparency, electrical conductivity, and mechanical flexibility, are at the forefront of this development. These electrodes are integral in various applications, from healthcare to solar cell technologies, enhancing sensor performance in complex environments. The paper addresses challenges in applying these electrodes, such as the need for mechanical flexibility, high optoelectronic performance, and biocompatibility. It explores new materials and innovative techniques to overcome these hurdles, aiming to broaden the capabilities of multimodal sensing devices. The review provides a comparative analysis of different transparent electrode materials, discussing their applications and the ongoing development of novel electrode systems for multimodal sensing. This exploration offers insights into future advancements in transparent electrodes, highlighting their transformative potential in bioelectronics and multimodal sensing technologies.

Engineering robust and transparent dual-crosslinked hydrogels for multimodal sensing without conductive additives.

Conductive hydrogels are pivotal for the advancement of flexible sensors, electronic skin, and healthcare monitoring systems, facilitating transformative innovations. However, issues such as inadequate intrinsic compatibility, mismatched mechanical properties, and limited stability curtail their potential, resulting in compromised device efficacy and performance degradation. In this research, we engineered functional hydrogels featuring a dual-crosslinked network composed of (PA/PVA)-P(AM-AA) to address these challenges. This design eliminates the need for conductive additives, thereby enhancing intrinsic compatibility. Notably, the hydrogels exhibit exceptional mechanical properties, with high tensile strength (∼700 %), Young's modulus (∼5.33 MPa), increased strength (∼2.46 MPa) and toughness (∼6.59 MJ m-3). They also achieve a compressive strength of âˆ¼7.33 MPa at 80 % maximal compressive strain and maintain about 89 % transparency. Moreover, flexible sensors derived from these hydrogels demonstrate enhanced multimodal sensing capabilities, including temperature, strain, and pressure detection, enabling precise monitoring of human movements. The integration of multiple hydrogels into a three-dimensional sensor array facilitates detailed spatial pressure distribution mapping. By strategically applying dual-crosslinked network engineering and eliminating conductive additives, we have streamlined the design and manufacturing of hydrogels to meet the rising demand for high-performance wearable sensors.

High-Density Dual-Structure Single-Atom Pt Electrocatalyst for Efficient Hydrogen Evolution and Multimodal Sensing.

Herein, we report a high-density dual-structure single-atom catalyst (SAC) by creating a large number of vacancies of O and Ti in two-dimensional (2D) Ti3C2 to immobilize Pt atoms (SA Pt-Ti3C2). The SA Pt-Ti3C2 showed excellent performance toward the pH-universal electrochemical hydrogen evolution reaction (HER) and multimodal sensing. For HER catalysis, compared to the commercial 20 wt % Pt/C, the Pt mass activities of SA Pt-Ti3C2 at the overpotentials of ∼30 and 110 mV in acid and alkaline media are 45 and 34 times higher, respectively. More importantly, during the alkaline HER process, an interesting synergetic effect between Pt-C and Pt-Ti sites that dominated the Volmer and Heyrovsky steps, respectively, was revealed. Moreover, the SA Pt-Ti3C2 catalyst exhibited high sensitivity (0.62-2.65 µA µM-1) and fast response properties for the multimodal identifications of ascorbic acid, dopamine, uric acid, and nitric oxide under the assistance of machine learning.

Smartphone Imaging Device for Multimodal Detection of Hydrogen Sulfide Using Cu-Doped MOF Sensors.

Multimodal sensing platforms may offer reliable, fast results, but it is still challenging to incorporate biosensors with high discriminating ability in complex biological samples. Herein, we established a highly sensitive dual colorimetric/electrochemical monitoring approach for the detection of hydrogen sulfide (H2S) utilizing Cu-doped In-based metal-organic frameworks (Cu/In-MOFs) combined with a versatile color selector software-based smartphone imaging device. H2S can result in the enhancement of the electrochemical signal because of the electroactive substance copper sulfide (CuxS), the decrease of the colorimetric signal of the characteristic absorption response caused by the strong coordination effect on Cu/In-MOFs, and the obvious changes of red-green-blue (RGB) values of images acquired via an intelligent smartphone. Attractively, the Cu/In-MOFs-based multimodal detection guarantees precise and sensitive detection of H2S with triple-signal detection limits of 0.096 µM (electrochemical signals), 0.098 µM (colorimetric signals), and 0.099 µM (smartphone signals) and an outstanding linear response. This analytical toolkit provides an idea for fabricating a robust, sensitive, tolerant matrix and reliable sensing platform for rapidly monitoring H2S in clinical disease diagnosis and visual supervision.

Ultrasensitive Biomimetic Skin with Multimodal and Photoelectric Dual-Signal Sensing.

Mimicking biological skin enabling direct, intelligent interaction between users and devices, multimodal sensing with optical/electrical (OE) output signals is urgently required. Owing to this, this work aims to logically design a stretchable OE biomimetic skin (OE skin), which can sensitively sense complex external stimuli of pressure, strain, temperature, and localization. The OE skin consists of elastic thin polymer-stabilized cholesteric liquid crystal films, an ion-conductive hydrogel layer, and an elastic protective membrane formed with thin polydimethylsiloxane. The as-designed OE skin exhibits customizable structural color on demand, good thermochromism, and excellent mechanochromism, with the ability to extend the full visible spectrum, a good linearity of over 0.99, fast response speed of 93 ms, and wide temperature range of 119 °C. In addition, the conduction resistance variation of ion-conductive hydrogel exhibits excellent sensing capabilities under pressure, stretch, and temperature, endowing a good linearity of 0.99998 (stretching from 0 to 150%) and high thermal sensitivity of 0.86% per °C. Such an outstanding OE skin provides design concepts for the development of multifunctional biomimetic skin used in human-machine interaction and can find wide applications in intelligent wearable devices and human-machine interactions.

Evaluating Users' Experiences of a Child Multimodal Wearable Device: Mixed Methods Approach.

BACKGROUND: Wearable devices permit the continuous, unobtrusive collection of data from children in their natural environments and can transform our understanding of child development. Although the use of wearable devices has begun to emerge in research involving children, few studies have considered families' experiences and perspectives of participating in research of this kind. OBJECTIVE: Through a mixed methods approach, we assessed parents' and children's experiences of using a new wearable device in the home environment. The wearable device was designed specifically for use with infants and young children, and it integrates audio, electrocardiogram, and motion sensors. METHODS: In study 1, semistructured phone interviews were conducted with 42 parents of children aged 1 month to 9.5 years who completed 2 day-long recordings using the device, which the children wore on a specially designed shirt. In study 2, a total of 110 parents of children aged 2 months to 5.5 years responded to a questionnaire assessing their experience of completing 3 day-long device recordings in the home. Guided by the Digital Health Checklist, we assessed parental responses from both studies in relation to the following three key domains: (1) access and usability, (2) privacy, and (3) risks and benefits. RESULTS: In study 1, most parents viewed the device as easy to use and safe and remote visits as convenient. Parents' views on privacy related to the audio recordings were more varied. The use of machine learning algorithms (vs human annotators) in the analysis of the audio data, the ability to stop recordings at any time, and the view that the recordings reflected ordinary family life were some reasons cited by parents who expressed minimal, if any, privacy concerns. Varied risks and benefits were also reported, including perceived child comfort or discomfort, the need to adjust routines to accommodate the study, the understanding gained from the study procedures, and the parent's and child's enjoyment of study participation. In study 2, parents' ratings on 5 close-ended items yielded a similar pattern of findings. Compared with a "neutral" rating, parents were significantly more likely to agree that (1) device instructions were helpful and clear (t109=-45.98; P<.001), (2) they felt comfortable putting the device on their child (t109=-22.22; P<.001), and (3) they felt their child was safe while wearing the device (t109=-34.48; P<.001). They were also less likely to worry about the audio recordings gathered by the device (t108=6.14; P<.001), whereas parents' rating of the burden of the study procedures did not differ significantly from a "neutral" rating (t109=-0.16; P=.87). CONCLUSIONS: On the basis of parents' feedback, several concrete changes can be implemented to improve this new wearable platform and, ultimately, parents' and children's experiences of using child wearable devices in the home setting.

Preliminary Technical Validation of LittleBeats™: A Multimodal Sensing Platform to Capture Cardiac Physiology, Motion, and Vocalizations.

Across five studies, we present the preliminary technical validation of an infant-wearable platform, LittleBeats™, that integrates electrocardiogram (ECG), inertial measurement unit (IMU), and audio sensors. Each sensor modality is validated against data from gold-standard equipment using established algorithms and laboratory tasks. Interbeat interval (IBI) data obtained from the LittleBeats™ ECG sensor indicate acceptable mean absolute percent error rates for both adults (Study 1, N = 16) and infants (Study 2, N = 5) across low- and high-challenge sessions and expected patterns of change in respiratory sinus arrythmia (RSA). For automated activity recognition (upright vs. walk vs. glide vs. squat) using accelerometer data from the LittleBeats™ IMU (Study 3, N = 12 adults), performance was good to excellent, with smartphone (industry standard) data outperforming LittleBeats™ by less than 4 percentage points. Speech emotion recognition (Study 4, N = 8 adults) applied to LittleBeats™ versus smartphone audio data indicated a comparable performance, with no significant difference in error rates. On an automatic speech recognition task (Study 5, N = 12 adults), the best performing algorithm yielded relatively low word error rates, although LittleBeats™ (4.16%) versus smartphone (2.73%) error rates were somewhat higher. Together, these validation studies indicate that LittleBeats™ sensors yield a data quality that is largely comparable to those obtained from gold-standard devices and established protocols used in prior research.

Nonprehensile Manipulation for Rapid Object Spinning via Multisensory Learning from Demonstration.

Dexterous manipulation concerns the control of a robot hand to manipulate an object in a desired manner. While classical dexterous manipulation strategies are based on stable grasping (or force closure), many human-like manipulation tasks do not maintain grasp stability and often utilize the dynamics of the object rather than the closed form of kinematic relation between the object and the robotic hand. Such manipulation strategies are referred as nonprehensile or dynamic dexterous manipulation in the literature. Nonprehensile manipulation often involves fast and agile movements such as throwing and flipping. Due to the complexity of such motions and uncertainties associated with them, it has been challenging to realize nonprehensile manipulation tasks in a reliable way. In this paper, we propose a new control strategy to realize practical nonprehensile manipulation. First, we make explicit use of multiple modalities of sensory data for the design of control law. Specifically, force data are employed for feedforward control, while position data are used for feedback control. Secondly, control signals (both feedback and feedforward) are obtained through multisensory learning from demonstration (LfD) experiments designed and performed for specific nonprehensile manipulation tasks of concern. To prove the concept of the proposed control strategy, experimental tests were conducted for a dynamic spinning task using a sensory-rich, two-finger robotic hand. The control performance (i.e., the speed and accuracy of the spinning task) was also compared with that of classical dexterous manipulation based on force closure and finger gaiting.

Label-free multimodal analysis of copper ions at below permissible exposure limit in the aqueous medium.

An anthraimidazoledione based amphiphilic dye molecule was synthesized that shows formation of tuneable charge-transfer state in solution, susceptible to change in pH, polarity and hydrogen bonding ability of the medium. The compound also showed formation of nanoscopic self-assembled structure in water medium. The probe molecule can achieve multimodal detection (colorimetric, fluorimetric and electrochemical) of copper ions as low as 0.3 ppm in the aqueous medium. Addition of copper leads to dose-dependent ratiometric change in solution color from yellow to purple. The mechanistic investigation indicates that the coordination of copper ions was possible via simultaneous engagement of both imidazole nitrogen ends and neighbouring hydroxyl unit. Not only optical property, the changes in microenvironment also influence the selectivity as well as sensitivity of the probe molecule towards Cu2+ ions. Further, the optical probe is used for detection as well as quantification of copper ions in natural water samples without any sample pretreatment. Low-cost, reusable paper strips are developed for rapid, on-location detection of residual Cu2+ in real-life samples.

Microphase-Separated Elastic and Ultrastretchable Ionogel for Reliable Ionic Skin with Multimodal Sensation.

Bioinspired artificial skins integrated with reliable human-machine interfaces and stretchable electronic systems have attracted considerable attention. However, the current design faces difficulties in simultaneously achieving satisfactory skin-like mechanical compliance and self-powered multimodal sensing. Here, this work reports a microphase-separated bicontinuous ionogel which possesses skin-like mechanical properties and mimics the multimodal sensing ability of biological skin by ion-driven stimuli-electricity conversion. The ionogel exhibits excellent elasticity and ionic conductivity, high toughness, and ultrastretchability, as well as a Young's modulus similar to that of human skin. Leveraging the ion-polymer interactions enabled selective ion transport, the ionogel can output pulsing or continuous electrical signals in response to diverse stimuli such as strain, touch pressure, and temperature sensitively, demonstrating a unique self-powered multimodal sensing. Furthermore, the ionogel-based I-skin can concurrently sense different stimuli and decouple the variations of the stimuli from the voltage signals with the assistance of a machine-learning model. The ease of fabrication, wide tunability, self-powered multimodal sensing, and the excellent environmental tolerance of the ionogels demonstrate a new strategy in the development of next-generation soft smart mechano-transduction devices.

Gravisensation and modulation of gravitactic responses by other sensory cues in the monarch butterfly (Danaus plexippus).

Using the monarch butterfly (Danaus plexippus), we studied how animals can use cues from multiple sensory modalities for deriving directional information from their environment to display oriented movement. Our work focused on determining how monarchs use gravity as a cue for oriented movement and determined how cues from other sensory modalities, cues that by themselves also produce oriented movement (visual and magnetic directional cues), might modulate gravisensation. In two tests of gravisensation (movement in a vertical tube; righting behavior), we found that monarchs display negative gravitaxis only (movement opposite to the direction of gravity). Negative gravitaxis can be modulated by either visual (light) or magnetic field cues (inclination angle) that provide directional information. The modulation of gravity-mediated responses, however, depends on the relationship between cues when presented during trials, such as when cues are in accord or in conflict. For example, when light cues that elicit positive phototaxis conflicted with negative gravitaxis (light from below the monarch), monarch gravisensation was unaffected by directional light cues. We also found that the antennae play a role in gravity-mediated movement (righting), as, with antennae removed, monarch movement behavior was no longer the same as when the antennae were intact. Our results demonstrate that monarchs can use and integrate multiple, multimodal cues for oriented movement, but that the use of such cues can be hierarchical (that is, one cue dominant for movement), and the hierarchy of cues, and the responses towards them when found together, depends on the physical relationships between cues during movement.

Remote sensing traffic scene retrieval based on learning control algorithm for robot multimodal sensing information fusion and human-machine interaction and collaboration.

In light of advancing socio-economic development and urban infrastructure, urban traffic congestion and accidents have become pressing issues. High-resolution remote sensing images are crucial for supporting urban geographic information systems (GIS), road planning, and vehicle navigation. Additionally, the emergence of robotics presents new possibilities for traffic management and road safety. This study introduces an innovative approach that combines attention mechanisms and robotic multimodal information fusion for retrieving traffic scenes from remote sensing images. Attention mechanisms focus on specific road and traffic features, reducing computation and enhancing detail capture. Graph neural algorithms improve scene retrieval accuracy. To achieve efficient traffic scene retrieval, a robot equipped with advanced sensing technology autonomously navigates urban environments, capturing high-accuracy, wide-coverage images. This facilitates comprehensive traffic databases and real-time traffic information retrieval for precise traffic management. Extensive experiments on large-scale remote sensing datasets demonstrate the feasibility and effectiveness of this approach. The integration of attention mechanisms, graph neural algorithms, and robotic multimodal information fusion enhances traffic scene retrieval, promising improved information extraction accuracy for more effective traffic management, road safety, and intelligent transportation systems. In conclusion, this interdisciplinary approach, combining attention mechanisms, graph neural algorithms, and robotic technology, represents significant progress in traffic scene retrieval from remote sensing images, with potential applications in traffic management, road safety, and urban planning.

Transcutaneous auricular Vagus Nerve Stimulation and Median Nerve Stimulation reduce acute stress in young healthy adults: a single-blind sham-controlled crossover study.

Stress is a major determinant of health and wellbeing. Conventional stress management approaches do not account for the daily-living acute changes in stress that affect quality of life. The combination of physiological monitoring and non-invasive Peripheral Nerve Stimulation (PNS) represents a promising technological approach to quantify stress-induced physiological manifestations and reduce stress during everyday life. This study aimed to evaluate the effectiveness of three well-established transcutaneous PNS modalities in reducing physiological manifestations of stress compared to a sham: auricular and cervical Vagus Nerve Stimulation (taVNS and tcVNS), and Median Nerve Stimulation (tMNS). Using a single-blind sham-controlled crossover study with four visits, we compared the stress mitigation effectiveness of taVNS, tcVNS, and tMNS, quantified through physiological markers derived from five physiological signals peripherally measured on 19 young healthy volunteers. Participants underwent three acute mental and physiological stressors while receiving stimulation. Blinding effectiveness was assessed via subjective survey. taVNS and tMNS relative to sham resulted in significant changes that suggest a reduction in sympathetic outflow following the acute stressors: Left Ventricular Ejection Time Index (LVETI) shortening (tMNS: p = 0.007, taVNS: p = 0.015) and Pre-Ejection Period (PEP)-to-LVET ratio (PEP/LVET) increase (tMNS: p = 0.044, taVNS: p = 0.029). tMNS relative to sham also reduced Pulse Pressure (PP; p = 0.032) and tonic EDA activity (tonicMean; p = 0.025). The nonsignificant blinding survey results suggest these effects were not influenced by placebo. taVNS and tMNS effectively reduced stress-induced sympathetic arousal in wearable-compatible physiological signals, motivating their future use in novel personalized stress therapies to improve quality of life.

Wearable-based human flow experience recognition enhanced by transfer learning methods using emotion data.

BACKGROUND: Flow experience is a specific positive and affective state that occurs when humans are completely absorbed in an activity and forget everything else. This state can lead to high performance, well-being, and productivity at work. Few studies have been conducted to determine the human flow experience using physiological wearable sensor devices. Other studies rely on self-reported data. METHODS: In this article, we use physiological data collected from 25 subjects with multimodal sensing devices, in particular the Empatica E4 wristband, the Emotiv Epoc X electroencephalography (EEG) headset, and the Biosignalplux RespiBAN - in arithmetic and reading tasks to automatically discriminate between flow and non-flow states using feature engineering and deep feature learning approaches. The most meaningful wearable device for flow detection is determined by comparing the performances of each device. We also investigate the connection between emotions and flow by testing transfer learning techniques involving an emotion recognition-related task on the source domain. RESULTS: The EEG sensor modalities yielded the best performances with an accuracy of 64.97%, and a macro Averaged F1 (AF1) score of 64.95%. An accuracy of 73.63% and an AF1 score of 72.70% were obtained after fusing all sensor modalities from all devices. Additionally, our proposed transfer learning approach using emotional arousal classification on the DEAP dataset led to an increase in performances with an accuracy of 75.10% and an AF1 score of 74.92%. CONCLUSION: The results of this study suggest that effective discrimination between flow and non-flow states is possible with multimodal sensor data. The success of transfer learning using the DEAP emotion dataset as a source domain indicates that emotions and flow are connected, and emotion recognition can be used as a latent task to enhance the performance of flow recognition.

Sensing in Soft Robotics.

Soft robotics is an exciting field of science and technology that enables robots to manipulate objects with human-like dexterity. Soft robots can handle delicate objects with care, access remote areas, and offer realistic feedback on their handling performance. However, increased dexterity and mechanical compliance of soft robots come with the need for accurate control of the position and shape of these robots. Therefore, soft robots must be equipped with sensors for better perception of their surroundings, location, force, temperature, shape, and other stimuli for effective usage. This review highlights recent progress in sensing feedback technologies for soft robotic applications. It begins with an introduction to actuation technologies and material selection in soft robotics, followed by an in-depth exploration of various types of sensors, their integration methods, and the benefits of multimodal sensing, signal processing, and control strategies. A short description of current market leaders in soft robotics is also included in the review to illustrate the growing demands of this technology. By examining the latest advancements in sensing feedback technologies for soft robots, this review aims to highlight the potential of soft robotics and inspire innovation in the field.

Flexible Optoelectronic Multimodal Proximity/Pressure/Temperature Sensors with Low Signal Interference.

Multimodal tactile sensors are a crucial part of intelligent human-machine interaction and collaboration. Simultaneous detection of proximity, pressure, and temperature on a single sensor can greatly promote the safety, interactivity, and compactness of interaction systems. However, severe signal interference and complex decoupling algorithms hinder the actual applications. Here, this work reports a flexible optoelectronic multimodal sensor capable of detecting and decoupling proximity/pressure/temperature by integrating a light waveguide and an interdigital electrode (IDE) into a compact fibrous sensor. Negligible signal interference is realized by combining heterogeneous sensing mechanisms of optics and electronics, which encodes proximity into capacitance, pressure into light intensity and temperature into resistance. The sensor exhibits a large sensing distance of 225 mm with fast responses for proximity detection, a pressure sensitivity of 0.42 N-1 , and a temperature sensitivity of 7% °C-1 . As a proof of concept, a doll equipped with the sensor can accurately discriminate and detect various stimuli, thus achieving safe and immersive interactions with the user. This work opens up promising paths for self-decoupled multimodal sensors and related human/machine/environment interaction applications.

Corrigendum: SemNet: Learning semantic attributes for human activity recognition with deep belief networks.

[This corrects the article DOI: 10.3389/fdata.2022.879389.].

Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild.

This paper investigates multimodal sensor architectures with deep learning for audio-visual speech recognition, focusing on in-the-wild scenarios. The term "in the wild" is used to describe AVSR for unconstrained natural-language audio streams and video-stream modalities. Audio-visual speech recognition (AVSR) is a speech-recognition task that leverages both an audio input of a human voice and an aligned visual input of lip motions. However, since in-the-wild scenarios can include more noise, AVSR's performance is affected. Here, we propose new improvements for AVSR models by incorporating data-augmentation techniques to generate more data samples for building the classification models. For the data-augmentation techniques, we utilized a combination of conventional approaches (e.g., flips and rotations), as well as newer approaches, such as generative adversarial networks (GANs). To validate the approaches, we used augmented data from well-known datasets (LRS2-Lip Reading Sentences 2 and LRS3) in the training process and testing was performed using the original data. The study and experimental results indicated that the proposed AVSR model and framework, combined with the augmentation approach, enhanced the performance of the AVSR framework in the wild for noisy datasets. Furthermore, in this study, we discuss the domains of automatic speech recognition (ASR) architectures and audio-visual speech recognition (AVSR) architectures and give a concise summary of the AVSR models that have been proposed.

MAV Localization in Large-Scale Environments: A Decoupled Optimization/Filtering Approach.

Developing new sensor fusion algorithms has become indispensable to tackle the daunting problem of GPS-aided micro aerial vehicle (MAV) localization in large-scale landscapes. Sensor fusion should guarantee high-accuracy estimation with the least amount of system delay. Towards this goal, we propose a linear optimal state estimation approach for the MAV to avoid complicated and high-latency calculations and an immediate metric-scale recovery paradigm that uses low-rate noisy GPS measurements when available. Our proposed strategy shows how the vision sensor can quickly bootstrap a pose that has been arbitrarily scaled and recovered from various drifts that affect vision-based algorithms. We can consider the camera as a "black-box" pose estimator thanks to our proposed optimization/filtering-based methodology. This maintains the sensor fusion algorithm's computational complexity and makes it suitable for MAV's long-term operations in expansive areas. Due to the limited global tracking and localization data from the GPS sensors, our proposal on MAV's localization solution considers the sensor measurement uncertainty constraints under such circumstances. Extensive quantitative and qualitative analyses utilizing real-world and large-scale MAV sequences demonstrate the higher performance of our technique in comparison to most recent state-of-the-art algorithms in terms of trajectory estimation accuracy and system latency.

Digital Phenotyping of Mental Health using multimodal sensing of multiple situations of interest: A Systematic Literature Review.

Many studies have used Digital Phenotyping of Mental Health (DPMH) to complement classic methods of mental health assessment and monitoring. This research area proposes innovative methods that perform multimodal sensing of multiple situations of interest (e.g., sleep, physical activity, mobility) to health professionals. In this paper, we present a Systematic Literature Review (SLR) to recognize, characterize and analyze the state of the art on DPMH using multimodal sensing of multiple situations of interest to professionals. We searched for studies in six digital libraries, which resulted in 1865 retrieved published papers. Next, we performed a systematic process of selecting studies based on inclusion and exclusion criteria, which selected 59 studies for the data extraction phase. First, based on the analysis of the extracted data, we describe an overview of this field, then presenting characteristics of the selected studies, the main mental health topics targeted, the physical and virtual sensors used, and the identified situations of interest. Next, we outline answers to research questions, describing the context data sources used to detect situations, the DPMH workflow used for multimodal sensing of situations, and the application of DPMH solutions in the mental health assessment and monitoring process. In addition, we recognize trends presented by DPMH studies, such as the design of solutions for high-level information recognition, association of features with mental states/disorders, classification of mental states/disorders, and prediction of mental states/disorders. We also recognize the main open issues in this research area. Based on the results of this SLR, we conclude that despite the potential and continuous evolution for using these solutions as medical decision support tools, this research area needs more work to overcome technology and methodological rigor issues to adopt proposed solutions in real clinical settings.