Munich Institute of Robotics and Machine Intelligence (MIRMI) Technical University of Munich. / Mensch-Roboter-Interaktion: vernetzte, lernfähige Maschinen in der Medizin.

The application of robotic and intelligent technologies in healthcare is dramatically increasing. The next generation of lightweight and tactile robots have provided a great opportunity to be used for a wide range of applications from medical examination, diagnosis, therapeutic procedures to rehabilitation and assistive robotics. They can potentially outperform current medical procedures by exploiting the com- plementary strengths of humans and computer-based technologies. In this study, the importance of human- robot interaction is discussed and technological re- quirements and challenges in making human-centered robot platforms for medical applications is addressed.

New Method for Surgical Diagnostics - a Robotic Telemedical Approach.

Apart from the tremendous increase in the demand for telemedicine during the COVID-19 pandemic, the use of telemedical technology offers many advantages, such as better coverage of rural areas and improved access to specialists. While current telediagnostic possibilities are often limited to a verbal consultation, the field of surgery has already made use of robotics for one of the most challenging areas of medicine: invasive procedures. Since comprehensive diagnostics are a prerequisite for each surgery, we built upon the knowledge gained in telesurgery and developed a telediagnostic system that allows for an extensive perioperative and emergency examination. It is based on a robotic platform consisting of a remote lead robotic arm at the physician's site and a follower robot at the patient's site. Mirroring all movements directly and using force-feedback, both parties can precisely interact, enabling tasks such as auscultation, percussion, and palpation without the need for extensive training. Our overall setup also includes the possibility to measure and monitor all relevant vital parameters and can be used to perform ear and nasopharyngeal inspections as well as an automatic swab to screen for COVID or other contagious diseases prior to hospital admission. In this paper, we focus on the potential of this technology for the surgical community by demonstrating the ease of adding an ultrasound probe to our modular setup to perform a high-quality emergency ultrasound examination. While the system is not yet ready for everyday use in a hospital and drawbacks such as a high cost persist, our setup paves the way for the future use of telediagnostics in surgery.

Multidigit force control during unconstrained grasping in response to object perturbations.

Because of the complex anatomy of the human hand, in the absence of external constraints, a large number of postures and force combinations can be used to attain a stable grasp. Motor synergies provide a viable strategy to solve this problem of motor redundancy. In this study, we exploited the technical advantages of an innovative sensorized object to study unconstrained hand grasping within the theoretical framework of motor synergies. Participants were required to grasp, lift, and hold the sensorized object. During the holding phase, we repetitively applied external disturbance forces and torques and recorded the spatiotemporal distribution of grip forces produced by each digit. We found that the time to reach the maximum grip force during each perturbation was roughly equal across fingers, consistent with a synchronous, synergistic stiffening across digits. We further evaluated this hypothesis by comparing the force distribution of human grasping vs. robotic grasping, where the control strategy was set by the experimenter. We controlled the global hand stiffness of the robotic hand and found that this control algorithm produced a force pattern qualitatively similar to human grasping performance. Our results suggest that the nervous system uses a default whole hand synergistic control to maintain a stable grasp regardless of the number of digits involved in the task, their position on the objects, and the type and frequency of external perturbations.NEW & NOTEWORTHY We studied hand grasping using a sensorized object allowing unconstrained finger placement. During object perturbation, the time to reach the peak force was roughly equal across fingers, consistently with a synergistic stiffening across fingers. Force distribution of a robotic grasping hand, where the control algorithm is based on global hand stiffness, was qualitatively similar to human grasping. This suggests that the central nervous system uses a default whole hand synergistic control to maintain a stable grasp.

Comparative analysis of motor skill acquisition in a novel bimanual task: the role of mental representation and sensorimotor feedback.

Introduction: This study investigates the multifaceted nature of motor learning in a complex bimanual task by examining the interplay between mental representation structures, biomechanics, tactile pressure, and performance. We developed a novel maze game requiring participants to maneuver a rolling sphere through a maze, exemplifying complex sequential coordination of vision and haptic control using both hands. A key component of this study is the introduction of cognitive primitives, fundamental units of cognitive and motor actions that represent specific movement patterns and strategies. Methods: Participants were divided into two groups based on initial performance: poor performers (PPG) and good performers (GPG). The experimental setup employed motion capture and innovative tactile sensors to capture a detailed multimodal picture of the interaction process. Our primary aims were to (1) assess the effects of daily practice on task performance, biomechanics, and tactile pressure, (2) examine the relationship between changes in mental representation structures and skill performance, and (3) explore the interplay between biomechanics, tactile pressure, and cognitive representation in motor learning. Results: Performance analysis showed that motor skills improved with practice, with the GPG outperforming the PPG in maze navigation efficiency. Biomechanical analysis revealed that the GPG demonstrated superior movement strategies, as indicated by higher peak velocities and fewer velocity peaks during task execution. Tactile feedback analysis showed that GPG participants applied more precise and focused pressure with their right-hand thumb, suggesting enhanced motor control. Cognitively, both groups refined their mental representation structures over time, but the GPG exhibited a more structured and sophisticated cognitive mapping of the task post-practice. Discussion: The findings highlight the intertwined nature of biomechanical control, tactile feedback, and cognitive processing in motor skill acquisition. The results support established theories, such as the cognitive action architecture approach, emphasizing the role of mental representation in planning and executing motor actions. The integration of cognitive primitives in our analysis provides a theoretical framework that connects observable behaviors to underlying cognitive strategies, enhancing the understanding of motor learning across various contexts. Our study underscores the necessity of a holistic approach to motor learning research, recognizing the complex interaction between cognitive and motor processes in skill acquisition.

Exploring motor skill acquisition in bimanual coordination: insights from navigating a novel maze task.

In this study, we introduce a novel maze task designed to investigate naturalistic motor learning in bimanual coordination. We developed and validated an extended set of movement primitives tailored to capture the full spectrum of scenarios encountered in a maze game. Over a 3-day training period, we evaluated participants' performance using these primitives and a custom-developed software, enabling precise quantification of performance. Our methodology integrated the primitives with in-depth kinematic analyses and thorough thumb pressure assessments, charting the trajectory of participants' progression from novice to proficient stages. Results demonstrated consistent improvement in maze performance and significant adaptive changes in joint behaviors and strategic recalibrations in thumb pressure distribution. These findings highlight the central nervous system's adaptability in orchestrating sophisticated motor strategies and the crucial role of tactile feedback in precision tasks. The maze platform and setup emerge as a valuable foundation for future experiments, providing a tool for the exploration of motor learning and coordination dynamics. This research underscores the complexity of bimanual motor learning in naturalistic environments, enhancing our understanding of skill acquisition and task efficiency while emphasizing the necessity for further exploration and deeper investigation into these adaptive mechanisms.

Autonomous swab robot for naso- and oropharyngeal COVID-19 screening.

The COVID-19 outbreak has triggered a global health and economic crisis, necessitating widespread testing to control viral spread amidst rising cases and fatalities. The recommended testing method, a combined naso- and oropharyngeal swab, poses risks and demands limited protective gear. In response to the COVID-19 pandemic, we developed and tested the first autonomous swab robot station for Naso- and Oropharyngeal Coronavirus Screening (SR-NOCS). A force-sensitive robot running under a Cartesian impedance controller is employed to drive the swab to the sampling area. This groundbreaking device underwent two clinical studies-one conducted during the initial pandemic lockdown in Europe (early 2021) and the other, more recently, in a public place after the pandemic had subsided earlier in the year 2023. In total, 52 patients suspected of COVID-19 infection were included in these clinical studies. The results revealed a complete positive correlation between autonomous and manual sampling. The test subjects exhibited a high acceptance rate, all expressing a willingness to undergo future tests with SR-NOCS. Based on our findings, such systems could enhance testing capabilities, potentially conducting up to 300 tests per robot per day with consistent precision. The tests can be carried out with minimal supervision, reducing infection risks and effectively safeguarding patients and healthcare workers.

Toward telemedical diagnostics-clinical evaluation of a robotic examination system for emergency patients.

Introduction: The SARS-CoV-2 pandemic has affected global public healthcare for several years. Numerous medical professionals have been infected since the outbreak in 2019, resulting in a shortage of healthcare providers. Since traditional personal protective wear was insufficient to eliminate the virus transmission reliably, new strategies to avoid cross-infection were imperative while enabling high-quality medical care. In the project ProteCT, we investigated the potential of robotic-assisted examination in providing medical examination via a telemedical approach. Material and Methods: We constructed a fully functional examination cabin equipped with cameras, microphones, screens and robotic arms to evaluate usability and perception. Therefore, we conducted a preliminary study with 10 healthy volunteers and 10 physicians to gain first insights and optimize the setup. In a second step, we performed telemedical examinations of actual patients from the local emergency department to compare the robotic approach with the classical method of measuring vital signs, auscultation, palpation and percussion. Results: The preliminary study identified basic requirements, such as the need for force-feedback and telemedical training for physicians. In the main study, acceptance was high and most patients indicated they would use a telemedical system again. Our setup enabled the physician to make the same diagnoses as by classic examination in the emergency department in most cases. Discussion: The potential acceptance of a telemedical system such as ProteCT is high. Robotic telemedical approaches could complement future healthcare beyond the Corona pandemic to reach rural areas or even war zones. Moreover, the daily clinical use of robotic telemedicine could improve patients' safety, the quality of perioperative management and the workflow in any medical facility. Conclusion: The development of telemedical and telerobotic systems is a multidisciplinary and complex challenge. However, acceptance of the proposed system was high among patients and physicians, indicating the potential use of similar systems for future healthcare.

From Human Hand to Grasp Surface Detection, Tracking & Analysis.

Hands are paramount for dexterous interactions that humans exhibit in daily life. Understanding the intricacies of human hand-object interactions is therefore necessary. Unfortunately, the limitations of state-of-the-art technologies make capturing the full hand-object complexity unfeasible, giving rise to the need for new technological means to achieve this aim. In this work, we propose an end-to-end framework in which individualized hand models are derived and used to capture quantitative personalized hand-object interaction information, precisely, hand shape, kinematics, and contact surfaces. The results of this study serve as a proof of concept that such a framework can significantly deepen personalized hand-object interaction analyses, providing, in perspective, insights for medical diagnoses and rehabilitation, among others.Clinical relevance- Our work showcases the need to incorporate bespoke human hand models in individualized hand function assessment technologies, as hand-object interaction information is subject-dependent.

Assessing Human-Human Kinematics for the Implementation of Robot-Assisted Physical Therapy in Humanoids: A Pilot Study.

The development of humanoids with bimanual manipulator arms may facilitate assistive robots to perform physical therapy with older adults living at home. As we assume the human-human interaction to be the gold standard of physical therapy, we propose a kinematics analysis to derive guidelines for implementing physical therapy assisted by humanoids. Therefore, a pilot study was carried out involving three physical therapists and two participants acting as exemplary patients. The study analyzes the therapists' movement strategy, including the position and orientation of the therapists' bodies in relation to the participants and the placement of the therapists' hands on the upper limb segment of the participants, as well as the inter- and intravariability during the performance of a ROM (range of motion) assessment. The results demonstrate that while physical therapists exhibit variation in their interaction strategies, they still achieve a consistently low level of variability in their manipulation space.

Tactile Robotic Telemedicine for Safe Remote Diagnostics in Times of Corona: System Design, Feasibility and Usability Study.

The current crisis surrounding the COVID-19 pandemic demonstrates the amount of responsibility and the workload on our healthcare system and, above all, on the medical staff around the world. In this work, we propose a promising approach to overcome this problem using robot-assisted telediagnostics, which allows medical experts to examine patients from distance. The designed telediagnostic system consists of two robotic arms. Each robot is located at the doctor and patient sites. Such a system enables the doctor to have a direct conversation via telepresence and to examine patients through robot-assisted inspection (guided tactile and audiovisual contact). The proposed bilateral teleoperation system is redundant in terms of teleoperation control algorithms and visual feedback. Specifically, we implemented two main control modes: joint-based and displacement-based teleoperation. The joint-based mode was implemented due to its high transparency and ease of mapping between Leader and Follower whereas the displacement-based is highly flexible in terms of relative pose mapping and null-space control. Tracking tests between Leader and Follower were conducted on our system using both wired and wireless connections. Moreover, our system was tested by seven medical doctors in two experiments. User studies demonstrated the system's usability and it was successfully validated by the medical experts.

Role of Tactile Noise in the Control of Digit Normal Force.

Whenever we grasp and lift an object, our tactile system provides important information on the contact location and the force exerted on our skin. The human brain integrates signals from multiple sites for a coherent representation of object shape, inertia, weight, and other material properties. It is still an open question whether the control of grasp force occurs at the level of individual fingers or whether it is also influenced by the control and the signals from the other fingers of the same hand. In this work, we approached this question by asking participants to lift, transport, and replace a sensorized object, using three- and four-digit grasp. Tactile input was altered by covering participant's fingertips with a rubber thimble, which reduced the reliability of the tactile sensory input. In different experimental conditions, we covered between one and three fingers opposing the thumb. Normal forces at each finger and the thumb were recorded while grasping and holding the object, with and without the thimble. Consistently with previous studies, reducing tactile sensitivity increased the overall grasping force. The gasping force increased in the covered finger, whereas it did not change from baseline in the remaining bare fingers (except the thumb for equilibrium constraints). Digit placement and object tilt were not systematically affected by rubber thimble conditions. Our results suggest that, in each finger opposing thumb, digit normal force is controlled locally in response to the applied tactile perturbation.

Depth discrimination of constant angular size stimuli in action space: role of accommodation and convergence cues.

In our daily life experience, the angular size of an object correlates with its distance from the observer, provided that the physical size of the object remains constant. In this work, we investigated depth perception in action space (i.e., beyond the arm reach), while keeping the angular size of the target object constant. This was achieved by increasing the physical size of the target object as its distance to the observer increased. To the best of our knowledge, this is the first time that a similar protocol has been tested in action space, for distances to the observer ranging from 1.4-2.4 m. We replicated the task in virtual and real environments and we found that the performance was significantly different between the two environments. In the real environment, all participants perceived the depth of the target object precisely. Whereas, in virtual reality (VR) the responses were significantly less precise, although, still above chance level in 16 of the 20 observers. The difference in the discriminability of the stimuli was likely due to different contributions of the convergence and the accommodation cues in the two environments. The values of Weber fractions estimated in our study were compared to those reported in previous studies in peripersonal and action space.

Path integration in tactile perception of shapes.

Whenever we move the hand across a surface, tactile signals provide information about the relative velocity between the skin and the surface. If the system were able to integrate the tactile velocity information over time, cutaneous touch may provide an estimate of the relative displacement between the hand and the surface. Here, we asked whether humans are able to form a reliable representation of the motion path from tactile cues only, integrating motion information over time. In order to address this issue, we conducted three experiments using tactile motion and asked participants (1) to estimate the length of a simulated triangle, (2) to reproduce the shape of a simulated triangular path, and (3) to estimate the angle between two-line segments. Participants were able to accurately indicate the length of the path, whereas the perceived direction was affected by a direction bias (inward bias). The response pattern was thus qualitatively similar to the ones reported in classical path integration studies involving locomotion. However, we explain the directional biases as the result of a tactile motion aftereffect.