A Concept of a Plug-In Simulator for Increasing the Effectiveness of Rescue Operators When Using Hydrostatically Driven Manipulators.

The introduction of Unmanned Ground Vehicles (UGVs) into the field of rescue operations is an ongoing process. New tools, such as UGV platforms and dedicated manipulators, provide new opportunities but also come with a steep learning curve. The best way to familiarize operators with new solutions are hands-on courses but their deployment is limited, mostly due to high costs and limited equipment numbers. An alternative way is to use simulators, which from the software side, resemble video games. With the recent expansion of the video game engine industry, currently developed software becomes easier to produce and maintain. This paper tries to answer the question of whether it is possible to develop a highly accurate simulator of a rescue and IED manipulator using a commercially available game engine solution. Firstly, the paper describes different types of simulators for robots currently available. Next, it provides an in-depth description of a plug-in simulator concept. Afterward, an example of a hydrostatic manipulator arm and its virtual representation is described alongside validation and evaluation methodologies. Additionally, the paper provides a set of metrics for an example rescue scenario. Finally, the paper describes research conducted in order to validate the representation accuracy of the developed simulator.

A Lightweight and Affordable Wearable Haptic Controller for Robot-Assisted Microsurgery.

In robot-assisted microsurgery (RAMS), surgeons often face the challenge of operating with minimal feedback, particularly lacking in haptic feedback. However, most traditional desktop haptic devices have restricted operational areas and limited dexterity. This report describes a novel, lightweight, and low-budget wearable haptic controller for teleoperated microsurgical robotic systems. We designed a wearable haptic interface entirely made using off-the-shelf material-PolyJet Photopolymer, fabricated using liquid and solid hybrid 3D co-printing technology. This interface was designed to resemble human soft tissues and can be wrapped around the fingertips, offering direct contact feedback to the operator. We also demonstrated that the device can be easily integrated with our motion tracking system for remote microsurgery. Two motion tracking methods, marker-based and marker-less, were compared in trajectory-tracking experiments at different depths to find the most effective motion tracking method for our RAMS system. The results indicate that within the 4 to 8 cm tracking range, the marker-based method achieved exceptional detection rates. Furthermore, the performance of three fusion algorithms was compared to establish the unscented Kalman filter as the most accurate and reliable. The effectiveness of the wearable haptic controller was evaluated through user studies focusing on the usefulness of haptic feedback. The results revealed that haptic feedback significantly enhances depth perception for operators during teleoperated RAMS.

Communication Delay Outlier Detection and Compensation for Teleoperation Using Stochastic State Estimation.

There have been numerous studies attempting to overcome the limitations of current autonomous driving technologies. However, there is no doubt that it is challenging to promise integrity of safety regarding urban driving scenarios and dynamic driving environments. Among the reported countermeasures to supplement the uncertain behavior of autonomous vehicles, teleoperation of the vehicle has been introduced to deal with the disengagement of autonomous driving. However, teleoperation can lead the vehicle to unforeseen and hazardous situations from the viewpoint of wireless communication stability. In particular, communication delay outliers that severely deviate from the passive communication delay should be highlighted because they could hamper the cognition of the circumstances monitored by the teleoperator, or the control signal could be contaminated regardless of the teleoperator's intention. In this study, communication delay outliers were detected and classified based on the stochastic approach (passive delays and outliers were estimated as 98.67% and 1.33%, respectively). Results indicate that communication delay outliers can be automatically detected, independently of the real-time quality of wireless communication stability. Moreover, the proposed framework demonstrates resilience against outliers, thereby mitigating potential performance degradation.

Network Latency in Teleoperation of Connected and Autonomous Vehicles: A Review of Trends, Challenges, and Mitigation Strategies.

With remarkable advancements in the development of connected and autonomous vehicles (CAVs), the integration of teleoperation has become crucial for improving safety and operational efficiency. However, teleoperation faces substantial challenges, with network latency being a critical factor influencing its performance. This survey paper explores the impact of network latency along with state-of-the-art mitigation/compensation approaches. It examines cascading effects on teleoperation communication links (i.e., uplink and downlink) and how delays in data transmission affect the real-time perception and decision-making of operators. By elucidating the challenges and available mitigation strategies, the paper offers valuable insights for researchers, engineers, and practitioners working towards the seamless integration of teleoperation in the evolving landscape of CAVs.

Gesture Recognition Framework for Teleoperation of Infrared (IR) Consumer Devices Using a Novel pFMG Soft Armband.

Wearable technologies represent a significant advancement in facilitating communication between humans and machines. Powered by artificial intelligence (AI), human gestures detected by wearable sensors can provide people with seamless interaction with physical, digital, and mixed environments. In this paper, the foundations of a gesture-recognition framework for the teleoperation of infrared consumer electronics are established. This framework is based on force myography data of the upper forearm, acquired from a prototype novel soft pressure-based force myography (pFMG) armband. Here, the sub-processes of the framework are detailed, including the acquisition of infrared and force myography data; pre-processing; feature construction/selection; classifier selection; post-processing; and interfacing/actuation. The gesture recognition system is evaluated using 12 subjects' force myography data obtained whilst performing five classes of gestures. Our results demonstrate an inter-session and inter-trial gesture average recognition accuracy of approximately 92.2% and 88.9%, respectively. The gesture recognition framework was successfully able to teleoperate several infrared consumer electronics as a wearable, safe and affordable human-machine interface system. The contribution of this study centres around proposing and demonstrating a user-centred design methodology to allow direct human-machine interaction and interface for applications where humans and devices are in the same loop or coexist, as typified between users and infrared-communicating devices in this study.

The human-factors' challenges of (tele)drivers of Autonomous Vehicles.

Autonomous capabilities, including Autonomous Vehicle (AV) technology, aim to reduce human effort, extend capabilities, and enhance safety. While AVs offer societal benefits, human intervention remains necessary, especially in complex situations. As communication technology advances, human intervention is possible from remote sites. In such remote locations, highly skilled tele-drivers (TEDs) are ready to face situations too complicated for the AV. However, current work still needs a comprehensive mapping of the challenges that TEDs would face. Some of these challenges are shared with IVDs but may have stronger or weaker effects on the remote driver's ability to maintain safety. Other challenges, such as limited situational awareness of the road scene, the indirect experience of vehicle motion, and communication latency, are unique to TEDs. We assess the challenges, comparing their impact on TEDs versus IVDs, and explore technological countermeasures aimed at mitigating specific challenges encountered by TEDs. Lastly, we identified knowledge gaps and areas lacking understanding in the literature, highlighting avenues for future research and practical implications for practitioners.

Until fully autonomous vehicles are introduced, even high-level AVs occasionally require some human interventionIntervention can be provided locally or remotely by in-vehicle or remote operators, respectivelyThe two types of operators (in-vehicle and remote) will likely have different needs and experience different challengesWe reviewed and mapped the human factors' challenges and compared their possible effects on the two types of operatorsWe discussed several technology countermeasures to mitigate some of these challengesWe map the literature and knowledge gaps regarding this new role in transportation.

Manipulation Tasks in Hazardous Environments Using a Teleoperated Robot: A Case Study at CERN.

Remote robotic systems are employed in the CERN accelerator complex to perform different tasks, such as the safe handling of cables and their connectors. Without dedicated control, these kinds of actions are difficult and require the operators' intervention, which is subjected to dangerous external agents. In this paper, two novel modules of the CERNTAURO framework are presented to provide a safe and usable solution for managing optical fibres and their connectors. The first module is used to detect touch and slippage, while the second one is used to regulate the grasping force and contrast slippage. The force reference was obtained with a combination of object recognition and a look-up table method. The proposed strategy was validated with tests in the CERN laboratory, and the preliminary experimental results demonstrated statistically significant increases in time-based efficiency and in the overall relative efficiency of the tasks.

A Mixed-Reality-Based Unknown Space Navigation Method of a Flexible Manipulator.

A hyper-redundant flexible manipulator is characterized by high degree(s) of freedom (DoF), flexibility, and environmental adaptability. It has been used for missions in complex and unknown spaces, such as debris rescue and pipeline inspection, where the manipulator is not intelligent enough to face complex situations. Therefore, human intervention is required to assist in decision-making and control. In this paper, we designed an interactive navigation method based on mixed reality (MR) of a hyper-redundant flexible manipulator in an unknown space. A novel teleoperation system frame is put forward. An MR-based interface was developed to provide a virtual model of the remote workspace and virtual interactive interface, allowing the operator to observe the real-time situation from a third perspective and issue commands to the manipulator. As for environmental modeling, a simultaneous localization and mapping (SLAM) algorithm based on an RGB-D camera is applied. Additionally, a path-finding and obstacle avoidance method based on artificial potential field (APF) is introduced to ensure that the manipulator can move automatically under the artificial command in the remote space without collision. The results of the simulations and experiments validate that the system exhibits good real-time performance, accuracy, security, and user-friendliness.

Learning Compliant Box-in-Box Insertion through Haptic-Based Robotic Teleoperation.

In modern logistics, the box-in-box insertion task is representative of a wide range of packaging applications, and automating compliant object insertion is difficult due to challenges in modelling the object deformation during insertion. Using Learning from Demonstration (LfD) paradigms, which are frequently used in robotics to facilitate skill transfer from humans to robots, can be one solution for complex tasks that are difficult to mathematically model. In order to automate the box-in-box insertion task for packaging applications, this study makes use of LfD techniques. The proposed framework has three phases. Firstly, a master-slave teleoperated robot system is used in the initial phase to haptically demonstrate the insertion task. Then, the learning phase involves identifying trends in the demonstrated trajectories using probabilistic methods, in this case, Gaussian Mixture Regression. In the third phase, the insertion task is generalised, and the robot adjusts to any object position using barycentric interpolation. This method is novel because it tackles tight insertion by taking advantage of the boxes' natural compliance, making it possible to complete the task even with a position-controlled robot. To determine whether the strategy is generalisable and repeatable, experimental validation was carried out.

Exploration into the Needs and Requirements of the Remote Driver When Teleoperating the 5G-Enabled Level 4 Automated Vehicle in the Real World-A Case Study of 5G Connected and Automated Logistics.

Connected and automated vehicles have the potential to deliver significant environmental, safety, economic and social benefits. The key advancement for automated vehicles with higher levels of automation (SAE Level 4 and over) is fail-operational. One possible solution for the failsafe mode of automated vehicles is a 5G-enabled teleoperation system controlled by remote drivers. However, knowledge is missing regarding understanding of the human-machine interaction in teleoperation from the perspective of remote drivers. To address this research gap, this study qualitatively investigated the acceptance, attitudes, needs and requirements of remote drivers when teleoperating a 5G-enabled Level 4 automated vehicle (5G L4 AV) in the real world. The results showed that remote drivers are positive towards the 5G L4 AV. They would like to constantly monitor the driving when they are not controlling the vehicle remotely. Improving their field of vision for driving and enhancing the perception of physical motion feedback are the two key supports required by remote drivers in 5G L4 AVs. The knowledge gained in this study provides new insights into facilitating the design and development of safe, effective and user-friendly teleoperation systems in vehicle automation.

An Evaluation of Sleepiness, Performance, and Workload Among Operators During a Real-Time Reactive Telerobotic Lunar Mission Simulation.

OBJECTIVE: We assessed operator performance during a real-time reactive telerobotic lunar mission simulation to understand how daytime versus nighttime operations might affect sleepiness, performance, and workload. BACKGROUND: Control center operations present factors that can influence sleepiness, neurobehavioral performance, and workload. Each spaceflight mission poses unique challenges that make it difficult to predict how long operators can safely and accurately conduct operations. We aimed to evaluate the performance impact of time-on-task and time-of-day using a simulated telerobotic lunar rover to better inform staffing and scheduling needs for the upcoming Volatiles Investigating Polar Exploration Rover (VIPER) mission. METHODS: We studied seven trained operators in a simulated mission control environment. Operators completed two five-hour simulations in a randomized order, beginning at noon and midnight. Performance was evaluated every 25 minutes using the Karolinska Sleepiness Scale, Psychomotor Vigilance Task, and NASA Task Load Index. RESULTS: Participants rated themselves as sleepier (5.06 ± 2.28) on the midnight compared to the noon simulation (3.12 ± 1.44; p < .001). Reaction time worsened over time during the midnight simulation but did not vary between simulations. Workload was rated higher during the noon (37.93 ± 20.09) compared to the midnight simulation (32.09 ± 21.74; p = .007). CONCLUSION: Our findings suggest that work shifts during future operations should be limited in duration to minimize sleepiness. Our findings also suggest that working during the day, when distractions are present, increases perceived workload. Further research is needed to understand how working consecutive shifts and taking breaks within a shift influence performance.

Comparison of Display Modality and Human-in-the-Loop Presence for On-Orbit Inspection of Spacecraft.

OBJECTIVE: To investigate the impact of interface display modalities and human-in-the-loop presence on the awareness, workload, performance, and user strategies of humans interacting with teleoperated robotic systems while conducting inspection tasks onboard spacecraft. BACKGROUND: Due to recent advancements in robotic technology, free-flying teleoperated robot inspectors are a viable alternative to extravehicular activity inspection operations. Teleoperation depends on the user's situation awareness; consequently, a key to successful operations is practical bi-directional communication between human and robot agents. METHOD: Participants (n = 19) performed telerobotic inspection of a virtual spacecraft during two degrees of temporal communication, a Synchronous Inspection task and an Asynchronous Inspection task. Participants executed the two tasks while using three distinct visual displays (2D, 3D, AR) and accompanying control systems. RESULTS: Anomaly detection performance was better during Synchronous Inspection than the Asynchronous Inspection of previously captured imagery. Users' detection accuracy reduced when given interactive exocentric 3D viewpoints to accompany the egocentric robot view. The results provide evidence that 3D projections, either demonstrated on a 2D interface or augmented reality hologram, do not affect the mean clearance violation time (local guidance performance), even though the subjects perceived a benefit. CONCLUSION: In the current implementation, the addition of augmented reality to a classical egocentric robot view for exterior inspection of spacecraft is unnecessary, as its margin of performance enhancement is limited in comparison. APPLICATION: Results are presented to inform future human-robot interfaces to support crew autonomy for deep space missions.

Temporal Sensitivity and Latency During Teleoperation: Using Track Clearance to Understand Errors in Future Projection.

OBJECTIVE: This study investigates the role of individual differences in time perception on task performance during teleoperation with latency. BACKGROUND: Long distance teleoperation induces latency, causing performance issues for the operator. Previous research demonstrated that individual differences in time perception predicted performance on a similar task, having participants navigate a radio controlled (RC) car around a track. This work extends the relationship into routes of varying course width to test whether differences in time perception predict movement over-/underestimation. METHOD: Participants completed a time estimation task and a route navigation task while experiencing latency. In the time estimation task, participants estimated the duration of multiple visual stimuli (2 s or less). In the route navigation task, participants moved a virtual cube across a route. Each trial varied in the amount of latency and the amount of horizontal clearance in the track (4-10 m for a 1.2-m-long/wide cube). RESULTS: The results showed fairly consistent latency by time estimation and latency by clearance interaction effects on a wide set of trial-level variables, such as completion time, and action-level performance variables, such as time spent moving per move event. However, the results were not consistently in the predicted direction. CONCLUSION: Results suggest that clearance and timing affect performance across latency, at both the overall level (i.e., trial completion time) and at the action level (time spent moving). An open question remains as to how these contextual factors affect movement strategy selection.

Do multimodal search cues help or hinder teleoperated search and rescue missions?

In search and rescue missions, teleoperated rovers equipped with sensor technology are deployed into harsh environments to search for targets. To support the search task, unimodal/multimodal cues can be presented via visual, acoustic and/or haptic channels. However, human operators often perform the search task in parallel with the driving task, which can cause interference of attentional resources based on multiple resource theory. Navigating corners can be a particularly challenging aspect of remote driving, as described with the Cornering Law. Therefore, search cues should not interfere with cornering. The present research explores how unimodal/multimodal search cues affect cornering performance, with typical communication delays of 50 ms and 500 ms. One-hundred thirty-one participants, distributed into two delay groups, performed a target search task with unimodal/multimodal search cues. Search cues did not interfere with cornering performance with 50 ms delays. For 500 ms delays, search cues presented via the haptic channel significantly interfered with the driving task. Practitioner summary: Teleoperated rovers can support search and rescue missions. Search cues may assist the human operator, but they may also interfere with the task of driving. The study examined interference of unimodal and multimodal search cues. Haptic cues should not be implemented for systems with a delay of 500 ms or more.

An Incremental Broad-Learning-System-Based Approach for Tremor Attenuation for Robot Tele-Operation.

The existence of the physiological tremor of the human hand significantly affects the application of tele-operation systems in performing high-precision tasks, such as tele-surgery, and currently, the process of effectively eliminating the physiological tremor has been an important yet challenging research topic in the tele-operation robot field. Some scholars propose using deep learning algorithms to solve this problem, but a large number of hyperparameters lead to a slow training speed. Later, the support-vector-machine-based methods have been applied to solve the problem, thereby effectively canceling tremors. However, these methods may lose the prediction accuracy, because learning energy cannot be accurately assigned. Therefore, in this paper, we propose a broad-learning-system-based tremor filter, which integrates a series of incremental learning algorithms to achieve fast remodeling and reach the desired performance. Note that the broad-learning-system-based filter has a fast learning rate while ensuring the accuracy due to its simple and novel network structure. Unlike other algorithms, it uses incremental learning algorithms to constantly update network parameters during training, and it stops learning when the error converges to zero. By focusing on the control performance of the slave robot, a sliding mode control approach has been used to improve the performance of closed-loop systems. In simulation experiments, the results demonstrated the feasibility of our proposed method.

Procedural Telementoring in Rural, Underdeveloped, and Austere Settings: Origins, Present Challenges, and Future Perspectives.

Telemedicine is perhaps the most rapidly growing area in health care. Approximately 15 million Americans receive medical assistance remotely every year. Yet rural communities face significant challenges in securing subspecialist care. In the United States, 25% of the population resides in rural areas, where less than 15% of physicians work. Current surgery residency programs do not adequately prepare surgeons for rural practice. Telementoring, wherein a remote expert guides a less experienced caregiver, has been proposed to address this challenge. Nonetheless, existing mentoring technologies are not widely available to rural communities, due to a lack of infrastructure and mentor availability. For this reason, some clinicians prefer simpler and more reliable technologies. This article presents past and current telementoring systems, with a focus on rural settings, and proposes aset of requirements for such systems. We conclude with a perspective on the future of telementoring systems and the integration of artificial intelligence within those systems.

Full-hand electrotactile feedback using electronic skin and matrix electrodes for high-bandwidth human-machine interfacing.

Tactile feedback is relevant in a broad range of human-machine interaction systems (e.g. teleoperation, virtual reality and prosthetics). The available tactile feedback interfaces comprise few sensing and stimulation units, which limits the amount of information conveyed to the user. The present study describes a novel technology that relies on distributed sensing and stimulation to convey comprehensive tactile feedback to the user of a robotic end effector. The system comprises six flexible sensing arrays (57 sensors) integrated on the fingers and palm of a robotic hand, embedded electronics (64 recording channels), a multichannel stimulator and seven flexible electrodes (64 stimulation pads) placed on the volar side of the subject's hand. The system was tested in seven subjects asked to recognize contact positions and identify contact sliding on the electronic skin, using distributed anode configuration (DAC) and single dedicated anode configuration. The experiments demonstrated that DAC resulted in substantially better performance. Using DAC, the system successfully translated the contact patterns into electrotactile profiles that the subjects could recognize with satisfactory accuracy ([Formula: see text] for static and [Formula: see text] for dynamic patterns). The proposed system is an important step towards the development of a high-density human-machine interfacing between the user and a robotic hand. This article is part of the theme issue 'Advanced neurotechnologies: translating innovation for health and well-being'.

Auditory Feedback for Enhanced Sense of Agency in Shared Control.

There is a growing need for robots that can be remotely controlled to perform tasks of one's own choice. However, the SoA (Sense of Agency: the sense of recognizing that the motion of an observed object is caused by oneself) is reduced because the subject of the robot motion is identified as external due to shared control. To address this issue, we aimed to suppress the decline in SoA by presenting auditory feedback that aims to blur the distinction between self and others. We performed the tracking task in a virtual environment under four different auditory feedback conditions, with varying levels of automation to manipulate the virtual robot gripper. Experimental results showed that the proposed auditory feedback suppressed the decrease in the SoA at a medium level of automation. It is suggested that our proposed auditory feedback could blur the distinction between self and others, and that the operator attributes the subject of the motion of the manipulated object to himself.

IMU Motion Capture Method with Adaptive Tremor Attenuation in Teleoperation Robot System.

Teleoperation robot systems can help humans perform tasks in unstructured environments. However, non-intuitive control interfaces using only a keyboard or joystick and physiological tremor reduce the performance of teleoperation. This paper presents an intuitive control interface based on the wearable device gForcePro+ armband. Two gForcePro+ armbands are worn at the centroid of the upper arm and forearm, respectively. Firstly, the kinematics model of the human arm is established, and the inertial measurement units (IMUs) are used to capture the position and orientation information of the end of the arm. Then, a regression model of angular transformation is developed for the phenomenon that the rotation axis of the torsion joint is not perfectly aligned with the limb segment during motion, which can be applied to different individuals. Finally, to attenuate the physiological tremor, a variable gain extended Kalman filter (EKF) fusing sEMG signals is developed. The described control interface shows good attitude estimation accuracy compared to the VICON optical capture system, with an average angular RMSE of 4.837° ± 1.433°. The performance of the described filtering method is tested using the xMate3 Pro robot, and the results show it can improve the tracking performance of the robot and reduce the tremor.

Teleoperation of High-Speed Robot Hand with High-Speed Finger Position Recognition and High-Accuracy Grasp Type Estimation.

This paper focuses on the teleoperation of a robot hand on the basis of finger position recognition and grasp type estimation. For the finger position recognition, we propose a new method that fuses machine learning and high-speed image-processing techniques. Furthermore, we propose a grasp type estimation method according to the results of the finger position recognition by using decision tree. We developed a teleoperation system with high speed and high responsiveness according to the results of the finger position recognition and grasp type estimation. By using the proposed method and system, we achieved teleoperation of a high-speed robot hand. In particular, we achieved teleoperated robot hand control beyond the speed of human hand motion.