Machine and deep learning for workflow recognition during surgery.

Recent years have seen tremendous progress in artificial intelligence (AI), such as with the automatic and real-time recognition of objects and activities in videos in the field of computer vision. Due to its increasing digitalization, the operating room (OR) promises to directly benefit from this progress in the form of new assistance tools that can enhance the abilities and performance of surgical teams. Key for such tools is the recognition of the surgical workflow, because efficient assistance by an AI system requires this system to be aware of the surgical context, namely of all activities taking place inside the operating room. We present here how several recent techniques relying on machine and deep learning can be used to analyze the activities taking place during surgery, using videos captured from either endoscopic or ceiling-mounted cameras. We also present two potential clinical applications that we are developing at the University of Strasbourg with our clinical partners.

OR black box and surgical control tower: Recording and streaming data and analytics to improve surgical care.

Effective and safe surgery results from a complex sociotechnical process prone to human error. Acquiring large amount of data on surgical care and modelling the process of surgery with artificial intelligence's computational methods could shed lights on system strengths and limitations and enable computer-based smart assistance. With this vision in mind, surgeons and computer scientists have joined forces in a novel discipline called Surgical Data Science. In this regard, operating room (OR) black boxes and surgical control towers are being developed to systematically capture comprehensive data on surgical procedures and to oversee and assist during operating rooms activities, respectively. Most of the early Surgical Data Science works have focused on understanding risks and resilience factors affecting surgical safety, the context and workflow of procedures, and team behaviors. These pioneering efforts in sensing and analyzing surgical activities, together with the advent of precise robotic actuators, bring surgery on the verge of a fourth revolution characterized by smart assistance in perceptual, cognitive and physical tasks. Barriers to implement this vision exist, but the surgical-technical partnerships set by ambitious efforts such as the OR black box and the surgical control tower are working to overcome these roadblocks and translate the vision and early works described in the manuscript into value for patients, surgeons and health systems.