Augmented-Reality Surgery to Guide Head and Neck Cancer Re-resection: A Feasibility and Accuracy Study.

BACKGROUND: Given the complex three-dimensional (3D) anatomy of head and neck cancer specimens, head and neck surgeons often have difficulty relocating the site of an initial positive margin to perform re-resection. This cadaveric study aimed to determine the feasibility and accuracy of augmented reality surgery to guide head and neck cancer re-resections. METHODS: This study investigated three cadaveric specimens. The head and neck resection specimen was 3D scanned and exported to the HoloLens augmented reality environment. The surgeon manually aligned the 3D specimen hologram into the resection bed. Accuracy of manual alignment and time intervals throughout the protocol were recorded. RESULTS: The 20 head and neck cancer resections performed in this study included 13 cutaneous and 7 oral cavity resections. The mean relocation error was 4 mm (range, 1-15 mm) with a standard deviation of 3.9 mm. The mean overall protocol time, from the start of 3D scanning to alignment into the resection bed, was 25.3 ± 8.9 min (range, 13.2-43.2 min). Relocation error did not differ significantly when stratified by greatest dimension of the specimen. The mean relocation error of complex oral cavity composite specimens (maxillectomy and mandibulectomy) differed significantly from that of all the other specimen types (10.7 vs 2.8; p < 0.01). CONCLUSIONS: This cadaveric study demonstrated the feasibility and accuracy of augmented reality to guide re-resection of initial positive margins in head and neck cancer surgery.

Crystal structure of 5-[bis-(4-eth-oxy-phenyl)amino]-thio-phene-2-carbaldehyde.

In the title compound, C21H21NO3S, the planes of the two benzene rings are nearly perpendicular to one another [dihedral angle = 84.50 (10)°] and they are oriented with respect to the plane of the thio-phene ring at dihedral angles of 59.15 (9) and 66.61 (9)°. In the crystal, mol-ecules are linked by weak C-H⋯O hydrogen bonds, forming supra-molecular chains propagating along the b-axis direction.

Crystal structure of 4-eth-oxy-N-(4-eth-oxy-phen-yl)-N-phenyl-aniline.

In the title compound, C22H23NO2, the planes of the eth-oxy-benzene rings are oriented with respect to that of the phenyl ring at dihedral angles of 61.77 (8) and 84.77 (8)°, and they are twisted with respect to one another, with a dihedral angle of 80.37 (7)°. In the crystal, weak C-H⋯π inter-actions link the mol-ecules into supra-molecular chains propagating along [101].

2-{(E)-[4-(Di-phenyl-amino)-phen-yl]imino-meth-yl}phenol.

The asymmetric unit of the title Schiff base molecule, C25H20N2O, contains two independent mol-ecules. In each mol-ecule, the C=N bond is in an E conformation. The most significant difference between the two mol-ecules is seen for the dihedral angles between the meth-oxy-substituted benzene ring and the two phenyl rings, which are 85.5 (1) and 82.3 (1)° in the first mol-ecule, and 49.0 (1) and 40.4 (1)° in the second. This conformational difference is reflected in the central C=N-C C torsion angle, which is 28.7 (2)° in the first mol-ecule and -29.8 (3)° in the other. In each mol-ecule, there is an intra-molecular O-H⋯N hydrogen bond.

ASSIST-U: A system for segmentation and image style transfer for ureteroscopy.

Kidney stones require surgical removal when they grow too large to be broken up externally or to pass on their own. Upper tract urothelial carcinoma is also sometimes treated endoscopically in a similar procedure. These surgeries are difficult, particularly for trainees who often miss tumours, stones or stone fragments, requiring re-operation. Furthermore, there are no patient-specific simulators to facilitate training or standardized visualization tools for ureteroscopy despite its high prevalence. Here a system ASSIST-U is proposed to create realistic ureteroscopy images and videos solely using preoperative computerized tomography (CT) images to address these unmet needs. A 3D UNet model is trained to automatically segment CT images and construct 3D surfaces. These surfaces are then skeletonized for rendering. Finally, a style transfer model is trained using contrastive unpaired translation (CUT) to synthesize realistic ureteroscopy images. Cross validation on the CT segmentation model achieved a Dice score of 0.853 ± 0.084. CUT style transfer produced visually plausible images; the kernel inception distance to real ureteroscopy images was reduced from 0.198 (rendered) to 0.089 (synthesized). The entire pipeline from CT to synthesized ureteroscopy is also qualitatively demonstrated. The proposed ASSIST-U system shows promise for aiding surgeons in the visualization of kidney ureteroscopy.

Intraoperative gaze guidance with mixed reality.

Efficient communication and collaboration are essential in the operating room for successful and safe surgery. While many technologies are improving various aspects of surgery, communication between attending surgeons, residents, and surgical teams is still limited to verbal interactions that are prone to misunderstandings. Novel modes of communication can increase speed and accuracy, and transform operating rooms. A mixed reality (MR) based gaze sharing application on Microsoft HoloLens 2 headset that can help expert surgeons indicate specific regions, communicate with decreased verbal effort, and guide novices throughout an operation is presented. The utility of the application is tested with a user study of endoscopic kidney stone localization completed by urology experts and novice surgeons. Improvement is observed in the NASA task load index surveys (up to 25.23%), in the success rate of the task (6.98% increase in localized stone percentage), and in gaze analyses (up to 31.99%). The proposed application shows promise in both operating room applications and surgical training tasks.

Development of an augmented reality guidance system for head and neck cancer resection.

The use of head-mounted augmented reality (AR) for surgeries has grown rapidly in recent years. AR aids in intraoperative surgical navigation through overlaying three-dimensional (3D) holographic reconstructions of medical data. However, performing AR surgeries on complex areas such as the head and neck region poses challenges in terms of accuracy and speed. This study explores the feasibility of an AR guidance system for resections of positive tumour margins in a cadaveric specimen. The authors present an intraoperative solution that enables surgeons to upload and visualize holographic reconstructions of resected cadaver tissues. The solution involves using a 3D scanner to capture detailed scans of the resected tissue, which are subsequently uploaded into our software. The software converts the scans of resected tissues into specimen holograms that are viewable through a head-mounted AR display. By re-aligning these holograms with cadavers with gestures or voice commands, surgeons can navigate the head and neck tumour site. This workflow can run concurrently with frozen section analysis. On average, the authors achieve an uploading time of 2.98 min, visualization time of 1.05 min, and re-alignment time of 4.39 min, compared to the 20 to 30 min typical for frozen section analysis. The authors achieve a mean re-alignment error of 3.1 mm. The authors' software provides a foundation for new research and product development for using AR to navigate complex 3D anatomy in surgery.

Towards navigation in endoscopic kidney surgery based on preoperative imaging.

Endoscopic renal surgeries have high re-operation rates, particularly for lower volume surgeons. Due to the limited field and depth of view of current endoscopes, mentally mapping preoperative computed tomography (CT) images of patient anatomy to the surgical field is challenging. The inability to completely navigate the intrarenal collecting system leads to missed kidney stones and tumors, subsequently raising recurrence rates. A guidance system is proposed to estimate the endoscope positions within the CT to reduce re-operation rates. A Structure from Motion algorithm is used to reconstruct the kidney collecting system from the endoscope videos. In addition, the kidney collecting system is segmented from CT scans using 3D U-Net to create a 3D model. The two collecting system representations can then be registered to provide information on the relative endoscope position. Correct reconstruction and localization of intrarenal anatomy and endoscope position is demonstrated. Furthermore, a 3D map is created supported by the RGB endoscope images to reduce the burden of mental mapping during surgery. The proposed reconstruction pipeline has been validated for guidance. It can reduce the mental burden for surgeons and is a step towards the long-term goal of reducing re-operation rates in kidney stone surgery.

Cognitive effort detection for tele-robotic surgery via personalized pupil response modeling.

PURPOSE: Gaze tracking and pupillometry are established proxies for cognitive load, giving insights into a user's mental effort. In tele-robotic surgery, knowing a user's cognitive load can inspire novel human-machine interaction designs, fostering contextual surgical assistance systems and personalized training programs. While pupillometry-based methods for estimating cognitive effort have been proposed, their application in surgery is limited by the pupil's sensitivity to brightness changes, which can mask pupil's response to cognitive load. Thus, methods considering pupil and brightness conditions are essential for detecting cognitive effort in unconstrained scenarios. METHODS: To contend with this challenge, we introduce a personalized pupil response model integrating pupil and brightness-based features. Discrepancies between predicted and measured pupil diameter indicate dilations due to non-brightness-related sources, i.e., cognitive effort. Combined with gaze entropy, it can detect cognitive load using a random forest classifier. To test our model, we perform a user study with the da Vinci Research Kit, where 17 users perform pick-and-place tasks in addition to auditory tasks known to generate cognitive effort responses. RESULTS: We compare our method to two baselines (BCPD and CPD), demonstrating favorable performance in varying brightness conditions. Our method achieves an average true positive rate of 0.78, outperforming the baselines (0.57 and 0.64). CONCLUSION: We present a personalized brightness-aware model for cognitive effort detection able to operate under unconstrained brightness conditions, comparing favorably to competing approaches, contributing to the advancement of cognitive effort detection in tele-robotic surgery. Future work will consider alternative learning strategies, handling the difficult positive-unlabeled scenario in user studies, where only some positive and no negative events are reliably known.

Cognitive load in tele-robotic surgery: a comparison of eye tracker designs.

PURPOSE: Eye gaze tracking and pupillometry are evolving areas within the field of tele-robotic surgery, particularly in the context of estimating cognitive load (CL). However, this is a recent field, and current solutions for gaze and pupil tracking in robotic surgery require assessment. Considering the necessity of stable pupillometry signals for reliable cognitive load estimation, we compare the accuracy of three eye trackers, including head and console-mounted designs. METHODS: We conducted a user study with the da Vinci Research Kit (dVRK), to compare the three designs. We collected eye tracking and dVRK video data while participants observed nine markers distributed over the dVRK screen. We compute and analyze pupil detection stability and gaze prediction accuracy for the three designs. RESULTS: Head-worn devices present better stability and accuracy of gaze prediction and pupil detection compared to console-mounted systems. Tracking stability along the field of view varies between trackers, with gaze predictions detected at invalid zones of the image with high confidence. CONCLUSION: While head-worn solutions show benefits in confidence and stability, our results demonstrate the need to improve eye tacker performance regarding pupil detection, stability, and gaze accuracy in tele-robotic scenarios.

A self-assembled nanohybrid composed of fluorophore-phenylamine nanorods and Ag nanocrystals: energy transfer, wavelength shift of fluorescence and TPEF applications for live-cell imaging.

A fluorophore-phenylamine derivative (L) has been coupled with silver nanocrystals (NCs) to construct an L-Ag nanohybrid. Owing to synergic effects of the L and Ag components, the exciton-plasmon interactions between L and Ag increase the strength of the donor-acceptor interaction within the nanohybrid, a fact that results in an energy-transfer process and further brings about a dramatic redshift of single-photon absorption and fluorescence, and a decreased fluorescence FL lifetime. The coupling effect also leads to enhancement of a series of nonlinear optical properties, including two-photon-excited fluorescence (TPEF), two-photon-absorption (TPA) cross section (δ), two-photon-absorption coefficient (ß), nonlinear refractive index (γ), and third order nonlinear optical susceptibility (χ((3))). The enhanced two-photon fluorescence of the nanohybrid is proven to be potentially useful for two-photon microscopy of live cells, such as HepG2. Moreover, cytotoxicity tests show that the low-micromolar concentrations of the nanohybrid do not cause significant reduction in cell viability over a period of at least 24 h and should be safe for further biological studies.

(E)-3-[5-(Di-phenyl-amino)-thio-phen-2-yl]-1-(pyridin-3-yl)prop-2-en-1-one.

In the title compound, C24H18N2OS, the pyridine and the two phenyl rings are oriented at dihedral angles of 10.1 (5), 71.7 (6) and 68.7 (5)°, respectively, to the central thio-phene ring. In the crystal, pairs of weak C-H⋯O hydrogen bonds link inversion-related mol-ecules, forming dimers. The dimers are linked by further weak C-H⋯O hydrogen bonds, forming chains running along the a-axis direction.

4-Eth-oxy-N-(4-eth-oxy-phen-yl)-N-(4-nitro-phen-yl)aniline.

In the title mol-ecule, C22H22N2O4, the eth-oxy-phenyl rings are oriented at dihedral angles of 69.31 (13) and 75.90 (13)° to the nitro-phenyl ring and are twisted to each other, making a dihedral angle of 78.55 (13)°. In the crystal, weak C-H⋯O hydrogen bonds and C-H⋯π inter-action link the mol-ecules into a three-dimensional supra-molecular architecture.

Eye gaze metrics for skill assessment and feedback in kidney stone surgery.

PURPOSE: Surgical skill assessment is essential for safe operations. In endoscopic kidney stone surgery, surgeons must perform a highly skill-dependent mental mapping from the pre-operative scan to the intraoperative endoscope image. Poor mental mapping can lead to incomplete exploration of the kidney and high reoperation rates. Yet there are few objective ways to evaluate competency. We propose to use unobtrusive eye-gaze measurements in the task space to evaluate skill and provide feedback. METHODS: We capture the surgeons' eye gaze on the surgical monitor with the Microsoft Hololens 2. To enable stable and accurate gaze detection, we develop a calibration algorithm to refine the eye tracking of the Hololens. In addition, we use a QR code to locate the eye gaze on the surgical monitor. We then run a user study with three expert and three novice surgeons. Each surgeon is tasked to locate three needles representing kidney stones in three different kidney phantoms. RESULTS: We find that experts have more focused gaze patterns. They complete the task faster, have smaller total gaze area, and the gaze fewer times outside the area of interest. While fixation to non-fixation ratio did not show significant difference in our findings, tracking the ratio over time shows different patterns between novices and experts. CONCLUSION: We show that a non-negligible difference holds between novice and expert surgeons' gaze metrics in kidney stone identification in phantoms. Expert surgeons demonstrate more targeted gaze throughout a trial, indicating their higher level of proficiency. To improve the skill acquisition process for novice surgeons, we suggest providing sub-task specific feedback. This approach presents an objective and non-invasive method to assess surgical competence.

Rethinking causality-driven robot tool segmentation with temporal constraints.

PURPOSE: Vision-based robot tool segmentation plays a fundamental role in surgical robots perception and downstream tasks. CaRTS, based on a complementary causal model, has shown promising performance in unseen counterfactual surgical environments in the presence of smoke, blood, etc. However, CaRTS requires over 30 iterations of optimization to converge for a single image due to limited observability. METHOD: To address the above limitations, we take temporal relation into consideration and propose a temporal causal model for robot tool segmentation on video sequences. We design an architecture named Temporally Constrained CaRTS (TC-CaRTS). TC-CaRTS has three novel modules to complement CaRTS-temporal optimization pipeline, kinematics correction network, and spatial-temporal regularization. RESULTS: Experiment results show that TC-CaRTS requires fewer iterations to achieve the same or better performance as CaRTS on different domains. All three modules are proven to be effective. CONCLUSION: We propose TC-CaRTS, which takes advantage of temporal constraints as additional observability. We show that TC-CaRTS outperforms prior work in the robot tool segmentation task with improved convergence speed on test datasets from different domains.

1,4-Dibut-oxy-2,5-bis-{(Z)-2-[4-(9H-carbazol-9-yl)phen-yl]ethen-yl}benzene.

The title compound, C(54)H(48)N(2)O(2), lies about an inversion centre. The carbazole ring system makes dihedral angles of 58.43 (7) and 88.96 (7)°, respectively, with the adjacent and central benzene rings. The dihedral angle between the two benzene rings is 52.01 (8)°. In the crystal, mol-ecules are linked by pairs of C-H⋯O inter-actions, forming a tape along the a axis. The methyl group is disordered over two sets of sites with occupancies of 0.63 (3) and 0.37 (3).

4,4'-{[4-(2,2':6',2''-Terpyridin-4'-yl)phen-yl]imino}-dibenzaldehyde.

The central pyridine ring of the 2,2':6',2''-terpyridine fragment of the title compound, C(35)H(24)N(4)O(2), forms dihedral angles of 8.3 (2), 10.6 (3) and 39.4 (3)°, respectively, with the two outer pyridine rings and the attached benzene ring. In the crystal, weak C-H⋯O inter-actions link the mol-ecules into chains in [010].

4-(4-Nitro-styr-yl)-N,N-diphenyl-aniline.

In the triaryl-amine group of the title compound, C(26)H(20)N(2)O(2), the N atom adopts an approximately trigonal-planar geometry, lying 0.046 (5) Šfrom the plane P defined by its three neighbouring C atoms; the benzene and two terminal phenyl rings are twisted by 37.4 (1), 31.4 (1) and 47.8 (1)°, respectively from plane P. In the trans-stilbene fragment, the two benzene rings form a dihedral angle of 31.3 (1)°. In the crystal, weak inter-molecular C-H⋯O inter-actions link the mol-ecules into ribbons in [100].