Mixed Reality and Artificial Intelligence: A Holistic Approach to Multimodal Visualization and Extended Interaction in Knee Osteotomy.

OBJECTIVE: Recent advancements in augmented reality led to planning and navigation systems for orthopedic surgery. However little is known about mixed reality (MR) in orthopedics. Furthermore, artificial intelligence (AI) has the potential to boost the capabilities of MR by enabling automation and personalization. The purpose of this work is to assess Holoknee prototype, based on AI and MR for multimodal data visualization and surgical planning in knee osteotomy, developed to run on the HoloLens 2 headset. METHODS: Two preclinical test sessions were performed with 11 participants (eight surgeons, two residents, and one medical student) executing three times six tasks, corresponding to a number of holographic data interactions and preoperative planning steps. At the end of each session, participants answered a questionnaire on user perception and usability. RESULTS: During the second trial, the participants were faster in all tasks than in the first one, while in the third one, the time of execution decreased only for two tasks ("Patient selection" and "Scrolling through radiograph") with respect to the second attempt, but without statistically significant difference (respectively [Formula: see text] = 0.14 and [Formula: see text] = 0.13, [Formula: see text]). All subjects strongly agreed that MR can be used effectively for surgical training, whereas 10 (90.9%) strongly agreed that it can be used effectively for preoperative planning. Six (54.5%) agreed and two of them (18.2%) strongly agreed that it can be used effectively for intraoperative guidance. DISCUSSION/CONCLUSION: In this work, we presented Holoknee, the first holistic application of AI and MR for surgical planning for knee osteotomy. It reported promising results on its potential translation to surgical training, preoperative planning, and surgical guidance. Clinical and Translational Impact Statement - Holoknee can be helpful to support surgeons in the preoperative planning of knee osteotomy. It has the potential to impact positively the training of the future generation of residents and aid surgeons in the intraoperative stage.

Combined Edge Loss UNet for Optimized Segmentation in Total Knee Arthroplasty Preoperative Planning.

Bone segmentation and 3D reconstruction are crucial for total knee arthroplasty (TKA) surgical planning with Personalized Surgical Instruments (PSIs). Traditional semi-automatic approaches are time-consuming and operator-dependent, although they provide reliable outcomes. Moreover, the recent expansion of artificial intelligence (AI) tools towards various medical domains is transforming modern healthcare. Accordingly, this study introduces an automated AI-based pipeline to replace the current operator-based tibia and femur 3D reconstruction procedure enhancing TKA preoperative planning. Leveraging an 822 CT image dataset, a novel patch-based method and an improved segmentation label generation algorithm were coupled to a Combined Edge Loss UNet (CEL-UNet), a novel CNN architecture featuring an additional decoding branch to boost the bone boundary segmentation. Root Mean Squared Errors and Hausdorff distances compared the predicted surfaces to the reference bones showing median and interquartile values of 0.26 (0.19-0.36) mm and 0.24 (0.18-0.32) mm, and of 1.06 (0.73-2.15) mm and 1.43 (0.82-2.86) mm for the tibia and femur, respectively, outperforming previous results of our group, state-of-the-art, and UNet models. A feasibility analysis for a PSI-based surgical plan revealed sub-millimetric distance errors and sub-angular alignment uncertainties in the PSI contact areas and the two cutting planes. Finally, operational environment testing underscored the pipeline's efficiency. More than half of the processed cases complied with the PSI prototyping requirements, reducing the overall time from 35 min to 13.1 s, while the remaining ones underwent a manual refinement step to achieve such PSI requirements, performing the procedure four to eleven times faster than the manufacturer standards. To conclude, this research advocates the need for real-world applicability and optimization of AI solutions in orthopedic surgical practice.

CEL-Unet: a novel CNN architecture for 3D Segmentation of Knee Bones affected by Severe Osteoarthritis for PSI-Based Surgical Planning.

Unet architectures are promising deep learning networks exploited to perform the automatic segmentation of bone CT images, in line with their ability to deal with pathological deformations and size-varying anatomies. However, bone degeneration, like the development of irregular osteophytes as well as mineral density alterations might interfere with this automated process and demand extensive manual refinement. The aim of this work is to implement an innovative Unet variant, the CEL-Unet, to improve the femur and tibia segmentation outcomes in osteoarthritic knee joints. In this network the decoding path is split into a region and contour-aware branch to increase the prediction reliability in such pathological conditions. The comparison between the segmentation results achieved with a standard Unet and its novel variant (CEL-Unet) was performed as follows: the Unet was trained with 5 different loss functions: Dice Loss, Focal Loss, Exponential Logarithmic Loss, Double Cross Entropy Loss and Distanced Cross Entropy loss. The CEL-Unet was instead trained with two loss functions, one for each of the network outputs, namely Mask and Edge, yielding the so-called Combined Edge Loss (CEL) function. A set of 259 knee CT scans was used to train the model and test segmentation performance. The CEL-Unet outperformed all other Unet-based models, reaching the highest Jaccard values of about 0.97 and 0.96 on femur and tibia, respectively. Clinical Relevance- With the increasing rate of Total Knee Arthoplasty deep learning-based methods can achieve fast accurate and automatic 3D segmentation of the knee joint bones to enhance new costumized pre-operative planning.