The SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA) for video-rate functional imaging of the skin.

Spectral imaging has the potential to become a key technique in interventional medicine as it unveils much richer optical information compared to conventional RBG (red, green, and blue)-based imaging. Thus allowing for high-resolution functional tissue analysis in real time. Its higher information density particularly shows promise for the development of powerful perfusion monitoring methods for clinical use. However, even though in vivo validation of such methods is crucial for their clinical translation, the biomedical field suffers from a lack of publicly available datasets for this purpose. Closing this gap, we generated the SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA). It comprises ten spectral videos (∼20 Hz, approx. 20,000 frames each) systematically recorded of the hands of ten healthy human participants in different functional states. We paired each spectral video with concisely tracked regions of interest, and corresponding diffuse reflectance measurements recorded with a spectrometer. Providing the first openly accessible in human spectral video dataset for perfusion monitoring, our work facilitates the development and validation of new functional imaging methods.

Test-time augmentation with synthetic data addresses distribution shifts in spectral imaging.

PURPOSE: Surgical scene segmentation is crucial for providing context-aware surgical assistance. Recent studies highlight the significant advantages of hyperspectral imaging (HSI) over traditional RGB data in enhancing segmentation performance. Nevertheless, the current hyperspectral imaging (HSI) datasets remain limited and do not capture the full range of tissue variations encountered clinically. METHODS: Based on a total of 615 hyperspectral images from a total of 16 pigs, featuring porcine organs in different perfusion states, we carry out an exploration of distribution shifts in spectral imaging caused by perfusion alterations. We further introduce a novel strategy to mitigate such distribution shifts, utilizing synthetic data for test-time augmentation. RESULTS: The effect of perfusion changes on state-of-the-art (SOA) segmentation networks depended on the organ and the specific perfusion alteration induced. In the case of the kidney, we observed a performance decline of up to 93% when applying a state-of-the-art (SOA) network under ischemic conditions. Our method improved on the state-of-the-art (SOA) by up to 4.6 times. CONCLUSION: Given its potential wide-ranging relevance to diverse pathologies, our approach may serve as a pivotal tool to enhance neural network generalization within the realm of spectral imaging.

HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological Organs.

Hyperspectral Imaging (HSI) is a relatively new medical imaging modality that exploits an area of diagnostic potential formerly untouched. Although exploratory translational and clinical studies exist, no surgical HSI datasets are openly accessible to the general scientific community. To address this bottleneck, this publication releases HeiPorSPECTRAL ( https://www.heiporspectral.org ; https://doi.org/10.5281/zenodo.7737674 ), the first annotated high-quality standardized surgical HSI dataset. It comprises 5,758 spectral images acquired with the TIVITA® Tissue and annotated with 20 physiological porcine organs from 8 pigs per organ distributed over a total number of 11 pigs. Each HSI image features a resolution of 480 × 640 pixels acquired over the 500-1000 nm wavelength range. The acquisition protocol has been designed such that the variability of organ spectra as a function of several parameters including the camera angle and the individual can be assessed. A comprehensive technical validation confirmed both the quality of the raw data and the annotations. We envision potential reuse within this dataset, but also its reuse as baseline data for future research questions outside this dataset. Measurement(s) Spectral Reflectance Technology Type(s) Hyperspectral Imaging Sample Characteristic - Organism Sus scrofa.

Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery.

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

ICG-augmented hyperspectral imaging for visualization of intestinal perfusion compared to conventional ICG fluorescence imaging: an experimental study.

BACKGROUND: Small bowel malperfusion (SBM) can cause high morbidity and severe surgical consequences. However, there is no standardized objective measuring tool for the quantification of SBM. Indocyanine green (ICG) imaging can be used for visualization, but lacks standardization and objectivity. Hyperspectral imaging (HSI) as a newly emerging technology in medicine might present advantages over conventional ICG fluorescence or in combination with it. METHODS: HSI baseline data from physiological small bowel, avascular small bowel and small bowel after intravenous application of ICG was recorded in a total number of 54 in-vivo pig models. Visualizations of avascular small bowel after mesotomy were compared between HSI only (1), ICG-augmented HSI (IA-HSI) (2), clinical evaluation through the eyes of the surgeon (3) and conventional ICG imaging (4). The primary research focus was the localization of resection borders as suggested by each of the four methods. Distances between these borders were measured and histological samples were obtained from the regions in between in order to quantify necrotic changes 6 h after mesotomy for every region. RESULTS: StO2 images (1) were capable of visualizing areas of physiological perfusion and areas of clearly impaired perfusion. However, exact borders where physiological perfusion started to decrease could not be clearly identified. Instead, IA-HSI (2) suggested a sharp-resection line where StO2 values started to decrease. Clinical evaluation (3) suggested a resection line 23 mm (±7 mm) and conventional ICG imaging (4) even suggested a resection line 53 mm (±13 mm) closer towards the malperfused region. Histopathological evaluation of the region that was sufficiently perfused only according to conventional ICG (R3) already revealed a significant increase in pre-necrotic changes in 27% (±9%) of surface area. Therefore, conventional ICG seems less sensitive than IA-HSI with regards to detection of insufficient tissue perfusion. CONCLUSIONS: In this experimental animal study, IA-HSI (2) was superior for the visualization of segmental SBM compared to conventional HSI imaging (1), clinical evaluation (3) or conventional ICG imaging (4) regarding histopathological safety. ICG application caused visual artifacts in the StO2 values of the HSI camera as values significantly increase. This is caused by optical properties of systemic ICG and does not resemble a true increase in oxygenation levels. However, this empirical finding can be used to visualize segmental SBM utilizing ICG as contrast agent in an approach for IA-HSI. Clinical applicability and relevance will have to be explored in clinical trials. LEVEL OF EVIDENCE: Not applicable. Translational animal science. Original article.

Deep Learning-based Assessment of Oncologic Outcomes from Natural Language Processing of Structured Radiology Reports.

Purpose: To train a deep natural language processing (NLP) model, using data mined structured oncology reports (SOR), for rapid tumor response category (TRC) classification from free-text oncology reports (FTOR) and to compare its performance with human readers and conventional NLP algorithms. Materials and Methods: In this retrospective study, databases of three independent radiology departments were queried for SOR and FTOR dated from March 2018 to August 2021. An automated data mining and curation pipeline was developed to extract Response Evaluation Criteria in Solid Tumors-related TRCs for SOR for ground truth definition. The deep NLP bidirectional encoder representations from transformers (BERT) model and three feature-rich algorithms were trained on SOR to predict TRCs in FTOR. Models' F1 scores were compared against scores of radiologists, medical students, and radiology technologist students. Lexical and semantic analyses were conducted to investigate human and model performance on FTOR. Results: Oncologic findings and TRCs were accurately mined from 9653 of 12 833 (75.2%) queried SOR, yielding oncology reports from 10 455 patients (mean age, 60 years ± 14 [SD]; 5303 women) who met inclusion criteria. On 802 FTOR in the test set, BERT achieved better TRC classification results (F1, 0.70; 95% CI: 0.68, 0.73) than the best-performing reference linear support vector classifier (F1, 0.63; 95% CI: 0.61, 0.66) and technologist students (F1, 0.65; 95% CI: 0.63, 0.67), had similar performance to medical students (F1, 0.73; 95% CI: 0.72, 0.75), but was inferior to radiologists (F1, 0.79; 95% CI: 0.78, 0.81). Lexical complexity and semantic ambiguities in FTOR influenced human and model performance, revealing maximum F1 score drops of -0.17 and -0.19, respectively. Conclusion: The developed deep NLP model reached the performance level of medical students but not radiologists in curating oncologic outcomes from radiology FTOR.Keywords: Neural Networks, Computer Applications-Detection/Diagnosis, Oncology, Research Design, Staging, Tumor Response, Comparative Studies, Decision Analysis, Experimental Investigations, Observer Performance, Outcomes Analysis Supplemental material is available for this article. © RSNA, 2022.

Robust deep learning-based semantic organ segmentation in hyperspectral images.

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases - consistently across modalities - with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean DSC of 0.90 ((standard deviation (SD)) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 ((standard deviation (SD)) 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine model.

Visual discrimination of tissue during surgery can be challenging since different tissues appear similar to the human eye. Hyperspectral imaging (HSI) removes this limitation by associating each pixel with high-dimensional spectral information. While previous work has shown its general potential to discriminate tissue, clinical translation has been limited due to the method's current lack of robustness and generalizability. Specifically, the scientific community is lacking a comprehensive spectral tissue atlas, and it is unknown whether variability in spectral reflectance is primarily explained by tissue type rather than the recorded individual or specific acquisition conditions. The contribution of this work is threefold: (1) Based on an annotated medical HSI data set (9059 images from 46 pigs), we present a tissue atlas featuring spectral fingerprints of 20 different porcine organs and tissue types. (2) Using the principle of mixed model analysis, we show that the greatest source of variability related to HSI images is the organ under observation. (3) We show that HSI-based fully-automatic tissue differentiation of 20 organ classes with deep neural networks is possible with high accuracy (> 95%). We conclude from our study that automatic tissue discrimination based on HSI data is feasible and could thus aid in intraoperative decisionmaking and pave the way for context-aware computer-assisted surgery systems and autonomous robotics.

Adapting Bidirectional Encoder Representations from Transformers (BERT) to Assess Clinical Semantic Textual Similarity: Algorithm Development and Validation Study.

BACKGROUND: Natural Language Understanding enables automatic extraction of relevant information from clinical text data, which are acquired every day in hospitals. In 2018, the language model Bidirectional Encoder Representations from Transformers (BERT) was introduced, generating new state-of-the-art results on several downstream tasks. The National NLP Clinical Challenges (n2c2) is an initiative that strives to tackle such downstream tasks on domain-specific clinical data. In this paper, we present the results of our participation in the 2019 n2c2 and related work completed thereafter. OBJECTIVE: The objective of this study was to optimally leverage BERT for the task of assessing the semantic textual similarity of clinical text data. METHODS: We used BERT as an initial baseline and analyzed the results, which we used as a starting point to develop 3 different approaches where we (1) added additional, handcrafted sentence similarity features to the classifier token of BERT and combined the results with more features in multiple regression estimators, (2) incorporated a built-in ensembling method, M-Heads, into BERT by duplicating the regression head and applying an adapted training strategy to facilitate the focus of the heads on different input patterns of the medical sentences, and (3) developed a graph-based similarity approach for medications, which allows extrapolating similarities across known entities from the training set. The approaches were evaluated with the Pearson correlation coefficient between the predicted scores and ground truth of the official training and test dataset. RESULTS: We improved the performance of BERT on the test dataset from a Pearson correlation coefficient of 0.859 to 0.883 using a combination of the M-Heads method and the graph-based similarity approach. We also show differences between the test and training dataset and how the two datasets influenced the results. CONCLUSIONS: We found that using a graph-based similarity approach has the potential to extrapolate domain specific knowledge to unseen sentences. We observed that it is easily possible to obtain deceptive results from the test dataset, especially when the distribution of the data samples is different between training and test datasets.