Time-domain feature extraction for target specificity in photoacoustic remote sensing microscopy.

Photoacoustic remote sensing (PARS) microscopy is an emerging label-free optical absorption imaging modality. PARS operates by capturing nanosecond-scale optical fluctuations produced by photoacoustic pressures. These time-domain (TD) variations are usually projected by amplitude to determine optical absorption magnitude. However, valuable details on a target's material properties (e.g., density, speed of sound) are contained within the TD signals. This work uses a novel, to the best of our knowledge, clustering method to learn TD features, based on signal shape, which relate to underlying material traits. A modified K-means method is used to cluster TD data, capturing representative signal features. These features are then used to form virtual colorizations which may highlight tissues based on their underlying material properties. Applied in fresh resected murine brain tissue, colorized visualizations highlight distinct regions of tissue. This may potentially facilitate differentiation of tissue constituents (e.g., myelinated and unmyelinated axons, cell nuclei) in a single acquisition.

Functional photoacoustic remote sensing microscopy using a stabilized temperature-regulated stimulated Raman scattering light source.

Stimulated Raman scattering (SRS) has been widely used in functional photoacoustic microscopy to generate multiwavelength light and target multiple chromophores inside tissues. Despite offering a simple, cost-effective technique with a high pulse repetition rate; it suffers from pulse-to-pulse intensity fluctuations and power drift that can affect image quality. Here, we propose a new technique to improve the temporal stability of the pulsed SRS multiwavelength source. We achieve this by lowering the temperature of the SRS medium. The results suggest that a decrease in temperature causes an improvement of temporal stability of the output, considerable rise in the intensity of the SRS peaks, and significant increase of SRS cross section. The application of the method is shown for in vivo functional imaging of capillary networks in a chicken embryo chorioallantois membrane using photoacoustic remote sensing microscopy.

A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms.

BACKGROUND: In classification and diagnostic testing, the receiver-operator characteristic (ROC) plot and the area under the ROC curve (AUC) describe how an adjustable threshold causes changes in two types of error: false positives and false negatives. Only part of the ROC curve and AUC are informative however when they are used with imbalanced data. Hence, alternatives to the AUC have been proposed, such as the partial AUC and the area under the precision-recall curve. However, these alternatives cannot be as fully interpreted as the AUC, in part because they ignore some information about actual negatives. METHODS: We derive and propose a new concordant partial AUC and a new partial c statistic for ROC data-as foundational measures and methods to help understand and explain parts of the ROC plot and AUC. Our partial measures are continuous and discrete versions of the same measure, are derived from the AUC and c statistic respectively, are validated as equal to each other, and validated as equal in summation to whole measures where expected. Our partial measures are tested for validity on a classic ROC example from Fawcett, a variation thereof, and two real-life benchmark data sets in breast cancer: the Wisconsin and Ljubljana data sets. Interpretation of an example is then provided. RESULTS: Results show the expected equalities between our new partial measures and the existing whole measures. The example interpretation illustrates the need for our newly derived partial measures. CONCLUSIONS: The concordant partial area under the ROC curve was proposed and unlike previous partial measure alternatives, it maintains the characteristics of the AUC. The first partial c statistic for ROC plots was also proposed as an unbiased interpretation for part of an ROC curve. The expected equalities among and between our newly derived partial measures and their existing full measure counterparts are confirmed. These measures may be used with any data set but this paper focuses on imbalanced data with low prevalence. FUTURE WORK: Future work with our proposed measures may: demonstrate their value for imbalanced data with high prevalence, compare them to other measures not based on areas; and combine them with other ROC measures and techniques.

Homotopic non-local regularized reconstruction from sparse positron emission tomography measurements.

BACKGROUND: Positron emission tomography scanners collect measurements of a patient's in vivo radiotracer distribution. The system detects pairs of gamma rays emitted indirectly by a positron-emitting radionuclide (tracer), which is introduced into the body on a biologically active molecule, and the tomograms must be reconstructed from projections. The reconstruction of tomograms from the acquired PET data is an inverse problem that requires regularization. The use of tightly packed discrete detector rings, although improves signal-to-noise ratio, are often associated with high costs of positron emission tomography systems. Thus a sparse reconstruction, which would be capable of overcoming the noise effect while allowing for a reduced number of detectors, would have a great deal to offer. METHODS: In this study, we introduce and investigate the potential of a homotopic non-local regularization reconstruction framework for effectively reconstructing positron emission tomograms from such sparse measurements. RESULTS: Results obtained using the proposed approach are compared with traditional filtered back-projection as well as expectation maximization reconstruction with total variation regularization. CONCLUSIONS: A new reconstruction method was developed for the purpose of improving the quality of positron emission tomography reconstruction from sparse measurements. We illustrate that promising reconstruction performance can be achieved for the proposed approach even at low sampling fractions, which allows for the use of significantly fewer detectors and have the potential to reduce scanner costs.

Noise-compensated homotopic non-local regularized reconstruction for rapid retinal optical coherence tomography image acquisitions.

BACKGROUND: Optical coherence tomography (OCT) is a minimally invasive imaging technique, which utilizes the spatial and temporal coherence properties of optical waves backscattered from biological material. Recent advances in tunable lasers and infrared camera technologies have enabled an increase in the OCT imaging speed by a factor of more than 100, which is important for retinal imaging where we wish to study fast physiological processes in the biological tissue. However, the high scanning rate causes proportional decrease of the detector exposure time, resulting in a reduction of the system signal-to-noise ratio (SNR). One approach to improving the image quality of OCT tomograms acquired at high speed is to compensate for the noise component in the images without compromising the sharpness of the image details. METHODS: In this study, we propose a novel reconstruction method for rapid OCT image acquisitions, based on a noise-compensated homotopic modified James-Stein non-local regularized optimization strategy. The performance of the algorithm was tested on a series of high resolution OCT images of the human retina acquired at different imaging rates. RESULTS: Quantitative analysis was used to evaluate the performance of the algorithm using two state-of-art denoising strategies. Results demonstrate significant SNR improvements when using our proposed approach when compared to other approaches. CONCLUSIONS: A new reconstruction method based on a noise-compensated homotopic modified James-Stein non-local regularized optimization strategy was developed for the purpose of improving the quality of rapid OCT image acquisitions. Preliminary results show the proposed method shows considerable promise as a tool to improve the visualization and analysis of biological material using OCT.

Multi-channel feature extraction for virtual histological staining of photon absorption remote sensing images.

Accurate and fast histological staining is crucial in histopathology, impacting diagnostic precision and reliability. Traditional staining methods are time-consuming and subjective, causing delays in diagnosis. Digital pathology plays a vital role in advancing and optimizing histology processes to improve efficiency and reduce turnaround times. This study introduces a novel deep learning-based framework for virtual histological staining using photon absorption remote sensing (PARS) images. By extracting features from PARS time-resolved signals using a variant of the K-means method, valuable multi-modal information is captured. The proposed multi-channel cycleGAN model expands on the traditional cycleGAN framework, allowing the inclusion of additional features. Experimental results reveal that specific combinations of features outperform the conventional channels by improving the labeling of tissue structures prior to model training. Applied to human skin and mouse brain tissue, the results underscore the significance of choosing the optimal combination of features, as it reveals a substantial visual and quantitative concurrence between the virtually stained and the gold standard chemically stained hematoxylin and eosin images, surpassing the performance of other feature combinations. Accurate virtual staining is valuable for reliable diagnostic information, aiding pathologists in disease classification, grading, and treatment planning. This study aims to advance label-free histological imaging and opens doors for intraoperative microscopy applications.

Energy-guided learning approach to compressive FD-OCT.

High quality, large size volumetric imaging of biological tissue with optical coherence tomography (OCT) requires large number and high density of scans, which results in large data acquisition volume. This may lead to corruption of the data with motion artifacts related to natural motion of biological tissue, and could potentially cause conflicts with the maximum permissible exposure of biological tissue to optical radiation. Therefore, OCT can benefit greatly from different approaches to sparse or compressive sampling of the data where the signal is recovered from its sub-Nyquist measurements. In this paper, a new energy-guided compressive sensing approach is proposed for improving the quality of images acquired with Fourier domain OCT (FD-OCT) and reconstructed from sparse data sets. The proposed algorithm learns an optimized sampling probability density function based on the energy distribution of the training data set, which is then used for sparse sampling instead of the commonly used uniformly random sampling. It was demonstrated that the proposed energy-guided learning approach to compressive FD-OCT of retina images requires 45% fewer samples in comparison with the conventional uniform compressive sensing (CS) approach while achieving similar reconstruction performance. This novel approach to sparse sampling has the potential to significantly reduce data acquisition while maintaining image quality.

Deep Learning for Instance Retrieval: A Survey.

In recent years a vast amount of visual content has been generated and shared from many fields, such as social media platforms, medical imaging, and robotics. This abundance of content creation and sharing has introduced new challenges, particularly that of searching databases for similar content - Content Based Image Retrieval (CBIR) - a long-established research area in which improved efficiency and accuracy are needed for real-time retrieval. Artificial intelligence has made progress in CBIR and has significantly facilitated the process of instance search. In this survey we review recent instance retrieval works that are developed based on deep learning algorithms and techniques, with the survey organized by deep feature extraction, feature embedding and aggregation methods, and network fine-tuning strategies. Our survey considers a wide variety of recent methods, whereby we identify milestone work, reveal connections among various methods and present the commonly used benchmarks, evaluation results, common challenges, and propose promising future directions.

Deep ROC Analysis and AUC as Balanced Average Accuracy, for Improved Classifier Selection, Audit and Explanation.

Optimal performance is desired for decision-making in any field with binary classifiers and diagnostic tests, however common performance measures lack depth in information. The area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve are too general because they evaluate all decision thresholds including unrealistic ones. Conversely, accuracy, sensitivity, specificity, positive predictive value and the F1 score are too specific-they are measured at a single threshold that is optimal for some instances, but not others, which is not equitable. In between both approaches, we propose deep ROC analysis to measure performance in multiple groups of predicted risk (like calibration), or groups of true positive rate or false positive rate. In each group, we measure the group AUC (properly), normalized group AUC, and averages of: sensitivity, specificity, positive and negative predictive value, and likelihood ratio positive and negative. The measurements can be compared between groups, to whole measures, to point measures and between models. We also provide a new interpretation of AUC in whole or part, as balanced average accuracy, relevant to individuals instead of pairs. We evaluate models in three case studies using our method and Python toolkit and confirm its utility.

Homotopic, non-local sparse reconstruction of optical coherence tomography imagery.

The resolution in optical coherence tomography imaging is an important parameter which determines the size of the smallest features that can be visualized. Sparse sampling approaches have shown considerable promise in producing high resolution OCT images with fewer camera pixels, reducing both the cost and the complexity of an imaging system. In this paper, we propose a non-local approach to the reconstruction of high resolution OCT images from sparsely sampled measurements. An iterative strategy is introduced for minimizing a homotopic, non-local regularized functional in the spatial domain, subject to data fidelity constraints in the k-space domain. The novel algorithm was tested on human retinal, corneal, and limbus images, acquired in-vivo, demonstrating the effectiveness of the proposed approach in generating high resolution reconstructions from a limited number of camera pixels.

Deep learning methods for inverse problems.

In this paper we investigate a variety of deep learning strategies for solving inverse problems. We classify existing deep learning solutions for inverse problems into three categories of Direct Mapping, Data Consistency Optimizer, and Deep Regularizer. We choose a sample of each inverse problem type, so as to compare the robustness of the three categories, and report a statistical analysis of their differences. We perform extensive experiments on the classic problem of linear regression and three well-known inverse problems in computer vision, namely image denoising, 3D human face inverse rendering, and object tracking, in presence of noise and outliers, are selected as representative prototypes for each class of inverse problems. The overall results and the statistical analyses show that the solution categories have a robustness behaviour dependent on the type of inverse problem domain, and specifically dependent on whether or not the problem includes measurement outliers. Based on our experimental results, we conclude by proposing the most robust solution category for each inverse problem class.

Virtual histological staining of label-free total absorption photoacoustic remote sensing (TA-PARS).

Histopathological visualizations are a pillar of modern medicine and biological research. Surgical oncology relies exclusively on post-operative histology to determine definitive surgical success and guide adjuvant treatments. The current histology workflow is based on bright-field microscopic assessment of histochemical stained tissues and has some major limitations. For example, the preparation of stained specimens for brightfield assessment requires lengthy sample processing, delaying interventions for days or even weeks. Therefore, there is a pressing need for improved histopathology methods. In this paper, we present a deep-learning-based approach for virtual label-free histochemical staining of total-absorption photoacoustic remote sensing (TA-PARS) images of unstained tissue. TA-PARS provides an array of directly measured label-free contrasts such as scattering and total absorption (radiative and non-radiative), ideal for developing H&E colorizations without the need to infer arbitrary tissue structures. We use a Pix2Pix generative adversarial network to develop visualizations analogous to H&E staining from label-free TA-PARS images. Thin sections of human skin tissue were first virtually stained with the TA-PARS, then were chemically stained with H&E producing a one-to-one comparison between the virtual and chemical staining. The one-to-one matched virtually- and chemically- stained images exhibit high concordance validating the digital colorization of the TA-PARS images against the gold standard H&E. TA-PARS images were reviewed by four dermatologic pathologists who confirmed they are of diagnostic quality, and that resolution, contrast, and color permitted interpretation as if they were H&E. The presented approach paves the way for the development of TA-PARS slide-free histological imaging, which promises to dramatically reduce the time from specimen resection to histological imaging.

In-vivo functional and structural retinal imaging using multiwavelength photoacoustic remote sensing microscopy.

Many important eye diseases as well as systemic disorders manifest themselves in the retina. Retinal imaging technologies are rapidly growing and can provide ever-increasing amounts of information about the structure, function, and molecular composition of retinal tissue in-vivo. Photoacoustic remote sensing (PARS) is a novel imaging modality based on all-optical detection of photoacoustic signals, which makes it suitable for a wide range of medical applications. In this study, PARS is applied for in-vivo imaging of the retina and estimating oxygen saturation in the retinal vasculature. To our knowledge, this is the first time that a non-contact photoacoustic imaging technique is applied for in-vivo imaging of the retina. Here, optical coherence tomography is also used as a well-established retinal imaging technique to navigate the PARS imaging beams and demonstrate the capabilities of the optical imaging setup. The system is applied for in-vivo imaging of both microanatomy and the microvasculature of the retina. The developed system has the potential to advance the understanding of the ocular environment and to help in monitoring of ophthalmic diseases.

Constrained watershed method to infer morphology of mammalian cells in microscopic images.

Precise information about the size, shape, temporal dynamics, and spatial distribution of cells is beneficial for the understanding of cell behavior and may play a key role in drug development, regenerative medicine, and disease research. The traditional method of manual observation and measurement of cells from microscopic images is tedious, expensive, and time consuming. Thus, automated methods are in high demand, especially given the increasing quantity of cell data being collected. In this article, an automated method to measure cell morphology from microscopic images is proposed to outline the boundaries of individual hematopoietic stem cells (HSCs). The proposed method outlines the cell regions using a constrained watershed method which is derived as an inverse problem. The experimental results generated by applying the proposed method to different HSC image sequences showed robust performance to detect and segment individual and dividing cells. The performance of the proposed method for individual cell segmentation for single frame high-resolution images was more than 97%, and decreased slightly to 90% for low-resolution multiframe stitched images.

A probabilistic cell model in background corrected image sequences for single cell analysis.

BACKGROUND: Methods of manual cell localization and outlining are so onerous that automated tracking methods would seem mandatory for handling huge image sequences, nevertheless manual tracking is, astonishingly, still widely practiced in areas such as cell biology which are outside the influence of most image processing research. The goal of our research is to address this gap by developing automated methods of cell tracking, localization, and segmentation. Since even an optimal frame-to-frame association method cannot compensate and recover from poor detection, it is clear that the quality of cell tracking depends on the quality of cell detection within each frame. METHODS: Cell detection performs poorly where the background is not uniform and includes temporal illumination variations, spatial non-uniformities, and stationary objects such as well boundaries (which confine the cells under study). To improve cell detection, the signal to noise ratio of the input image can be increased via accurate background estimation. In this paper we investigate background estimation, for the purpose of cell detection. We propose a cell model and a method for background estimation, driven by the proposed cell model, such that well structure can be identified, and explicitly rejected, when estimating the background. RESULTS: The resulting background-removed images have fewer artifacts and allow cells to be localized and detected more reliably. The experimental results generated by applying the proposed method to different Hematopoietic Stem Cell (HSC) image sequences are quite promising. CONCLUSION: The understanding of cell behavior relies on precise information about the temporal dynamics and spatial distribution of cells. Such information may play a key role in disease research and regenerative medicine, so automated methods for observation and measurement of cells from microscopic images are in high demand. The proposed method in this paper is capable of localizing single cells in microwells and can be adapted for the other cell types that may not have circular shape. This method can be potentially used for single cell analysis to study the temporal dynamics of cells.

Improving maximal safe brain tumor resection with photoacoustic remote sensing microscopy.

Malignant brain tumors are among the deadliest neoplasms with the lowest survival rates of any cancer type. In considering surgical tumor resection, suboptimal extent of resection is linked to poor clinical outcomes and lower overall survival rates. Currently available tools for intraoperative histopathological assessment require an average of 20 min processing and are of limited diagnostic quality for guiding surgeries. Consequently, there is an unaddressed need for a rapid imaging technique to guide maximal resection of brain tumors. Working towards this goal, presented here is an all optical non-contact label-free reflection mode photoacoustic remote sensing (PARS) microscope. By using a tunable excitation laser, PARS takes advantage of the endogenous optical absorption peaks of DNA and cytoplasm to achieve virtual contrast analogous to standard hematoxylin and eosin (H&E) staining. In conjunction, a fast 266 nm excitation is used to generate large grossing scans and rapidly assess small fields in real-time with hematoxylin-like contrast. Images obtained using this technique show comparable quality and contrast to the current standard for histopathological assessment of brain tissues. Using the proposed method, rapid, high-throughput, histological-like imaging was achieved in unstained brain tissues, indicating PARS' utility for intraoperative guidance to improve extent of surgical resection.

Texture Classification in Extreme Scale Variations Using GANet.

Research in texture recognition often concentrates on recognizing textures with intraclass variations, such as illumination, rotation, viewpoint, and small-scale changes. In contrast, in real-world applications, a change in scale can have a dramatic impact on texture appearance to the point of changing completely from one texture category to another. As a result, texture variations due to changes in scale are among the hardest to handle. In this paper, we conduct the first study of classifying textures with extreme variations in scale. To address this issue, we first propose and then reduce scale proposals on the basis of dominant texture patterns. Motivated by the challenges posed by this problem, we propose a new GANet network where we use a genetic algorithm to change the filters in the hidden layers during network training in order to promote the learning of more informative semantic texture patterns. Finally, we adopt a Fisher vector pooling of a convolutional neural network filter bank feature encoder for global texture representation. Because extreme scale variations are not necessarily present in most standard texture databases, to support the proposed extreme-scale aspects of texture understanding, we are developing a new dataset, the extreme scale variation textures (ESVaT), to test the performance of our framework. It is demonstrated that the proposed framework significantly outperforms the gold-standard texture features by more than 10% on ESVaT. We also test the performance of our proposed approach on the KTHTIPS2b and OS datasets and a further dataset synthetically derived from Forrest, showing the superior performance compared with the state-of-the-art.

Extended-Hungarian-JPDA: exact single-frame stem cell tracking.

The fields of bioinformatics and biotechnology rely on the collection, processing and analysis of huge numbers of biocellular images, including cell features such as cell size, shape, and motility. Thus, cell tracking is of crucial importance in the study of cell behaviour and in drug and disease research. Such a multitarget tracking is essentially an assignment problem, NP-hard, with the solution normally found in practice in a reduced hypothesis space. In this paper we introduce a novel approach to find the exact association solution over time for single-frame scan-back stem cell tracking. Our proposed method employs a class of linear programming optimization methods known as the Hungarian method to find the optimal joint probabilistic data association for nonlinear dynamics and non-Gaussian measurements. The proposed method, an optimal joint probabilistic data association approach, has been successfully applied to track hematopoietic stem cells.

Sorted self-similarity for multi-modal image registration.

In medical image analysis, registration of multimodal images has been challenging due to the complex intensity relationship between images. Classical multi-modal registration approaches evaluate the degree of the alignment by measuring the statistical dependency of the intensity values between images to be aligned. Employing statistical similarity measures, such as mutual information, is not promising in those cases with complex and spatially dependent intensity relations. A new similarity measure is proposed based on the assessing the similarity of pixels within an image, based on the idea that similar structures in an image are more probable to undergo similar intensity transformations. The most significant pixel similarity values are considered to transmit the most significant self-similarity information. The proposed method is employed in a framework to register different modalities of real brain scans and the performance of the method is compared to the conventional multi-modal registration approach. Quantitative evaluation of the method demonstrates the better registration accuracy in both rigid and non-rigid deformations.