Shape-Based Measures Improve Scene Categorization.

Converging evidence indicates that deep neural network models that are trained on large datasets are biased toward color and texture information. Humans, on the other hand, can easily recognize objects and scenes from images as well as from bounding contours. Mid-level vision is characterized by the recombination and organization of simple primary features into more complex ones by a set of so-called Gestalt grouping rules. While described qualitatively in the human literature, a computational implementation of these perceptual grouping rules is so far missing. In this article, we contribute a novel set of algorithms for the detection of contour-based cues in complex scenes. We use the medial axis transform (MAT) to locally score contours according to these grouping rules. We demonstrate the benefit of these cues for scene categorization in two ways: (i) Both human observers and CNN models categorize scenes most accurately when perceptual grouping information is emphasized. (ii) Weighting the contours with these measures boosts performance of a CNN model significantly compared to the use of unweighted contours. Our work suggests that, even though these measures are computed directly from contours in the image, current CNN models do not appear to extract or utilize these grouping cues.

Memorability of line drawings of scenes: the role of contour properties.

Why are some images more likely to be remembered than others? Previous work focused on the influence of global, low-level visual features as well as image content on memorability. To better understand the role of local, shape-based contours, we here investigate the memorability of photographs and line drawings of scenes. We find that the memorability of photographs and line drawings of the same scenes is correlated. We quantitatively measure the role of contour properties and their spatial relationships for scene memorability using a Random Forest analysis. To determine whether this relationship is merely correlational or if manipulating these contour properties causes images to be remembered better or worse, we split each line drawing into two half-images, one with high and the other with low predicted memorability according to the trained Random Forest model. In a new memorability experiment, we find that the half-images predicted to be more memorable were indeed remembered better, confirming a causal role of shape-based contour features, and, in particular, T junctions in scene memorability. We performed a categorization experiment on half-images to test for differential access to scene content. We found that half-images predicted to be more memorable were categorized more accurately. However, categorization accuracy for individual images was not correlated with their memorability. These results demonstrate that we can measure the contributions of individual contour properties to scene memorability and verify their causal involvement with targeted image manipulations, thereby bridging the gap between low-level features and scene semantics in our understanding of memorability.

Anger is red, sadness is blue: Emotion depictions in abstract visual art by artists and non-artists.

Through the manipulation of color and form, visual abstract art is often used to convey feelings and emotions. Here, we explored how colors and lines are used to express basic emotions and whether non-artists express emotions through art in similar ways as trained artists. Both artists and non-artists created abstract color drawings and line drawings depicting six emotions (i.e., anger, disgust, fear, joy, sadness, and wonder). To test whether people represented basic emotions in similar ways, we computationally predicted the emotion of a given drawing by comparing it to a set of references created by averaging across all other participants' drawings within each emotion category. We found that prediction accuracy was higher for color drawings than line drawings and higher for color drawings by non-artists than by artists. In a behavioral experiment, we found that people (N = 242) could also accurately infer emotions, showing the same pattern of results as our computational predictions. Further computational analyses of the drawings revealed systematic use of certain colors and line features to depict each basic emotion (e.g., anger is generally redder and more densely drawn than other emotions, sadness is more blue and contains more vertical lines). Taken together, these results imply that abstract color and line drawings are able to convey certain emotions based on their visual features, which are also used by human observers to understand the intended emotional connotation of abstract artworks.

Single-cell spatial landscape of immunotherapy response reveals mechanisms of CXCL13 enhanced antitumor immunity.

BACKGROUND: Immunotherapy has revolutionized clinical outcomes for patients suffering from lung cancer, yet relatively few patients sustain long-term durable responses. Recent studies have demonstrated that the tumor immune microenvironment fosters tumorous heterogeneity and mediates both disease progression and response to immune checkpoint inhibitors (ICI). As such, there is an unmet need to elucidate the spatially defined single-cell landscape of the lung cancer microenvironment to understand the mechanisms of disease progression and identify biomarkers of response to ICI. METHODS: Here, in this study, we applied imaging mass cytometry to characterize the tumor and immunological landscape of immunotherapy response in non-small cell lung cancer by describing activated cell states, cellular interactions and neighborhoods associated with improved efficacy. We functionally validated our findings using preclinical mouse models of cancer treated with anti-programmed cell death protein-1 (PD-1) immune checkpoint blockade. RESULTS: We resolved 114,524 single cells in 27 patients treated with ICI, enabling spatial resolution of immune lineages and activation states with distinct clinical outcomes. We demonstrated that CXCL13 expression is associated with ICI efficacy in patients, and that recombinant CXCL13 potentiates anti-PD-1 response in vivo in association with increased antigen experienced T cell subsets and reduced CCR2+ monocytes. DISCUSSION: Our results provide a high-resolution molecular resource and illustrate the importance of major immune lineages as well as their functional substates in understanding the role of the tumor immune microenvironment in response to ICIs.

Single-cell spatial landscapes of the lung tumour immune microenvironment.

Single-cell technologies have revealed the complexity of the tumour immune microenvironment with unparalleled resolution1-9. Most clinical strategies rely on histopathological stratification of tumour subtypes, yet the spatial context of single-cell phenotypes within these stratified subgroups is poorly understood. Here we apply imaging mass cytometry to characterize the tumour and immunological landscape of samples from 416 patients with lung adenocarcinoma across five histological patterns. We resolve more than 1.6 million cells, enabling spatial analysis of immune lineages and activation states with distinct clinical correlates, including survival. Using deep learning, we can predict with high accuracy those patients who will progress after surgery using a single 1-mm2 tumour core, which could be informative for clinical management following surgical resection. Our dataset represents a valuable resource for the non-small cell lung cancer research community and exemplifies the utility of spatial resolution within single-cell analyses. This study also highlights how artificial intelligence can improve our understanding of microenvironmental features that underlie cancer progression and may influence future clinical practice.

Single-cell spatial immune landscapes of primary and metastatic brain tumours.

Single-cell technologies have enabled the characterization of the tumour microenvironment at unprecedented depth and have revealed vast cellular diversity among tumour cells and their niche. Anti-tumour immunity relies on cell-cell relationships within the tumour microenvironment1,2, yet many single-cell studies lack spatial context and rely on dissociated tissues3. Here we applied imaging mass cytometry to characterize the immunological landscape of 139 high-grade glioma and 46 brain metastasis tumours from patients. Single-cell analysis of more than 1.1 million cells across 389 high-dimensional histopathology images enabled the spatial resolution of immune lineages and activation states, revealing differences in immune landscapes between primary tumours and brain metastases from diverse solid cancers. These analyses revealed cellular neighbourhoods associated with survival in patients with glioblastoma, which we leveraged to identify a unique population of myeloperoxidase (MPO)-positive macrophages associated with long-term survival. Our findings provide insight into the biology of primary and metastatic brain tumours, reinforcing the value of integrating spatial resolution to single-cell datasets to dissect the microenvironmental contexture of cancer.

Improved Detection of Chronic Obstructive Pulmonary Disease at Chest CT Using the Mean Curvature of Isophotes.

PURPOSE: To determine if the mean curvature of isophotes (MCI), a standard computer vision technique, can be used to improve detection of chronic obstructive pulmonary disease (COPD) at chest CT. MATERIALS AND METHODS: In this retrospective study, chest CT scans were obtained in 243 patients with COPD and 31 controls (among all 274: 151 women [mean age, 70 years; range, 44-90 years] and 123 men [mean age, 71 years; range, 29-90 years]) from two community practices between 2006 and 2019. A convolutional neural network (CNN) architecture was trained on either CT images or CT images transformed through the MCI algorithm. Separately, a linear classification based on a single feature derived from the MCI computation (called hMCI1) was also evaluated. All three models were evaluated with cross-validation, using precision-macro and recall-macro metrics, that is, the mean of per-class precision and recall values, respectively (the latter being equivalent to balanced accuracy). RESULTS: Linear classification based on hMCI1 resulted in a higher recall-macro relative to the CNN trained and applied on CT images (0.85 [95% CI: 0.84, 0.86] vs 0.77 [95% CI: 0.75, 0.79]) but with a similar reduction in precision-macro (0.66 [95% CI: 0.65, 0.67] vs 0.77 [95% CI: 0.75, 0.79]). The CNN model trained and applied on MCI-transformed images had a higher recall-macro (0.85 [95% CI: 0.83, 0.87] vs 0.77 [95% CI: 0.75, 0.79]) and precision-macro (0.85 [95% CI: 0.83, 0.87] vs 0.77 [95% CI: 0.75, 0.79]) relative to the CNN trained and applied on CT images. CONCLUSION: The MCI algorithm may be valuable toward the automated detection and diagnosis of COPD on chest CT scans as part of a CNN-based pipeline or with stand-alone features.Keywords: Chronic Obstructive Pulmonary Disease, Quantification, Lung, CT Supplemental material is available for this article. See also the invited commentary by Vannier in this issue.© RSNA, 2021.

Neural correlates of local parallelism during naturalistic vision.

Human observers can rapidly perceive complex real-world scenes. Grouping visual elements into meaningful units is an integral part of this process. Yet, so far, the neural underpinnings of perceptual grouping have only been studied with simple lab stimuli. We here uncover the neural mechanisms of one important perceptual grouping cue, local parallelism. Using a new, image-computable algorithm for detecting local symmetry in line drawings and photographs, we manipulated the local parallelism content of real-world scenes. We decoded scene categories from patterns of brain activity obtained via functional magnetic resonance imaging (fMRI) in 38 human observers while they viewed the manipulated scenes. Decoding was significantly more accurate for scenes containing strong local parallelism compared to weak local parallelism in the parahippocampal place area (PPA), indicating a central role of parallelism in scene perception. To investigate the origin of the parallelism signal we performed a model-based fMRI analysis of the public BOLD5000 dataset, looking for voxels whose activation time course matches that of the locally parallel content of the 4916 photographs viewed by the participants in the experiment. We found a strong relationship with average local symmetry in visual areas V1-4, PPA, and retrosplenial cortex (RSC). Notably, the parallelism-related signal peaked first in V4, suggesting V4 as the site for extracting paralleism from the visual input. We conclude that local parallelism is a perceptual grouping cue that influences neuronal activity throughout the visual hierarchy, presumably starting at V4. Parallelism plays a key role in the representation of scene categories in PPA.

Neutrophil oxidative stress mediates obesity-associated vascular dysfunction and metastatic transmigration.

Metastasis is the leading cause of cancer-related deaths, and obesity is associated with increased breast cancer (BC) metastasis. Preclinical studies have shown that obese adipose tissue induces lung neutrophilia associated with enhanced BC metastasis to this site. Here we show that obesity leads to neutrophil-dependent impairment of vascular integrity through loss of endothelial adhesions, enabling cancer cell extravasation into the lung. Mechanistically, neutrophil-produced reactive oxygen species in obese mice increase neutrophil extracellular DNA traps (NETs) and weaken endothelial junctions, facilitating the influx of tumor cells from the peripheral circulation. In vivo treatment with catalase, NET inhibitors or genetic deletion of Nos2 reversed this effect in preclinical models of obesity. Imaging mass cytometry of lung metastasis samples from patients with cancer revealed an enrichment in neutrophils with low catalase levels correlating with elevated body mass index. Our data provide insights into potentially targetable mechanisms that underlie the progression of BC in the obese population.

Local contour symmetry facilitates scene categorization.

People are able to rapidly categorize briefly flashed images of real-world environments, even when they are reduced to line drawings. This setting allows for the study of time-limited perceptual grouping processes in the human visual system that are applicable to line drawings. Previous work (Wilder, Dickinson, Jepson, & Walther, 2018) showed that standard local features of individual contours, or junctions between contours, do not account for this rapid classification ability but, rather, the relative placement of these contours appeared to be important. Here we provide strong support for this observation by demonstrating that local ribbon symmetry between neighboring pairs of contours facilitates the categorization of complex real-world environments. To this end, we introduce a novel computational approach, based on the medial axis transform, for measuring the degree of local ribbon symmetry in a line drawing. We use this measure to separate the contour pixels for a given scene into the most ribbon symmetric half and the least ribbon symmetric half. We then show human observers the resulting half-images in a rapid-categorization experiment. Our results demonstrate that local ribbon symmetry facilitates the categorization of complex real-world environments. This is the first study of the role of local symmetry in inter-contour grouping for human scene classification. We conclude that local ribbon symmetry appears to play an important role in jump-starting the grouping of image content into meaningful units, even in flashed presentations.

Subject-Specific Longitudinal Shape Analysis by Coupling Spatiotemporal Shape Modeling with Medial Analysis.

Modeling subject-specific shape change is one of the most important challenges in longitudinal shape analysis of disease progression. Whereas anatomical change over time can be a function of normal aging; anatomy can also be impacted by disease related degeneration. Shape changes to anatomy may also be affected by external structural changes from neighboring structures, which may cause non-linear pose variations. In this paper, we propose a framework to analyze disease related shape changes by coupling extrinsic modeling of the ambient anatomical space via spatiotemporal deformations with intrinsic shape properties from medial surface analysis. We compare intrinsic shape properties of a subject-specific shape trajectory to a normative 4D shape atlas representing normal aging to separately quantify shape changes related to disease. The spatiotemporal shape modeling establishes inter/intra subject anatomical correspondence, which in turn enables comparisons between subjects and the 4D shape atlas, and also quantitative analysis of disease related shape change. The medial surface analysis captures intrinsic shape properties related to local patterns of deformation. The proposed framework simultaneously models extrinsic longitudinal shape changes in the ambient anatomical space, as well as intrinsic shape properties to give localized measurements of degeneration. Six high risk subjects and six controls are randomly sampled from a Huntington's disease image database for quantitative and qualitative comparison.