Protecting world leaders against deep fakes using facial, gestural, and vocal mannerisms.

Since their emergence a few years ago, artificial intelligence (AI)-synthesized media-so-called deep fakes-have dramatically increased in quality, sophistication, and ease of generation. Deep fakes have been weaponized for use in nonconsensual pornography, large-scale fraud, and disinformation campaigns. Of particular concern is how deep fakes will be weaponized against world leaders during election cycles or times of armed conflict. We describe an identity-based approach for protecting world leaders from deep-fake imposters. Trained on several hours of authentic video, this approach captures distinct facial, gestural, and vocal mannerisms that we show can distinguish a world leader from an impersonator or deep-fake imposter.

AI-synthesized faces are indistinguishable from real faces and more trustworthy.

Artificial intelligence (AI)-synthesized text, audio, image, and video are being weaponized for the purposes of nonconsensual intimate imagery, financial fraud, and disinformation campaigns. Our evaluation of the photorealism of AI-synthesized faces indicates that synthesis engines have passed through the uncanny valley and are capable of creating faces that are indistinguishable-and more trustworthy-than real faces.

Assessing the reliability of a clothing-based forensic identification.

A 2009 report by the National Academy of Sciences was highly critical of many forensic practices. This report concluded that significant changes and advances were required to ensure the reliability across the forensic sciences. We examine the reliability of one such forensic technique used for identification based on purported distinct patterns on the seams of denim pants. Although first proposed more than 20 years ago, no thorough analysis of reliability or reproducibility of this forensic technique has previously been reported. We performed a detailed analysis of this forensic technique to determine its reliability and efficacy.

The art of the float.

For more than 2000 years, artists have exploited cast shadows to influence how objects appear to be positioned in a scene. A contact cast shadow can anchor an object to the ground and a detached cast shadow can make an object appear to float. However, there is a period of approximately 1000 years when there were virtually no cast shadows in art. How were states of contact versus floating depicted by artists without cast shadows? Here, we survey various techniques used by artists to anchor relative position with and without cast shadows. We then conduct experimental tests of the hypothesized surface attraction principles that underlie these techniques. In the absence of cast shadows, an object (a wooden box) was often seen as resting on a surface as long as that surface offered information about ground orientation and support (a tiled floor). When the ground surface was ambiguous and cloud-like (1/f noise), the box was more likely to be seen to float. The presence of cast shadows made the box appear to contact the ground whether it was well-defined or ambiguous. Both shadows and surface support also increased the accuracy with which participants detected when the box was tilted up from the ground. These results indicate that artists long ago discovered the important power of support relationships to anchor objects to surfaces in the absence of shadows.

Perceptual and computational detection of face morphing.

A relatively new type of identity theft uses morphed facial images in identification documents in which images of two individuals are digitally blended to create an image that maintains a likeness to each of the original identities. We created a set of high-quality digital morphs from passport-style photos for a diverse set of people across gender, race, and age. We then examine people's ability to detect facial morphing both in terms of determining if two side-by-side faces are of the same individual or not and in terms of identifying if a face is the result of digital morphing. We show that human participants struggle at both tasks. Even modern machine-learning-based facial recognition struggles to distinguish between an individual and their morphed version. We conclude with a hopeful note, describing a computational technique that holds some promise in recognizing that one facial image is a morphed version of another.

Identifying Computer-Generated Portraits: The Importance of Training and Incentives.

The past two decades have seen remarkable advances in photo-realistic rendering of everything from inanimate objects to landscapes, animals, and humans. We previously showed that despite these tremendous advances, human observers remain fairly good at distinguishing computer-generated from photographic images. Building on these results, we describe a series of follow-up experiments that reveal how to improve observer performance. Of general interest to anyone performing psychophysical studies on Mechanical Turk or similar platforms, we find that observer performance can be significantly improved with the proper incentives.

A perceptual metric for photo retouching.

In recent years, advertisers and magazine editors have been widely criticized for taking digital photo retouching to an extreme. Impossibly thin, tall, and wrinkle- and blemish-free models are routinely splashed onto billboards, advertisements, and magazine covers. The ubiquity of these unrealistic and highly idealized images has been linked to eating disorders and body image dissatisfaction in men, women, and children. In response, several countries have considered legislating the labeling of retouched photos. We describe a quantitative and perceptually meaningful metric of photo retouching. Photographs are rated on the degree to which they have been digitally altered by explicitly modeling and estimating geometric and photometric changes. This metric correlates well with perceptual judgments of photo retouching and can be used to objectively judge by how much a retouched photo has strayed from reality.

A comparative analysis of human and AI performance in forensic estimation of physical attributes.

Human errors in criminal investigations have previously led to devastating miscarriages of justice. For example, flaws in forensic identification based on physical or photographic evidence are notoriously unreliable. The criminal justice system has, therefore, started to turn to artificial intelligence (AI) to improve the reliability and fairness of forensic identification. So as not to repeat history, it is critical to evaluate the appropriateness of deploying these new AI forensic tools. We assess the feasibility of measuring basic physical attributes in a photo using a state-of-the-art AI system, and compare performance with human experts and non-experts. Our results raise concerns as to the use of current AI-based forensic identification.

Art and the science of generative AI.

Understanding shifts in creative work will help guide AI's impact on the media ecosystem.

The specificity of the search template.

When searching for a target object, observers use an internal representation of the target's appearance as a search template. This study used naturalistic stimuli to examine the specificity of this template. Observers first learned several name-image pairs; they then participated in a search experiment in which the names served as cues and the images served as targets. To test whether the observers searched for the targets using an exact image template, we included targets that were transformations of the studied image and targets that belonged to the same subordinate-level category as the studied image. The same stimuli were also used in a search experiment involving image cues. The name cue and image cue experiments produced different patterns of results. Unlike image cues, name cues produced similar benefits for transformations of the studied images as for the studied images themselves. Also unlike image cues, names cues produced no benefit for members of the same subordinate-level category as the studied image. These results suggest that when observers are trained on an image, they develop a search template that is relatively specific for the image but still tolerant to changes in scale and orientation.

Image Forensics.

From mainstream media outlets to social media and everything in between, doctored photographs are appearing with growing frequency and sophistication. The resulting lack of trust is impacting law enforcement, national security, the media, e-commerce, and more. While some types of manipulations can be detected with a careful visual examination, our visual system seems unable to reliably detect other types of manipulations. The field of image forensics has emerged to help return some trust in photography. I describe the perceptual limits of detecting manipulated images, as well as representative examples of computational techniques for authenticating images.

Can people detect errors in shadows and reflections?

The increasing sophistication of photo-editing software means that even amateurs can create compelling doctored images. Yet recent research suggests that people's ability to detect image manipulations is limited. Given the prevalence of manipulated images in the media, on social networking sites, and in other domains, the implications of mistaking a fake image as real, or vice versa, can be serious. In seven experiments, we tested whether people can make use of errors in shadows and reflections to determine whether or not an image has been manipulated. Our results revealed that people's ability to identify authentic and manipulated scenes based on shadow and reflection information increased with the size of the manipulation, but overall, detection rates remained poor. Consistent with theories of incomplete visual representation, one possible reason for these findings could be that people rarely encode the details of scenes that provide useful cues as to the authenticity of images. Overall, our findings indicate that people do not readily make use of shadow and reflection cues to help determine the authenticity of images-yet it remains possible that people could make use of these cues, but they are simply unaware of how to do so.

The tempo and mode of three-dimensional morphological evolution in male reproductive structures.

Various evolutionary forces may shape the evolution of traits that influence the mating decisions of males and females. Phenotypic traits that males and females use to judge the species identify of potential mates should evolve in a punctuated fashion, changing significantly at the time of speciation but changing little between speciation events. In contrast, traits experiencing sexual selection or sexually antagonistic interactions are generally expected to change continuously over time because of the directional selection pressures imposed on one sex by the actions of the other. To test these hypotheses, we used spherical harmonic representations of the shapes of male mating structures in reconstructions of the evolutionary tempo of these structures across the history of the Enallagma damselfly clade. Our analyses show that the evolution of these structures is completely consistent with a punctuated model of evolutionary change and a constant evolutionary rate throughout the clade's history. In addition, no interpopulation variation in shape was detected across the range of one species. These results indicate that male mating structures in this genus are used primarily for identifying the species of potential mates and experience little or no selection from intraspecific sexual selection or sexual antagonism. The implications of these results for speciation are discussed.

A scale invariant measure of clutter.

We propose a measure of clutter for real images that can be used to predict search times. This measure uses an efficient segmentation algorithm (P. Felzenszwalb & D. Huttenlocher, 2004) to count the number of regions in an image. This number is not uniquely defined, however, because it varies with the scale of segmentation. The relationship between the number of regions and the scale of segmentation follows a power law, and the exponent of the power law is similar across images. We fit power law functions to the multiple scale segmentations of 160 images. The power law exponent was set to the average value for the set of images, and the constant of proportionality was used as a measure of image clutter. The same 160 images were also used as stimuli in a visual search experiment. This scale-invariant measure of clutter accounted for about 40% of the variance in the visual search times.

The accuracy, fairness, and limits of predicting recidivism.

Algorithms for predicting recidivism are commonly used to assess a criminal defendant's likelihood of committing a crime. These predictions are used in pretrial, parole, and sentencing decisions. Proponents of these systems argue that big data and advanced machine learning make these analyses more accurate and less biased than humans. We show, however, that the widely used commercial risk assessment software COMPAS is no more accurate or fair than predictions made by people with little or no criminal justice expertise. We further show that a simple linear predictor provided with only two features is nearly equivalent to COMPAS with its 137 features.

Estimating planar surface orientation using bispectral analysis.

In this correspondence, we propose a direct method for estimating the orientation of a plane from a single view under perspective projection. Assuming that the underlying planar texture has random phase, we show that the nonlinearities introduced by perspective projection lead to higher order correlations in the frequency domain. We also empirically show that these correlations are proportional to the orientation of the plane. Minimization of these correlations, using tools from polyspectral analysis, yields the orientation of the plane. We show the efficacy of this technique on synthetic and natural images.

Geographic Variation of Plant Circadian Clock Function in Natural and Agricultural Settings.

The increasing demand for improved agricultural production will require more efficient breeding for traits that maintain yield under heterogeneous environments. The internal circadian oscillator is essential for perceiving and coordinating environmental cues such as day length, temperature, and abiotic stress responses within physiological processes. To investigate the contribution of the circadian clock to local adaptability, we have analyzed circadian period by leaf movement in natural populations of Mimulus guttatus and domesticated cultivars of Glycine max. We detected consistent variation in circadian period along a latitudinal gradient in annual populations of the wild plant and the selectively bred crop, and this provides novel evidence of natural and artificial selection for circadian performance. These findings provide new support that the circadian clock acts as a central regulator of plant adaptability and further highlight the potential of applying circadian clock gene variation to marker-assisted breeding programs in crops.

Medical image registration with partial data.

We have developed a general-purpose registration algorithm for medical images and volumes. The transformation between images is modeled as locally affine but globally smooth, and explicitly accounts for local and global variations in image intensities. An explicit model of missing data is also incorporated, allowing us to simultaneously segment and register images with partial or missing data. The algorithm is built upon a differential multiscale framework and incorporates the expectation maximization algorithm. We show that this approach is highly effective in registering a range of synthetic and clinical medical images.

Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge.

We conducted an online survey using a set of factual science questions that are commonly administered to assess fact-based scientific literacy. We report that the online population performed substantially better on this standard assessment than the traditional survey population. For example, it has been widely reported that 1 in 4 Americans does not know that the Earth revolves around the Sun, whereas among the online population, this ratio is reduced to 1 in 25. While new online platforms provide researchers with unprecedented ease of access to a large sample population for studying trends in public knowledge and attitudes, generalizing from online population samples to the US population at large poses a considerable challenge. We discuss the potential reasons for this discrepancy and the implications for conducting research online.