Project Overview

GauGAN, named after the post-Impressionist painter, in an interactive app that turns rough doodles into photorealistic masterpieces with breathtaking ease.

Organisation

NVIDIA Research

Project Context

GauGAN utilises a deep learning model developed by NVIDIA Research that leverages generative adversarial networks, or GANs, to convert segmentation maps into lifelike images.

Project Innovation

GauGAN allows users to draw their own segmentation maps and manipulate the scene, labelling each segment with labels like sand, sky, sea or snow.

Trained on a million images, the deep learning model then fills in the landscape with show-stopping results: Draw in a pond, and nearby elements like trees and rocks will appear as reflections in the water. Swap a segment label from “grass” to “snow” and the entire image changes to a winter scene, with a formerly leafy tree turning barren.

Despite lacking an understanding of the physical world, GANs can produce convincing results because of their structure as a cooperating pair of networks: a generator and a discriminator. The generator creates images that it presents to the discriminator. Trained on real images, the discriminator coaches the generator with pixel-by-pixel feedback on how to improve the realism of its synthetic images.

After training on real images, the discriminator knows that real ponds and lakes contain reflections — so the generator learns to create a convincing imitation.

GauGAN doesn't just stitch together pieces of other images, or cut and paste textures, It actually syntheses new images, similar to how an artist would draw.