Semantic Mapping of Images

When we look at a scene, our visual systems naturally separate objects from each other and from the background.

Illustration by Robert Cunningham

Objects that are especially important, such as people, may appear more detailed and nuanced. When artists interpret the chaos of reality into a painting, they often apply this hierarchy to the finished image, resulting in what might be called a "style."

Here's another way to think about style: What artists really give us is reality seen through the filter of human perception.

Illustration by Bernie Fuchs

In the illustration above, for example, the elements are separated. Less important elements, such as the ground surface and the seats, are rendered in mostly flat color, but the faces and clothing patterns are rendered with more tones and details.

This ability to a) distinguish, b) identify, and c) prioritize elements within a scene is called "semantic mapping."

Until recently it has been a distinctly human ability. But machine-learning systems are getting good at it too.

A new scientific paper explains how artificial systems can analyze a photo or video of a scene into its constituent parts and identify each of them, something that would have taken a human a lot of time-consuming work with Photoshop.

This technology will have powerful implications for creating and editing photographic images, but also for interpreting reality into images that seem to have a subjective artistic "style."

Once a computer can semantically map a scene, it can re-render it in any style you want: whether as flat shapes, a line drawing, a caricature, or an impressionist painting.


-----
Here's a video explaining the new tools, which are free to download. (Link to YouTube video: "AI Learns Semantic Image Manipulation")
Scientific Paper: "High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs"
Related Posts:
Image Parsing