Thursday, January 4, 2018

Semantic Mapping of Images

When we look at a scene, our visual systems naturally separate objects from each other and from the background.

Illustration by Robert Cunningham
Objects that are especially important, such as people, may appear more detailed and nuanced. When artists interpret the chaos of reality into a painting, they often apply this hierarchy to the finished image, resulting in what might be called a "style."

Here's another way to think about style: What artists really give us is reality seen through the filter of human perception.
Illustration by Bernie Fuchs
In the illustration above, for example, the elements are separated. Less important elements, such as the ground surface and the seats, are rendered in mostly flat color, but the faces and clothing patterns are rendered with more tones and details.

This ability to a) distinguish, b) identify, and c) prioritize elements within a scene is called "semantic mapping."

Until recently it has been a distinctly human ability. But machine-learning systems are getting good at it too.


A new scientific paper explains how artificial systems can analyze a photo or video of a scene into its constituent parts and identify each of them, something that would have taken a human a lot of time-consuming work with Photoshop.

This technology will have powerful implications for creating and editing photographic images, but also for interpreting reality into images that seem to have a subjective artistic "style."

Once a computer can semantically map a scene, it can re-render it in any style you want: whether as flat shapes, a line drawing, a caricature, or an impressionist painting.


-----
Here's a video explaining the new tools, which are free to download. (Link to YouTube video: "AI Learns Semantic Image Manipulation")
Scientific Paper: "High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs"
Related Posts:
Image Parsing

2 comments:

jytte said...

Oh dear you might be out of work soon :o) LOL

Allen Garns said...

Fascinating. Much of what the narrator was saying went over my head but still can see that this is some pretty powerful stuff. It's interesting to compare this technology to the Cunningham and Fuchs paintings. Looking at both of them, I am struck by the beauty of the shapes/negative shapes. For example in the way he put the objects and figures together with such awareness of the negative shapes! It reminds me of the battle scenes of Piero della Francesca. Particularly the sequence of the lady in the red coat to the half hidden lady in the patterned coat to the rump of the lady in the pink turban and gray skirt. wonderful shapes. Also, all the yellow background shapes behind the figures. That's one reason why it's going to to take a long time before you (or any other good artist) "might be out of work soon".