Friday, October 28, 2016

Style Transfer

Computers are able to take any photo and reinterpret it in any given artist's style. You can give the computer some examples of an artist's work along with a photo of your own, and then the app will come up with an image that superficially resembles the style of that artist.

Neil deGrasse Tyson plus Kandinsky’s Jane Rouge Bleu.
Photo by Guillaume Piolle, Via Google Research

Modern apps can accomplish more than a Photoshop filter can, because they enlist neural algorithms to separate style from content when they look at images.

They appear to set up a hierarchy of what's important about an image. In the portrait above, they keep the eyes and mouth in place while scrambling the less important jacket and tie.

Image by Manenti1 using the Aptitude filter via Dreamscope
With all these deep learning apps, I notice that the realism of the photograph always asserts itself through the shapes and colors, much in the same way rotoscoping does with animation.

In order to better simulate childlike, subjective, or naïve styles, such as those of Cezanne, Renoir, or Matisse, the computer will have to redraw the image to make the placement and proportions deviate from photographic reality in ways that those humans practitioners do.

Nat and Lo, two Google employees who go around the company asking how things work, do a good job explaining how deep learning techniques help computers solve this problem. You may need to follow this link to watch the video on YouTube.

Understanding this process helps us understand how we humans see and interpret images, and it also can help us as artists if we want to develop our own style, or on the contrary, if we want to try to rid ourselves of stylistic conventions.
Previous Related Posts:
Using Computers to Create a Typical Rembrandt
Image Parsing


Roca said...

Maybe I’m paranoid but I see this as a further threat to artists, if ripping off their work from Google Image Search wasn’t already. I wish artist had some sort of lobbying voice, because we really need to launch a nationwide campaign to help people understand the value in original art and design. I only see artists’ income going down as a result of the internet, cheap foreign competition, and now this.

Tom Hart said...

I understand your concern, Meredith, and from time to time when I read about such things I feel the same way. But I always come back to more or less the same conclusions: First, no one is going to, or can, stop the development of tools like the ones described here. In fact, these applications will be used and manipulated by artists into other, newer forms of art or used as tools in the service of traditional art (i.e, analysis and preliminary work - the way some use Photoshop and/or computer design applications today). Second, there will always be a market for quality hand made and produced artwork; people who just want to own or use an image without care for its quality or how it's made are always going to be "out there" and will find a way to do so cheaply. Now that being said, to the extent your fear relates to copyright and ownership issues in (in both the legal and general sense), I think concerns like the ones you express are well founded.

Unknown said...

I'm making my comments in regards to Meredith D's comments. Why is it that artists always have to feel threatened by technology? Technology can also threaten jobs outside of the arts.

_ said...

I enjoyed reading your books about technique Mister Gurney, but your opinions on art are somehow... childlike and naïve!

Warren JB said...

Well, that escalated quickly.

I hesitate to comment, because things are already getting a bit heated and personal, and I know I sound like a luddite; but as interesting and amazing as these developments are, what's the intended application?

James Gurney said...

Warren, I'm not sure if there is an application just now. I suppose it could be used to generate artwork that imitates the output of established artists, and that might be threatening to people. Nat and Lo, who work at Google, seem to describe it as an offshoot of Google's efforts to make their computers understand what they're looking at so that they can identify and classify images. It probably will make us continue to redefine what is art and what is an artist.

I think Tom H. has it right that these developments are inexorable, and that the tools will almost certainly be useful to artists in ways we can't anticipate.

dinodanthetrainman said...

I can think of at least one practical use in the tv and motion picture industry.

Goodbye motion capture, hello practical effects that you can cheat on like the days of black and wight, transferring the style of the concept art onto the film or tv series itself.

Linda said...

The Pikazo app available on iPad is one such example of deep neural net software. It is really fun to play with. Think mashups and strangely altered photographs, collages, kaleidoscopes. As an artist I wouldn't feel threatened at all, it presents a world of new opportunities.

Roca said...

I don’t expect the march of technology to slow any time soon. What I mean instead is that artists have to find some way to bring awareness of the value of handmade art back to people’s consciousness. When you say “there will always be a market for quality handmade artwork,” that really means a very, very small market. I have friends in the fine art industry and only one that comes close to making a living and he has a pension to pay most of the bills. Those in the commercial arts continually find their pay shrinking ever smaller, the jobs disappearing and going overseas, and being pressed into doing more programming, web design, etc. that is really so far outside their original interest. I don’t think the outlook of the job market is good for anyone with an interest in drawing and painting. It’s really just sad to me. Is it possible to make a career out of it? Maybe for 1/2 of 1% of those who try. In that way I suppose it’s analagous to actors or musicians, very few of whom are able to make a career of their passion.