Sunday, September 4, 2022

Image-to-Image Style Transfer

Suppose you made this rough sketch and wanted to finish it in a photo-real or painterly style.


Twitter user @TomLikesRobots is a digital artist who did just that, using the new image generation tool called Stable Diffusion (#stablediffusion), the open-source model which has a powerful feature called Image to Image (#img2img). 


He uploaded the sketch and told the system to render it as "A black and white photo of a young woman, studio lighting, realistic, Ilford HP5 400." The hair style is a little different, and the ear is weird, but the basic pose and lighting is pretty close to the sketch.

Tom says: "Overall, composition is controlled by the sketch and the details are controlled by the prompt. 



This time he told it he wanted it to make the sketch look like "A portrait of a young woman by Norman Rockwell." 

It looks like a painting, but not much like Rockwell—more like a Victorian artist like Charles S. Lidderdale. On close scrutiny it doesn't hold up too well: the blue ribbon and the costume don't make sense, and her left eye has too many eyelashes.


Here's the sketch rendered in the style of Gustav Klimt. It's got a lot of drawing problems that Klimt never would have allowed, but it is somewhat reminiscent of his style.


And here's the sketch as painted by Vincent Van Gogh. Again, we can pick it apart, but it's in the ballpark.

.
And finally Alfonse Mucha. He is one of the hardest to emulate. It used soft internal transitions and clear outline.

The system was also able to translate the sketch into the features of famous celebrities:

Selena Gomez

Scarlett Johansson

Nicole Kidman

Emma Stone

All of them have big problems with the hair, but they're recognizable, and let's admit the tech is in its infancy and will only get better.

If you're an artist, you might find all this a little scary, threatening, or astounding. I do too! I feel like we've been introduced to a magical genie who can bring whatever we wish into existence.

Some have responded by calling for laws to ban the technology from creating images in the styles of living, working artists. What if someone created a print for sale that was supposedly painted by you or me?

I think we have to be careful how we respond to this apparent threat. Let's keep in mind that the technology is transformative, meaning it doesn't copy/paste images. It creates something new. And artistic styles can't be—and shouldn't be—copyrightable. These tools are here to stay. They're only going to get more advanced, and they're open-source. Also the people who are developing it are artists, too. 

I'm wary of laws or AI bots that restrict the growth of this new art form, or that drive the prompts underground. Instead of involving politicians and lawyers and AI bots, we should encourage a culture of mutual respect and fair play. Perhaps we should encourage generative artists to share their prompts when they use a living artist's name, and never to mislead their audience into thinking the work was actually painted by that artist. Basically, people should give credit where credit is due.

How do you think we should regulate or guide this new art industry?

12 comments:

Bill Wilson said...

My name is Bill Wilson. I've been a portrait painter and illustrator for thirty-five years and worked in movies and television for over fifteen years. At sixty I've watched the art world change unimaginably. Seeing posts like yours, I feel like I'm on an episode of Star Trek. It is scary, but rather than give in to our fears or leap to call our lawyers, I agree with you, we should embrace the new world. Notwithstanding the obvious encroachment AI is making into our craft, it is also opening up incredible opportunities. Despite not living in New York or Hollywood, it's given me the opportunity to work with Sony and Universal Pictures and allowed me to be the art director for a national magazine.

Lotheredoisee said...

Super interesting, and yes a little unsettling. I don’t see how regulating the production of any imagery is proper, just need to beware of fraud as with any transaction.

Bbirreg@gbis.com

elgin said...

There is no law that could be written that could reasonably control this, certainly under our Constitution. The lesson to learn is th NEVER trust anything you see again.

Ive said...

I feel like all my time and effort I put in developing the skills to become a freelance artist has been for nothing, since the influx of new AI artists will overrun the market and it's going to drive the prices down. It's going to be even tougher to make a living and that's what scares me.

Anne said...

I've been around the craft sales world for a long time. And time after time I've seen ideas started by an artist,taken over by corporate, cutting out the original smaller creator who could not compete with the big guys. The only way artists have ever stayed ahead is by innovation and ideas. What we have over AI. Even with a thousand filters, AI still needs a human creator to get the ball rolling.

Garrett said...

I don't know, the argument seems to be "you can't stop it, so why bother"... That strikes me as begging the question. To me, there is a lot to still be worked out in terms of legality of using copywritten material for training sets.

This technology ultimately relies on human-generated art and photography as a starting point. Without that input material, it can't make anything. I highly doubt these companies are paying for the photos of celebrities or compensating the photographers & artists they've fed into their machines (and could never remove it now that they've done it). Sure, the imagery is all out there to consume online, but much of it is copyright protected.

These tools exemplify the way in which artists are constantly taken advantage of, where the products of their labor are frequently stolen and repurposed without appropriate compensation. Shades of Napster / Spotify, etc.

Yes, governments can certainly control these kinds of technologies if they have the will to do so, mainly by defining what constitutes "fair use" of copywritten material and requiring full disclosure of training set materials. There's no reason we have to just let companies create and sell technology products simply because they can.

To me, that will likely only happen if powerful corporations begin to object.. Surely Disney would not lay down if they discover their characters have been fed into these systems?

Anonymous said...

I think the last one looks more like Margot Robbie than Emma Stone.

Wendy said...

If people want sterile, empty, vapid, artificial art created by a unthinking machine then they are welcome to it. Say goodbye to the commercial art industry.

Nicholas Burns said...

Only time will tell what impact AI image-generation will have on art and society. Will it have the impact of photography in the mid-1800s? Possibly. Like photography, AI enables people with no art training to create images. According to some historians, photography also changed how people understood reality, mortality, and egalitarianism. Not sure AI imagery will be that impactful. Two things worth noting: 1)there's no 'Turing Test' for images; most audiences simply don't care how an image was created and 2)the uncritical reliance on tech such as phones, social media and/or the internet already challenges people's ability to separate fact from fiction. Adding AI images to the mix might exacerbate that.

Nora said...

We have devolved into a culture of content over creation, or insurmountable quantity over quality. This is the leg stuck out before the tumble into the swamp of thoughtless, derivative work, which exists only to parlay your attention to sponsored products and services. I am sick of the series, posts and channels dedicated to churning out Content™, and I've got a rope tied around my waist as a young artist, to hopefully pull myself out of this mess and come out with a career to show for it.

James Gurney said...

Thanks, Nora, you've expressed a very interesting view in a most poetic way.

ArtistwithDharma said...

Hi I'm a visual artist of 43 years. 33 of those years I've been selling my works to private collections and public commissions, both nationally and internationally. Within the graffiti scene ideas are generally inspired by or cleanly and clearly BITTEN which mean stolen. In graffiti there isn't a system whereby an artist painting a wall will give credit to the artist they stole the idea from, but those within the scene recognise this so it's quite often an artist will dig their own grave by clearly BITING another artist's work. Yet nowadays even with the internet I've pulled one of my friends and called him out for biting and he blatantly denies ever seeing the work he bitten from when it's a bloody exact CARBON COPY.. and for me being Autistic I love on a principal of being honest and all respect I had is now lost as it's not the first time he's BITTEN a style and I've recognised it from another side of the world or another artist in the UK but I let it slide. I've had work and concepts stolen where artists have gone on to make a lucrative short lived fame on the back of my ideas, but they often end short as they lack originality and the more commissions that roll in the more they drift away from the OG ideas I had shown then into horrendous works of art.

I used to openly share my sketchbooks in public at street art events I've spoke about many of my wildest ideas from my head that never even made it to paper, yet three concepts I had ended up in banksys exit through the gift shop movie

Clearly BITTEN and clearly NO CREDIT given to my ideas, most artists at that time wanted to view my latest sketchbooks as they called me the machine for the amount of quick drawings I made, never any time to colour anything in as I would be onto the next groundbreaking concept, at the end of the day I know I'm the artist that changed the direction of street art into what it became today.
Everything I saw was nothing new, nothing advancing or evolving the artform of graffiti. The whole scene stagnant and I decided to paint landscapes and architecture in the early days of this new movement called street art. Most the artists of that time said impossible and BOOM I did it .. anyone knows my pioneering methods from that era. That doesn't need credit.. but the banksy movie situation led me to isolate myself and my sketchbooks from prying eyes. No more will anyone steak my concepts

Now in the world of AI and NFTs I think just the way that YouTube can pick a piece of music I chose to use ina video froma DJ friend of mine yet it refused to post as it's embedded with some code preventing use of that work without permission

This is the solution that us visual artists need applying to our works preventing anyone using ai art for fraudulent and monetary purposes in the style of me or anyone else. That would be the solution and that's what
I think what needs to be done