Thursday, January 19, 2012

Visually Similar

--Thanks to Courtney at Artist Daily for spotlighting this post in on Artist Daily--

For the last couple of years, Google has had an image search option called “visually similar.” This locates images that are related by their abstract qualities, rather than their associated keywords.

For example, here’s a painting from Dinotopia: Journey to Chandara called “Irish Elk,” showing an extinct giant deer in a high mountain landscape. The colors are yellow ochres and browns, along with pale blues. There are no greens and hardly any reds.

Google sifts through millions of images on the web searching for other pictures with related image attributes, and presents those that it finds "visually similar." In this case, the images all have the same basic color gamut, a cluster of warm colors combined with grays and blues. 

Presumably it selects other attributes, such as gradation, complexity, texture, and shape. Other visually-similar programs such as Piximilar do the same thing.

In the previous example, what I found surprising was that, except for the helicopter and the dog, the results are all food ads. Why food ads? I'm guessing that the curving vignette shape surrounding the busy warm texture associated my picture with the curving shapes of plated food. 

Here’s a sketch that I did with marker pens, a high contrast rendering of a man at a podium. 

Google’s search program yielded results with dark silhouettes (not surprising) but the subjects are mostly clothes that are symmetrical and laid out flat. I find this surprising. Why clothes? Why symmetrical?

Here’s another Dinotopia painting, a stone monument at dusk, painted in brown tones, with a golden sky behind and a few cool or gray notes for contrast.

What does Google’s algorithm think is similar? A lot of interior scenes. Why interiors? Why so few outdoor scenes or so few paintings? Perhaps the particular color ranges I chose for my gamut happen to match those of indoor photos with white balance problems. 

I find it fascinating that the results cluster around specific families of subject matter that are so different from the source image. Google explain exactly how their algorithm works, but it's fun to guess at it.

Anyway, searching for visually similar images is a great way to see our own color schemes from a fresh perspective. To use it, go to and press the little camera button in the search window. You can upload any image from your computer, including one of your own paintings in progress, or drop in a URL address of an image you found on the web. 
Both paintings from Dinotopia: Journey to Chandara 


D said...

You can restrict your visually similar image search to a specific word. I tried your first elk image with "painting" and this is what I got.

Marc Hudgins said...

You should check out this video demonstrating an experiment that asks "What happens if you ask Google Images what's most similar, starting with a blank image, repeating the process 2951 times?"

Chris Beatrice said...

It looks to me like the analysis is simply number (or portion) of pixels of similar colors, so it's not taking into account shape at all. In other words, if you posted a picture that had 50% black pixels and 50% white (regardless of how they were arranged), it'd match other pictures that had close to that same breakdown, regardless of how they were arranged.

Anonymous said...

love it

Anonymous said...

If the color analysis could be bypassed, I wouldn't be surprised if any given search listed practically everything as similar.

Janet Oliver said...

Thanks for the tip. I didn't know this search capacity existed. I uploaded a recently commissioned dog portrait (a female yellow lab), and got back mostly images of women in wedding dresses. Only one dog image was in the mix, and it was a little fluff-ball thingee. Fun. I wonder: Can it be used to identify a painting, of whose title one doesn't remember the name, or the artist?

Janet Oliver said...

Oh, dear. I could easily get hooked on this. Just uploaded another completed illustration, and got back some lovely images with a similar palette. This function could be used to plan the color transitions within an illustrated book.

James Gurney said...

Yes, if you open the main page for "" you can drag and drop an image onto the center of the page (in Mac at least) and it will come back with its best guess for the name of the painting, the artist who did it, plus other places on the web where that image occurs, and other sizes of the image. Playing with this can definitely use up hours of time.

D: thanks for that tip. Interesting way to narrow it.

Marc, thanks. There's something hypnotic about that video, and funny how it comes back to certain images.

Chris, I think there's more to it than just a pixel percentage analysis. So, with a simple black and white bunch of pixels, a zebra stripe pattern will bring back different results from a checkerboard pattern.

R.L. Delight said...

I couldn't restrain my curiosity. I had to try it. I uploaded a photo of a painting I did of a new born seal on a northern Oregon beach. Google showed pictures of beaches and birds but also a lot of pictures of decks and stonework. Interesting! Thanks!

Janet Oliver said...

OMG. I tried it with a well-known painting by Jimmy Whistler, Nocturne in Black and Gold, and sure enough, back came Google's best guesses, along with links to the artist's bio, and web pages where the image appears. I also uploaded some of my own images, and Google naturally came back with links to my web site, but also with some interesting images with the same palette. It might be fun to take my own image, along with a Google-returned similar image, and run them through a Photoshop analysis, to see what they have in common. I had no idea; thank you! Way too much fun.

jeckert55 said...

The reason Google created this feature was so that photographers, artists, and uploaders of content (graphs, etc) could track who was reposting or reusing their content. For instance, James, you'd be able to track who was using your color wheel images.

This is the first time you'd be able to run such a search, to my knowledge.

Frances said...

If you squint your eyes and look at the dark shape above the man's head in your drawing... it looks like an almost symmetrical pair of dark pants. I wonder if that is why you got the results you did for that drawing.

Frances said...

The picture of the chairs looks like the black shape on the bottom left that represents the podium.... if looking at just the black shape alone. Also, there is a shape in the middle, at the bottom, that looks like a tank top. I'm not much of an artist, but dated one long ago, who was always talking about looking at art with squinted eyes to "see." :D

Frances said...

The picture of men's ties also mimics the lines in the bottom left corner... the part that looks like the shape of the folding chairs, as well. Fascinating stuff!

Frances said...

In the outdoor Dinitopia painting, you placed a rectangular shape under the statue's legs... at about the "Golden Third" intersection. Google gave you pictures with other rectangular shapes.... interior rooms, because rectangles aren't often found naturally in nature. That's my hypothesis, anyway! :)

Theresa Taylor Bayer said...

Why the similar pair of pants silhouette? I'm seeing in the white shapes of your man it podium sketch, something that resembles a pair of pants, where his arm and shoulder are. Fascinating!