How Do Our Brains Process Images—Decoding or Predicting?

For a long time, scientists believed that images were decoded in a bottom-up process.

It was thought that the image arrived on the retina and was sorted out in stages, starting with edges and shapes that were then assembled into recognizable objects, faces, or symbols by specialized areas of the brain.

You may recognize what this image represents just from the top of the picture; you may even guess who painted it.

Detail of portrait by John Singer Sargent

The alternate way of understanding visual perception is the top-down predictive model. This idea suggests that the brain contains a stockpile of representations of reality that it imposes on what you see. You don't really see reality directly; you see your brain's prediction of what's in front of you. 

According to neuroscientist Andy Clark, "the brain is essentially a prediction machine." 

Most scientists agree that these two systems interact at various stages. Bottom-up constructive image processing meets top-down prediction throughout the hierarchy of image decoding from the most basic to the most sophisticated. 

This process moves like greased lightning until you run across a prediction error or an ambiguity.

Green lines are equal. Source

Optical illusions present anomalies where the top down prediction comes into conflict with the bottom-up construction. Sometimes this conflict happens at relatively basic levels of processing, creating estimation errors of perspective, shape, or overlapping.

This skull illusion creates an ambiguity of interpretation at a higher level of interpretation. 

It's good for us as image makers to be aware of how our eyes and minds work as we decode the world around us and reconstruct it for the enjoyment of others.

--

Read the science paper: Whatever next? Predictive brains, situated agents, and the future of cognitive science