Neuroscientists at University of California at Berkeley have developed a technique for creating digital images that correspond with neural activity in the brain.
This represents one of the first steps toward a computer being able to tap directly into what our brain sees, imagines, and even dreams.
(Link to video) Every image that we see activates photoreceptors in the retina of the eye. The information is fed through the optic nerve to the back of the brain. There, the information is assembled and interpreted by increasingly higher-level processes of the brain.
In this experiment, subjects watched clips of movie trailers while an fMRI machine scanned their brains in real time. The computer mapped activity throughout millions of “voxels” (3D pixels).
The computer gradually learned to associate qualities of shape, edges, and motion occurring in the film with corresponding patterns of brain activity.
It then built “dictionaries” by matching video images with patterns of brain activity, and then predicting patterns that it guessed would be created by novel videos, using a palette of 18 million seconds of random clips taken from the internet. Over time, the computer could crunch all this data into a set of images that played out alongside the original video.
If I understand the process correctly, the images we’re seeing on the right side (“clips reconstructed from brain activity”) are actually running averages created by blending a hundred or so random YouTube clips that met the computer’s predictions of what images would match the patterns it was monitoring in the brain.
In other words, the right-hand image is generated from existing clips, not from scratch. In this video (link), you can see the novel video that's causing the brain activity in the upper left of the screen, and some of the samples (strung out in a line) that the computer is guessing must be causing that kind of brain activity.
That would explain the momentary ghostly word fragments that pop up in the images, as well as the strange color and shape-shifts from the original.
The result is a moving image that looks a bit like a blurry version of the original video, but one that has been a bit generalized based on the available palette of average images. Evidently, the perception of faces triggers the brain in very active ways, judging from the relative clarity of the computer’s generated images, compared to other kinds of images.
I wonder what would happen if you set this system up in a biofeedback loop, so that the brain activity and image generation could play off against each other? It might be like a computer-aided hallucination.
Article on Gizmodo
Thanks, Christian Schlierkamp
Yeah this is a real interesting video. I saw it a few days ago, but in reading about the experiment I wondered if it wasn't slightly misleading.
ReplyDeleteThe way I understood it, the subjects watched videos and the corresponding cerebral activity was matched up to that footage. Then when they were played completely new footage the brain activities were measured. The program then goes back to the images it previously associated most closely with that signature of brain activity and creates an average of them. So it is not in any way 'reconstructing' the image you are currently viewing as claimed by some of the headlines. It is simply recalling a past reference library of associations made when studying you and your stimulus.
The important difference being that it would not work without the specific calibration for each person to build a dictionary (as everyones brain works differently). If you extrapolate that it means it would be very hard for the program to reconstruct images from a blind person as claimed in the newscenter article. And it creates doubts in my mind about reconstructing dreams and imagination as that would require the assumption that these very different activities activate the brain in precisely the same way as seeing.
Still a cool video and cool area of study though, and the technology will improve. Just sensational reporting I'm sure.
Here is a Ted Talk from July 2010, that seems to me to be demonstrating the same principle. The ability to recognise individual thoughts by their measurable signature in the brain and build a library of them.
ReplyDeletehttp://www.ted.com/talks/tan_le_a_headset_that_reads_your_brainwaves.html
The guy in the wheelchair at the end really demonstrates the point. His action of "smile" causes the wheelchair to go straight. Amazing technology, but the actual meaning of the brain signature is irrelevant. "Smile" essentially has nothing to do with "wheelchair proceed directly forward". It is simply a brain signature that can be identified, and therefore tagged to something else.
Similarly in the "Constructing images of Brain Activity" experiment, it seems to me that they are building these associations beforehand. They are indexing brainwaves whose correlating stimulus they already know, not actually deciphering the brain signatures themselves.
Still amazing tho. Go science!
Fascinating.
ReplyDeleteI wonder when it will be possible to create artwork by just thinking of the image in your brain.
ReplyDeleteThat's an interesting system. It doesn't know exactly what the subject is seeing. It doesn't know what are the visual stimuli reaching the brain. What it does know is that some things trigger specific responses from the brain.
ReplyDeleteSo, instead of trying to decipher what the person is actually seeing in visual terms, they guess it by comparing the reactions. It doesn't know whose face it is the subject is seeing, but it does know it is a face because that's the pattern that matches its database the best. Then it's just a matter of compositing the most relevant "hits", Google-style.
Maybe it could be useful for recovering at least some information from hazy memories? Probably not reliably, since memories at actually rather treacherous the longer they last. Well, at least it's yet another interesting scientific curio that could serve as a springboard for something more interesting in the future.
Science marches on!