Inside the Science That Plays Your Brain Like a Videotape

More

Our senses take in a vast array of sights and sounds every day, sparking activity in our neural circuits in corresponding patterns. Now, scientists are developing ways to decode these patterns and recreate the original audiovisual stimuli -- not unlike the video playback of a VHS tape. This episode of Science Bytes, produced by Kikim Media for PBS and the Public Library of Science, explores some of the research in this field. Last year, scientists at U.C. Berkeley demonstrated that they could reconstruct the YouTube videos their research subjects were watching by analyzing fMRI scans of their brain activity. Associate professor of psychology Jack Gallant at Berkeley's Vision Science lab describes how they did it. Later in the segment, Bob Knight, a professor of neuroscience and psychology also at Berkeley, successfully reconstructs the audio that patients are hearing during brain surgery for epilepsy. Thanks to electrodes measuring electrical activity in the cortex, they could decode this activity and play back the audio the patient heard -- with spooky accuracy. 

The episode doesn't linger on the stimuli and the resulting videos Gallant's lab recreated, but the shot-by-shot comparison is amazing (below). It's not hard to imagine a cyborg future in which we can "make movies with our brains," to quote music video directors Daniel Scheinert and Daniel Kwan.

On their YouTube page, the researchers describe how they built a library of random YouTube videos and then mapped them to the responses in their subjects' brains: 

The procedure is as follows:

[1] Record brain activity while the subject watches several hours of movie trailers.

[2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured. (For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.)

[3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions.

[4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction.

For more videos from Kikim Media, visit http://www.kikim.com.

Jump to comments

Kasia Cieplak-Mayr von Baldegg is the executive producer for video at The AtlanticMore

Cieplak-Mayr von Baldegg joined The Atlantic in 2011 to launch its video channel and, in 2013, create its in-house video production department. She leads the development and production of original documentaries, interviews, and other video content for The Atlantic. Previously, she worked as a producer at Al Gore’s Current TV and as a content strategist and documentary producer in San Francisco. She studied filmmaking and digital media at Harvard University.
Get Today's Top Stories in Your Inbox (preview)

A Wild Vacation in the Pacific Northwest

A not-so-ordinary road trip, featuring extra-tall art bikes, skateboards, and hand-painted vans


Elsewhere on the web

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus

Video

Adventures in Legal Weed

Colorado is now well into its first year as the first state to legalize recreational marijuana. How's it going? James Hamblin visits Aspen.

Video

What Makes a Story Great?

The storytellers behind House of CardsandThis American Life reflect on the creative process.

Video

Tracing Sriracha's Origin to Thailand

Ever wonder how the wildly popular hot sauce got its name? It all started in Si Racha.

Video

Where Confiscated Wildlife Ends Up

A government facility outside of Denver houses more than a million products of the illegal wildlife trade, from tigers and bears to bald eagles.

Video

Is Wine Healthy?

James Hamblin prepares to impress his date with knowledge about the health benefits of wine.

Video

The World's Largest Balloon Festival

Nine days, more than 700 balloons, and a whole lot of hot air

Writers

Up
Down

More in Video

Just In