Behind the Scenes: Making Sesame Street's iPad App

More

Shortly after the iPad's launch, when curiosity overwhelmed reason, my wife purchased a Wi-Fi model, which was promptly nationalized by our then-twenty-month-old daughter, Margaux. Thinking that she would grow frustrated and tire of it quickly, I indulged her desire to play with the shiny newness. But only a few tentative swipes later, she was fully in command, navigating through apps like a pro. It dawned on me: This is her first computer. And it was so intuitive, so elegant, so refined, that it was completely usable by someone who eight months before was just learning to walk. Seeing the future playing with the future was magical and unsettling. I realized that if I was going to be a part of her world, I wanted to help create it.

Developing an app is very much like trying to carry a tray full of champagne flutes across a raging torrent by log rolling. It is precarious, requires balance and nimble footwork, and seems like a cool job to someone who is watching from dry land. It also involves frequent dunkings in frigid water. I lucked out in the fact that the hardest part of the process had already been taken care of by the time I joined the team at Callaway Digital Arts; they had already commenced the log rolling by producing several apps, including "Miss Spider's Tea Party" for both the iPad and iPhone. Having a well-received app in the store afforded us the opportunity to work with some incredible partners, including Martha Stewart, Thomas the Tank Engine, and Sesame Street. Most recently, we released the app version of The Monster at the End of This Book, Sesame Street's classic children's book starring "loveable, furry old Grover." First published in 1971, it has entertained and enthralled generations of kids, including myself, with Grover's admonition: "Please do not turn the page."

Story continues after the gallery.

Working with Sesame is a privilege; it's rare to find anyone in any field more passionate about their work. They truly understand kids -- and hold themselves (and their partners) to the highest standards. They also have the experience, and weight, of forty years of tradition bearing down on them. Monster is one of their crown jewels, and from the outset we felt an immense responsibility bringing this classic to the new medium of the iPad.

Our early brainstorming sessions included members of the development team recounting their memories of being read the book as children. Our head of technology, Nelson Gomez, even graced us with an impromptu rendition of his reading the book to his four sons. I confessed that I had memorized the book as a child and would recount it, in Grover's voice, to my younger sisters. Despite a thirty-year hiatus, I revived my rendition of Grover, which was deemed passable, albeit slightly haggard -- as though Grover had spent a few rough nights out on the town.

When I worked at Pixar I witnessed a lecture by Hayao Miyazaki describing his creative process. He explained that he locked himself in a small room, sketched, and dreamed, awash in pure creativity. He would then emerge, and for the next three years: "much suffering." We wanted to get the app out for the holidays, and it was early October. It wouldn't be three years, but we wouldn't get much sleep in the next few months.

Nothing quite like Monster existed in the app store. We wanted it to be a fully interactive reading experience, as though the book itself had come alive. Our creative mandate was to preserve the appeal and charm of Jon Stone's original story as well as capture the whimsy and funk of Michael Smollin's illustrations. Unfortunately, most of the tools available to us were focused on creating 3D games, not interactive, illustrated storybooks. So we engineered a hybrid solution to preserve the line quality of the original illustrations while animating Grover, the text, the pages, and the myriad obstacles Grover throws in our way. Using the original book illustrations, our illustrator re-drew Grover in a variety of poses: holding a hammer, walking, gesturing frantically, cowering. Our animators then sliced Grover into hundreds of pieces, down to the individual finger, and created incredibly intricate character rigs in After Effects from which to animate the scenes. Through much painful trial and error we managed to get the first few scenes onto the iPad.

Developing an app is very much like trying to carry a tray full of champagne flutes across a raging torrent by log rolling.

Sesame arranged to test a very early build of three scenes from the app with parents and children so that we could refine the work in progress. This research is a rarity in the app world, where development constraints don't allow for much beta testing. Unfortunately, we weren't able to get Eric Jacobson, the actual voice of Grover, to record all of the necessary lines for our focus group. I was conscripted to record my aging-castrati-coalminer Grover in the coat closet of our offices (where I scared a coworker who unknowingly opened the door mid-record). In addition to great feedback from kids and parents, one child tester rejoiced after knocking down Grover's brick wall, "He said that I was very strong!" Somehow, the agony of sprite sheet optimization doesn't seem so burdensome after that.

The iPad as a platform presents a whole host of challenges for developers. My initial apathy has been replaced with a deep respect for the creators of both the device and iOS. It truly is changing the way that people think about and use computers -- perhaps even more so than the iPhone did. But forget that the iPad has 256Mb of RAM at your own peril. (Think Porche 911 turbo body with a VW Beetle Engine.) This comes as a rude shock to someone with a background in feature 3D animation, where scene files can stretch into the tens of Gigabytes. But I firmly believe that great art comes from being forced to find innovative solutions to difficult constraints. As painful as it is to discard a great moment or hilarious sequences, it really forces you to focus on that which is essential to telling a spellbinding story. You must capture an entire sentence in a single gesture, a monologue in a nuance. The apps we're creating focus on core interactive experiences that are also visually arresting, simply implemented. Like the writer's adage of Arthur Quiller-Couch, it's the agonizing process of "murdering your darlings" that makes for a great experience on iPad.

We finally managed to roll the app across the river, soggy but unscathed, and uploaded it to Apple in the wee small hours before they closed the store for the holidays. The next morning, I handed my iPad to Margaux, now two and a half, for her examination. She immediately found the new app's icon amidst her clutter of favorites and touched it, only to be greeted by Grover's lithe "Hello Everybodeee." She needed a slight amount of encouragement to overcome her anxiety about defying a direct order not to turn the page. (In retrospect, cultivating defiance in a two-year-old: not such a good idea.) Soon enough, she was tearing through the book, untying knots, sawing through wood, and knocking down the brick wall that Grover threw up in her way. The magic of holding Grover in your hands and having him respond to your actions was quickly apparent.

Then I had a profound iPad Aha moment. My first computer was an Apple II+ in 1980. The thing that Margaux was holding in her hands was infinitely more powerful, more friendly, and brimming with potential. The iPad is still a 1.0 product. Imagine for a moment what it will be capable of in five years -- or thirty. If we're lucky, Grover will be still be on it, telling us "Please, Please, Please do not turn the page."

Jump to comments
Presented by

Gabriel Schlumberger is a senior app producer at Callaway Digital Arts. Before joining Callaway, he worked at Pixar and Blue Sky studios. His film credits include The Incredibles, Cars, and Horton Hears a Who.

Get Today's Top Stories in Your Inbox (preview)

Why Do Men Assume They're So Great?

Katty Kay and Claire Shipman, authors of this month's Atlantic cover story, sit down with Hanna Rosin to discuss the power of confidence and how self doubt holds women back. 


Elsewhere on the web

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus

Video

Where Time Comes From

The clocks that coordinate your cellphone, GPS, and more

Video

Computer Vision Syndrome and You

Save your eyes. Take breaks.

Video

What Happens in 60 Seconds

Quantifying human activity around the world

Writers

Up
Down

More in Technology

Just In