Sesame Street
Taking a walk down Sesame Street with augmented reality
For the 50th anniversary of their flagship program, Sesame Street, Sesame Workshop teamed up with Weyo and DEPT® to bring their characters to life. Together, we developed an app that uses augmented reality (AR) to let Sesame Street’s loyal audience become their favourite muppets through the power of AR.
A special celebration
Since its groundbreaking launch in 1969, Sesame Street has become an international beacon in children’s education. As they prepared to celebrate their 50th anniversary, Sesame Workshop was eager to highlight their five-decade journey of innovation. From being the pioneers of television-based educational programming in 1969, they were ready to showcase forward-thinking approaches to play-based learning on digital platforms to children and fans alike.
Tapping into a half-century’s worth of episodes, songs, and clips, we integrated AR to let kids transform into their favourite characters, such as Elmo, Grover, and Count Von Count, and be a part of their favourite video clips and songs.
Sesame Street Yourself
Weyo and DEPT® teamed up with Sesame Workshop to build an AR solution to make this dream a reality with “Sesame Street Yourself,” an app that uses AR technology to transform people’s faces—and facial movements—into animated versions of Sesame Street characters. Kids can interact and play with the likes of Bert, Ernie, Elmo, and Cookie Monster.
DEPT® built the engine by utilising the for these AR masks, along with a fully customised content management system that allows Weyo to manage all aspects of the app, including masks, costumes, songs, and games.
The engine behind the app
Weyo and DEPT® originally collaborated for an AR engine solution for Aussie kids’ entertainment icons’ The Wiggles: Fun Time with Faces App in January 2019. With Sesame Street Yourself, the engine levelled up. Using OpenCV and dlib, DEPT®’s engineers began experimenting with state of the art approaches to detecting the position of eyes, noses, ears and mouths on faces from video, hence rendering things to fit those positions in real-time.
The team wanted a fast solution, so they tried using mobile platform SDKs, but none of them were fast enough. Instead, they decided to create their own ML model called Kaiju, a cross-platform engine that works on iOS, Android, Linux, MacOS, and Windows. They created an Edit Decision List (EDL) format for controlling interactive content, like a video editor that processes text descriptions to show visuals.
To make everything work smoothly on low-end phones, the team built a special cross-platform runtime in Rust that runs the EDL files, detects faces, adds augmentations, and dynamic elements to the video in real-time.
Kaiju’s unique functionalities have led to the innovative play and features that bring the Sesame Street Yourself app to life. Users wear a 3D mask that is tracked to their head, watch the mask conform to their face shape and can trigger actions by opening their mouth. To add to the fun, users can also create and save recordings within the app and watch them back later.
Making an internationally-recognised impact
Sesame Street Yourself allows kids everywhere – including the kid in all of us – to become a part of the world we have grown up loving and learning from.
The app was recognised as the People’s Choice and Gold Telly winner in 2020 and Forbes 2019’s best AR app for children. It was also featured as part of Apple’s showcase event, WWDC in 2019. Since its launch, it has been awarded Apple’s “App of the Day” multiple times worldwide, including in the USA, UK and Australia.
Questions?
VP Games & Emerging Technology
Joey Egger
Discover more
