Imagine sitting on your couch next to your favorite living musician.
That’s the concept behind the immersive technology start-up BLANK XR, which recently unveiled plans for a platform that allows artists to chat directly with fans via holograms. The mixed-reality program is called IMMORTAL XR, and it’s launching as a smartphone and mixed-reality application in 2021, the company said.
The entertainment platform announcement comes amid what’s been a tumultuous year for the music industry, with live concert venues shuttered, and major summer festivals canceled. Ticketmaster operator Live Nation Entertainment saw its business drop more than 95 percent. A new report from the music publication Pollstar calculates the industry losses to be over $30 billion.
Live-streaming performances and music downloads brought in some revenue, but those experiences aren’t as captivating or money-generating as in-person concerts. That’s where innovative tech offerings like holograms and personalized digital concerts fit in, according to Denise White, CEO of BLANK XR and former director of direct-to-consumer technologies at the Walt Disney Company.
“From our point of view, the new normal is holographic,” White said. “What that will enable you to do is put on a headset and actually have a conversation with your favorite artists.”
Digitizing musicians and capturing their likeness involves a two-step process.
For a 3-D hologram, the company relies on 360-degree cameras that capture a person from all angles. The voice is cloned based on audio clips and artificial intelligence software. The more information the artist chooses to feed to the program, the more accurate their representation will be, the company said.
Essentially, it’s mixing artificial intelligence with video and sound manipulation.
Manchester, England-based BLANK XR modeled its concept demo product on Grimes, a Canadian musician with 1.6 million Instagram followers. Amid the pandemic, the pop star gave birth to a baby boy with billionaire Tesla CEO Elon Musk.
Over the summer, Grimes, whose real name is Claire Boucher, permitted creators to manipulate her voice and image using raw files she dumped on the Internet. By pulling an audio sample of Grimes’s voice and running it through the company’s “deep voice engine,” BLANK XR developed a synthetic voice that sounds a lot like the artist.
The artificial voice was then paired with some of the performer’s tweets, capturing her ideas and reading them aloud. The algorithm also analyzed the body of text for sentiment to generate more realistic tonal inflections.
The experience is not meant to replace the need for artists, the company said. In fact, celebrity participation is one of the building blocks behind the technology.
The company doesn’t need very much data for a short demo. But for an engaging, realistic hologram, it needs access to an evolving amount of text, audio and a full body scan from the artist. To re-create an artist’s look, IMMORTAL XR uses up to 108 high-definition 5K cameras to capture a data image and volumetric video.
“It's like stitching together a hologram and then putting the brain inside, that's what gives you that realistic experience where it feels as if you're speaking to that person on a one-to-one basis,” White said.
For users to have a personalized experience, participating artists will need to answer an evolving list of questions. That way, the company can feed that data into the algorithm that is meant to capture the performer's likeness.
“If you’re an artist who doesn’t spend a lot of time inside of the training tool, that is not going to be as intuitive of an experience for the end-user, as someone who’s in there every day,” said Mike Koss, chief technology officer at BLANK XR.
The platform will first come to Microsoft’s HoloLens 2 and smartphones toward the end of 2021, White said.
The company is in the process of recruiting performers who will share ownership of the holographic image. The service is primarily geared toward Gen Z and younger millennials who prioritize sensory experiences, White said.
“They’re not so much about having things as they are about doing things together,” White said.
The HoloLens is a $3,500 headset that allows wearers to interact with holograms that are superimposed into real-world environments. Unlike virtual reality, which is fully immersive, or augmented reality, which overlays virtual objects over the real world, mixed reality projects 3-D images that respond to your voice or gestures.
Augmented reality is used in games like Pokémon Go, where you can see the real world with characters layered on top. But in mixed reality, you can use gestures and voice to interact with the object or hologram. Entertainment headsets have yet to go mainstream, even with virtual-reality options such as Facebook’s Oculus Quest priced under $300. That is part of the reason the platform is also launching as a smartphone app, White said.
BLANK XR will roll out its musician experience with a $9.99 per month membership. Users will also be able to subscribe to individual artists and can choose to donate money to support them, White said.
Koss, a former cybersecurity officer at Walt Disney, said the company has a policy against creating holograms of dead musicians. “I’m not bringing anyone back from the dead,” Koss said.
On the company’s advisory board is Mike Pell, who leads Microsoft Garage, an innovations and new ideas program.
Bringing holographic artists to people’s living rooms is the first step in a 10-year plan to revolutionize retail and entertainment, White said. BLANK XR is laying out a road map to bring mixed-reality experiences to physical stores post-pandemic. Imagine taking Zoom meetings to the next level with holograms of people you work with or going on immersive video dates with people on the other side of the globe; that’s where things are headed, White said.
“The most refreshing part about working in extended reality is that we get the opportunity to do things for the first time,” White said. “In 2030, when people have moved beyond smartphones, you probably won’t need a big clunky headset. You’ll probably have a cool pair of Apple glasses, and you’re certainly going to have a holographic representation.”