Augmented Reality Makes Musical Entrance at This Year’s Grammys

This year’s Grammy awards in early April will have more than the usual performances by nominated awards winners and the pomp and circumstance of the awards ceremonies. Thanks to innovative augmented reality technology by a young company called M Entertainment Works (MEW), those fortunate enough to attend the festivities live will be to witness a virtual performance by legendary Blues musician Robert Johnson, who has been recreated as a 3-D avatar.

Traditional motion capture technology in 3-D presentations uses specialized 3-D motion suits. MEW has developed algorithms that stitch elements of existing images and videos of the desired subject together to create 3-D motion sequences.

The augmented reality technology goes beyond the subject’s body language during action sequences to their personal effects as well. “We have the technology to use the subject’s clothing and put it in the avatar, and create additional clothing as well,” said Taku Kodaira, CEO at MEW, during a recent interview with Design News. “When the avatar is moving, so is the clothing as the simulations create a stream of movement.”

Blues musician Robert Johnson was chosen to be the first subject the technology was developed to demonstrate, as he was a favorite of MEW’s director, Hitoshi Miyata, who is also a musician. Johnson is considered the “King of Delta Blues” and influenced other blues musicians, as well as the Rolling Stones and Eric Clapton. Johnson wrote 29 songs and died at the age of 27, so there were few images of him performing.

MEW’s augmented reality technology tries to overcome that limitation through the magic of augmented reality. At the Grammy Awards, guests can exclusively experience the show through iPads in a lounge area. Downloading an accompanying app,  the guests will be then able to view Robert Johnson playing songs and interacting through their smartphone. 

Kodaira hopes to offer the virtual experience to participants through a nominal fee, available through blockchain, that he considers analogous to purchasing tickets for a movie or concert. He thinks a marketplace could eventually be set up where users could exchange and trade the “ticket” to view the virtual experiences, with a small transaction fee, to help monetize the development and use of the technology.

According to Kodaira, MEW’s company partners spoke to Johnson’s family to get permission to feature him in the technology promotion at the Grammy ceremony. He added that while no other musician has so far been recreated with the technology, the possibility is there in the future.

While the augmented reality technology demonstration was set up to illustrate a musician that is known in some circles, Kodaira said one reason the technology was developed is to help somewhat lesser-known artists, such as dancers whose careers typically fade in their ‘30s due to the physical demands of the profession. Kodaira’s wife is a dancer.

Kodaira suggested that dancers can use the technology to illustrate themselves in motion and create online performances that might eventually be monetized. Kodaira added that the sports industry has expressed interest in this technology.

Beyond entertainment, Kodaira believes the motion capture technology could be adapted for uses such as medical and industrial training.