Story 2: A New Age in Experience Value Engineered by Sports x Technology

Photo: Scene of the opening performance at the “1 Year to Go!” ceremony for the Olympic Games Tokyo 2020 using real-time tracking and projection mapping technology using projectors.

With the Olympic and Paralympic Games Tokyo 2020 just a year away, a “1 Year to Go!” ceremony was held on July 24, 2019. It opened with a stunning performance that was engineered and rendered with Panasonic’s real-time tracking and projection mapping technology and high-luminance projectors. The innovative production showed a captivated audience the potential of fusing sports and technology.

Summary of this article:

  • The opening performance at the "1 Year to Go!" ceremony for the Olympic Games Tokyo 2020 was engineered with real-time tracking and projection mapping technology using projectors.
  • The system was developed by Panasonic, an Official Worldwide Partner of the Olympic Games Tokyo 2020, and opens the door to new performances choreographed for speed.
  • Those involved shared their thoughts on how interaction between imagery and human motion is accelerating the fusion of sports and technology.

A spectacular opening performance

Some 4,000 people packed Hall A of the Tokyo International Forum, waiting anxiously for the show to begin. Junior high and high school students seated in the front rows posed for commemorative photos. Stewards of the next generation, these boys and girls will experience the Olympic Games Tokyo 2020 during their most impressionable years. So, what's not to get excited about?

Suddenly, the stage went dark. The audience looked on as shamisen virtuosos the Yoshida Brothers performed an energizing duet. The brothers were then whisked to opposite sides of the stage to make way for a mesh screen.

Behind the screen, rhythmic gymnast Honami Tsuboi began dancing across the stage like a fairy as the Yoshida Brothers played. The tip of the wand she held in her right hand painted a living, glowing trail of light that followed her movement. The dynamic Japanese imagery was fused into the dance using high-definition 3D projection mapping technology. The loud applause at the end showed just how amazed the audience was.

Most people probably thought—logically—that Tsuboi danced beautifully in tune with the images. It was actually the opposite: the images followed Tsuboi's motions in real time. The technology was developed by Panasonic, and this article goes behind the scenes—so to speak—to discover how real-time tracking and projection mapping was born and refined.

Technology has the power to change sports

Panasonic’s real-time tracking and projection mapping technology and high-luminance projectors had been announced in March 2019 at the "TOKYO VISION: 500 Days to Go! Night," a 4-day event planned 500 days out from the Olympic and Paralympic Games Tokyo 2020.

"As is laid out in our vision of next year’s Olympic Games, we want to bring unprecedented innovation to the Games and bring about positive change in the world,” explained Hiroyuki Kobayashi, Tokyo Olympic Organizing Committee senior director of events and promotions. “Our goal is to create new value beyond just the sports component, by mixing in technology and culture. To fuse them, we decided to try using real-time tracking and compatible projectors."

Kobayashi rated the performance as "powerful on a whole new level in terms of the imagery, acoustics and movement." New content had been created by fusing together the disparate elements of shamisen, rhythmic gymnastics and the latest in imaging technology.

"Technological advances are creating opportunities for more people to appreciate sports,” he added. “Not just for playing but watching as well. We also hope to spark interest by conveying the fun and attraction of sports through the Olympic Games."

"We believe sports can change the future and technology can change sports. So, we expect something beyond our imagination to come from combining the two."

Real-time tracking with a delay of less than 0.002 seconds

The system was developed by the Media Entertainment Business Division at Panasonic's Connected Solutions Company. The division makes projectors, flat panels, broadcasting equipment, sound equipment, conference systems and other BtoB solutions.

Division member Kenji Fujiune recounted, "We started developing the system in 2015, bearing in mind that it was a new challenge in imaging and that we would be pushing speeds as far as they could go to enhance the user experience."

Projector picture quality and brightness have both improved. In terms of picture quality, there are SD, HD, Full HD and 4K products, while some high-end commercial projectors have reached 30,000 lumens in brightness. Panasonic's focus on speed challenged the traditional industry approach to development.

According to Hirotatsu Asai of Connected Solutions Company, Panasonic Corporation, "With projection mapping everywhere these days, people are tired of looking at just images projected on a wall. Real-time tracking and projection mapping integrate performances and imagery into something lifelike."

The system consists of a projector, an infrared light for sensing and a high-speed camera. The infrared light is radiated on the marker at the tip of the performer's wand and detected by the camera. The location of the marker in space is then pinpointed with a virtually imperceptible time lag by feeding the captured image into a processor within the projector, without going through a PC. Based on this positional information, images are instantly projected at 1,920 frames per second (fps). This configuration allows real-time tracking with a lag of a mere 0.0016 seconds from detection to projection.

Image: Real-time tracking with a time lag of less than 2/1000 sec from the detection of positional information to content projection is rendered with a high-speed projector that processes information without going through a PC.
Real-time tracking with a time lag of less than 2/1000 sec from the detection of positional information to content projection is rendered with a high-speed projector that processes information without going through a PC.
Photo: Four high-speed projectors were used for the opening performance at the 1 Year to Go! ceremony.
Four high-speed projectors were used for the opening performance at the "1 Year to Go!" ceremony.

Asai noted, "You cannot detect the time lag. Since a typical projector is 60 fps, the smoothness of the images is remarkable."

"At the 'TOKYO VISION: 500 Days to Go! Night' event, dancer Kento Mori performed three days in a row. But what struck me was when he said it was a lot of fun to change up his dancing. Since the images track his movements, I gather he slightly changed his routine," added Fujiune.

Though the same technology as the "TOKYO VISION: 500 Days to Go! Night" event was used for the opening performance of the "1 Year to Go!" ceremony, the screen for the latter was four times larger. And four projectors (including a backup) were employed to enhance the brightness. "The system blurs the boundaries between the real and virtual worlds,” said Fujiune. “It augments the 3D effect and blending sensation by tracking high-definition, high-brightness images as an added component, enabling higher performance. This performance perfectly demonstrated that power of expression."

Looking to the future, Asai said, "If further evolved, the system could possibly project images on the walls and floors in someone's home for a more immersive experience. Moreover, advances in technology may create new sports. This system takes us one step closer to realizing those kinds of dreams."

With virtual reality, the requirement of goggles makes it a solitary experience, but with real-time tracking and projection mapping, multiple people can share the experience. Users who tried using the tracking feature with ping-pong, soccer and other sports responded favorably. The fusion of sports and technology has already begun via the interaction between high-speed tracking and human motion.

The Olympic Games are closing in fast

The "1 Year to Go!" ceremony was a great success, but it was just one stop along the way. Come April 2020, the Olympic Torch Relay will make its way to Japan and the TOKYO 2020 NIPPON FESTIVAL will start.

"The Olympic Games are one opportunity for us to continue developing sports and entertainment,” said Fujiune. “We can firmly support those foundations with our technology." Asai noted that "These will be the first Olympic Games since Panasonic celebrated its 100th anniversary. We want to make the biggest technological impact ever on the Games."

New events like karate, skateboarding and surfing have been added to the Olympic Games Tokyo 2020. It’s exciting to imagine the adoption of technology-derived sports as new events.

Photo: Real-time tracking and projection mapping make the viewer think that the dancer is moving in sync with the images.
Real-time tracking and projection mapping make the viewer think that the dancer is moving in sync with the images.

A capable creative team to support the content

Montreal, Canada-based Moment Factory, founded in 2001 as a digital art group focused on visual imagery, oversaw the “1 Year to Go!” opening ceremony image content.

Moment Factory pursues an organic reality where large groups of people can share the excitement. The company has been exchanging and polishing ideas with Panasonic’s development team since 2017. That all came together as the content for the "TOKYO VISION: 500 Days to Go! Night" and "1 Year to Go!" ceremony.

Moment Factory Co-Founder Dominic Audet commented, "The concept behind the content was to achieve a collaboration between tradition and state-of-the-art technology. We reached new ground where athletes aren't bound by limitations." Looking back, Fujiune said, "In working with their creators, we discussed what everyone wanted to create, what people would enjoy and what would be best as a performance. It was a very motivating experience."

Moment Factory has its sights set on pioneering augmented sports. "The big goal is to fuse technology and actual sports like was done with ping-pong and soccer at the 'TOKYO VISION: 500 Days to Go! Night,'" shared Audet. Asai is looking forward to playing tag more with his cohorts from Moment Factory. "In this kind of back-and-forth collaboration, both sides stimulate each other, which ultimately should lead to new modes of expression."

- Source: Adroit Visuals That Track an Athlete's Movements - A New Age in Experience Value Engineered by Sports x Technology | Towards 2020, Beyond 2020 |  [Global Site]