Technology
©T.Tairadate/MUZA
Technology that Expands the Classical Music Experience
The mainstream ways to appreciate music have typically been limited to two options: a live performance in a concert hall or listening to recorded audio. However, with the evolution of technology, the era has arrived where the “experience” of music itself can be expanded to be deeper and more interactive.
The mission of the ポルタメタ -portameta- project is to push the boundaries of musical expression through the power of technology and build a new relationship between artists and the audience.
We introduce two technical challenges that led to the success of the “World's First Real-Time Concerto” by a virtual artist and a full orchestra, realized at Muza Kawasaki Symphony Hall on August 12, 2024.
CHALLENGE 1
For a virtual artist to truly “perform,” merely existing as an image is insufficient. We challenged ourselves to completely reproduce every movement that embodies the musician's soul—especially the rapid finger movements characteristic of a pianist's virtuosity—using a 3D model.
Technical Challenge
Virtual artist is a new type of artist who is not bound by anything other than his or her own performance skills.Using the latest model development and streaming technologies, virtual artists are able to maximize their own performance technique and musical expression, regardless of their appearance, gender, nationality, age, education, or awards.They are a breath of fresh air in the world of classical music.
ポルタメタ -portameta- Solution
To overcome this challenge, we introduced a high-definition motion capture system capable of precisely capturing the movement of every single fingertip. Furthermore, we developed a “Real-Time Fingering Reflection System” that combines three technologies: spatial position information from cameras, the motion capture data, and MIDI signals. This system allowed the virtual artist to exist on stage not merely as an image, but as a realistic “entity” imbued with technology and soul.
To enable the virtual artist's realistic performance, this system accurately captures the performer's movements, smoothly reproduces the movements of the virtual pianist's fingers and the keys in real-time, and minimizes audio and visual latency.
Through this technology, the audience was able to experience the pianist's extraordinary technique visually and profoundly, from a perspective and resolution that could never be seen from the audience seats.
©T.Tairadate/MUZA
Technical Challenge
We needed to achieve musical communication between the conductor, dozens of orchestra players, and the virtual pianist with ultra-low latency that completely surpasses human perceptual limits. This required a system design to perfectly synchronize, in milliseconds, two complex signal chains: motion capture to CG rendering and video output to the venue's screen, simultaneously fusing the live piano sound with the orchestra's acoustics via the PA system.
ポルタメタ -portameta- Solution
We built a proprietary system architecture that governs the synchronization of video and audio. We thoroughly optimized the placement of on-stage monitors and the acoustic design of the monitor speakers so that the conductor and musicians could naturally perceive the virtual soloist as a “collaborator who is physically present.”
As a result, we proved that the essential musical communication—where a virtual artist and a living orchestra listen to each other's sound, watch the conductor's baton, and create a single piece of music—can be established beyond the boundaries of the real and the virtual.
These technologies lead to a newer experience of classical music.
The era has arrived where talent from all over the world can collaborate freely, transcending the constraints of body and location. The ポルタメタ -portameta- project will continue to pursue the fusion of technology and art, and expand the possibilities of music.
Top
