Mo-Sys and Vizrt combine to create augmented reality at Plazamedia

Plazamedia uses the latest technology including augmented reality to increase audience engagement.

Munich-based Plazamedia is one of Europe’s leading sports production companies. To support its production activities, it has eight studios of various sizes, with all the latest technology including augmented reality to increase audience engagement.

For its current presentation of Champions League football for German-speaking Europe (Austria and Switzerland as well as Germany). Plazamedia included augmented reality to give audiences a clearer understanding of the action and more involvement. The augmented reality content comes from Vizrt graphics engines, using models and materials designed in-house.

The augmented elements include virtual video screens and Jumbotrons dropping from the ceiling, player cut-outs and models of pitches to show statistics. The system is also used to bring in social media comments, which appear on virtual boards in the studio.

The challenge for live augmented reality production is to ensure that the virtual material stays in the same place relative to the live video. When presenters are moving around, as they would be to interact with football diagrams and charts, then the camera has to be able to follow them.

This means the camera has to report to the Vizrt graphics engine, in real time, where it is and which way it is pointing. Achieving this to the required level of accuracy is a significant issue. Getting sub-pixel accuracy in high definition requires sophisticated tracking.

Plazamedia had experience with mechanically-encoded pedestals, which essentially use dead reckoning from a central reference point to provide location data. In use these systems tend to drift, meaning their virtual position separates from their actual position over time.

The company needed something which would not drift, and therefore would not limit creativity. There have been a number of different technologies proposed over the years to track cameras in live productions. Some of these require costly cameras; others require markers with unique patterns, mounted under the lighting grid.

The disadvantage of systems using ceiling codes, for instance, was that they required a detailed measuring and mapping process before the system can be used, a time-consuming process. The need to have the codes below the level of the lights can cause issues and limit creativity.

Plazamedia needed a system that was accurate, fast to set up, and did not compromise the operation of the studio. It investigated all available technologies, and decided to work with Mo-Sys and its StarTracker technology. At this stage StarTracker was still in development, and Plazamedia and Vizrt contributed significantly to the work of maturing the system through testing and discussions in Plazamedia studios.

StarTracker, too, uses a pattern of markers on the ceiling, but there are considerable differences to earlier technologies.

First, the markers are not codes but simple reflective stickers. They are fixed to the ceiling itself, so need no additional hardware to mount them. The “stars” are just reflectors, placed in a random pattern.

Second, because they are applied to the ceiling they are above the level of the lights so cause no problems. Removing constraints on lighting was one of the primary design considerations for Mo-Sys.

Third, they can – and indeed need to be – fitted at random, so work around other services on the ceiling.

The broadcast camera has a spotter sensor pointed upwards. This has an LED ring around the lens, which gives enough light to bounce off the “stars” – the retro-reflective markers on the ceiling. This light has no visual impact as far as the broadcast cameras are concerned.

The system learns the positions of the stars virtually instantly, then from the pattern seen by the camera at any time, the processor can calculate the position of the camera in three dimensional space, plus the angles of pitch and roll. Through digital interfaces to the lens, the stream of positional data also includes the zoom and focus settings.

“With the Mo-Sys StarTracker we are able to get all the parameters of the camera tracked in a very solid manner,” said Michael Ermair, head of graphics technology at Plazamedia. “For Champions League we have one camera on a Vinten Quattro pedestal and the other on a Mo-Sys e-Crane.

“The pedestal and the crane can move around in all axes while on air, and deliver exact positional data to the Vizrt tracking interface,” he explained. “With that data we can simulate the exact movements in the realtime graphics engine, and create augmented reality that sticks to its locations without drifting away.”

While the Mo-Sys processor outputs positional data in a standard format, the company has worked closely with Vizrt over recent years to perfect a very tight integration. The result is that setting up and starting is virtually instantaneous, which is particularly important in a busy studio that is responsible for a number of fast turnaround productions a day. The right suite of graphics can be called up in the Viz Engine, with positional information available as soon as the StarTracker software has booted.

“You turn them on at the beginning of production and start working without worrying about stability, drift or re-homing,” according to Ermair. “It just tracks.”

Plazamedia’s audiences react extremely positively to the augmented reality elements in the presentation. “Augmented reality – integrating virtual objects in a real environment – offers an impressive way to work with graphics and information. There is a lot of potential for using augmented reality in many different applications.”

At Plazamedia, it is clear that the combination of Vizrt graphics and Mo-Sys camera tracking is delivering a real benefit, allowing producers to enhance their programmes without adding to studio time, thereby increasing their engagement with audiences.