NRK's augmented reality graphics World Cup studio

Norwegian national broadcaster NRK used augmented reality (AR) graphics in the studio during the World Cup (skiing) in Falun Sweden.

Originally published in Norwegian by NRK.

During the World Cup at Falun, NRK used augmented reality (AR) graphics in the studio for the evening broadcasts with Anne Rimmen and Thomas Alsgaard. This was the first time broadcasting with a new tracking system.

What is AR?

Television production distinguishes between augmented reality (AR) and virtual reality (VR). The traditional form of VR is when a presenter is keyed or cut out and placed into a graphic generated virtual studio. This is used in NRK programs such as SuperNytt.

In a lot of ways, AR is the opposite of VR: Instead of placing a real object into a graphical world, you place a graphic element into the real world. The goal is to create the illusion that the graphics exist in the physical TV studio.

Traditional TV graphics are placed on top of the camera images and do not react to changes in the image. With AR, the graphics retain their position in the room when the camera moves. You could, for example, place a graphic element on a table, and the graphic will keep its position on the table when the camera moves in the studio. To achieve this, the graphics system needs to know how the physical camera in the studio moves. This is known as “tracking”.

How does it work?

AR graphics work by having the real camera movements matched up with a virtual camera that is internal in the graphics rendering software. If the real camera is moving to the left and zooms in, the internal virtual camera in the graphics engine does the same. Viewers get the impression that the graphics and the live images are connected.  To achieve this camera tracking, all you need is a system for continuous reading of the camera's position; pan, tilt, zoom, focus, height and so on.

In this production NRK used for the first time a tracking system from Stype GRIP. This is a mechanical tracking system for the Stanton Jimmy Jib camera crane. The Stype GRIP tracking system is connected to the crane and includes a pair of sensors on the crane base and sensors on the crane arm. This gives continuous data on the pan and tilt of the crane arm, and information on the pan and tilt of the camera. In addition, decoders on the camera read focus and zoom. Data from Stype GRIP is then sent over ethernet cable to the graphics engine to interpret the data stream.

At NRK we use tools from the Norwegian company Vizrt in most productions. They provide a real-time 3D graphics system (Viz Engine) that allows us to generate everything from simple lower thirds, to virtual graphics. We use the design software Viz Artist to create 3D elements.

We typically manually check the virtual cameras in Viz Artist to display graphics from the desired angle. However, to view AR graphics, we let the virtual cameras be controlled by the movements of the tracker camera in the studio. This requires extensive calibration of camera position, lens and focus.

Real-time graphics

A challenge of AR graphics is that they must be generated in real-time. When we produce vignettes and other heavier graphics, it is not uncommon to spend hours or days to render seconds with graphics. This is a luxury we don’t have when working with real-time graphics. We must deal with the CPU and GPU on the graphics machine to render in real-time. This means that we cannot use motion blur, global illumination, reflections, shadows and other effects to make the model as realistic as possible.

What did we design?

NRK introduced the new design package for NRK Sport in the beginning of this year's winter season. This design was mainly the graphics package that we developed for broadcasts from the World Cup studio. We based the graphics designed in Viz Artist on the light bands from NRK Sport.

These designs were used as the background for individual photos, schedule items, Instagram pictures, Twitter messages, head-to-head images and logos. We also made artificial reflections in the floor to sell the illusion that the graphics were actually was in the room.

We also made a wall to display video and medal statistics that was inspired by the Swedish Viasat virtual wall. In close collaboration with stage designer Audun Stjern, we got drawings of the World Cup studio and where we would place each virtual item.

The elements were then recreated in 3D. Through a painstaking calibration process in Falun, these elements were 3D mapped against the real elements in the studio. Type color and lighting were fine tuned so that the 3D elements appeared as a seamless part of the stage design.

We produced a virtual space behind the graphic wall. The idea being that the wall would open up and show a landscape that continued into the stage design. This is to create depth and parallax between the foreground and the background wall.

The opening shot

We wanted an entrance to the studio after the show opening animations and mood videos. To achieve this, we set up a system that seamlessly shifts from the camera animation in the opening animation, to the graphics system used for tracking camera movement. In the studio, we put up a camera on the wall that sent a video input to the graphics system. We could then play the video through opening graphics system.

When the opening approached the end, our graphics operator, Chris Nicolay Wernersen, could trigger a transition between the two cameras. The camera would then move backward and "expose" the video we had just seen, which was actually in a virtual environment. When the "World Cup studio in Falun" logo revealed, it was in the actual studio with Anne Rimmen and Thomas Alsgaard. Crane operator Jan Erik Finsæther started a movement with the real camera on the crane at the same speed as the transition between these two cameras and the result was a completely fluid transition. Here it was important to have good timing. Too early and quickly meant that the graphics would not sit perfectly calibrated to the wall. Too late meant that the motion would abruptly stop.

The terrain model

We also constructed a 3D model of the terrain at Lugnet ski stadium in Falun where the events were held. We used this to display the trail profile for cross-country skiing and combined. We also added models of jumps in Falun. For the models to be useful, we were dependent on accurate models showing all the elevations in the area, including the infamous "Mørdarbakken."

It was difficult to obtain the elevation data we needed, known as DEM (digital elevation model) data, in a high enough resolution for the site in Falun. NRK has access to the elevation data for the entire world with a 30 meter grid spacing, and in Norway we have access to 5 meter elevation data from the Mapping Authority.

Measuring the elevation with 30 meter accuracy is too rough for such a small area. The Swedish mapping and land registration authority, Lantmäteriet, have laser scanned elevation data for all of Sweden with a resolution of 2 meters, but unfortunately this data is not freely available.

We eventually found publicly available map data raster images with contours down to 5 feet for the area we wanted. This meant that we could create vector-based map data with contours of the Lugnet ski stadium.

We converted this model to gray scale, where black is the lowest point on the map, and white is highest. This gave us a gray scale we could take into the 3D application CINEMA 4D and use as a displacement map.

The 14 different slopes that were used during the championship were projected onto this model, animated, and exported as 3D object sequences from Viz Artist.

Experiences

NRK was one of the first broadcasters in the world to use complex AR graphics on TV during the 2006 Olympics in Turin.

Since that time, NRK has been focused on virtual studios. This was one of the first attempts of using AR graphics since 2006. This pilot project by NRK has given us a wide range of experiences for both applications and technical solutions.

"Working with AR graphics requires a closer cooperation between the crane operator and graphics. You have to have a dialogue about how you add graphics and go through trial and error to see how it works best. This is a new and fun way to work on overcoming challenges," said Jan Erik Finsæther, crane operator.

It is important to find the right application for this technology and identify the places where it can provide an added value to the production, both visually and informative. Sometimes a regular full screen graphic provides the same benefits as AR graphics. Finding the right combination of traditional lower-thirds, full screen graphics, displays in the studio, and AR graphics, is a balancing act we will continue to work for in future productions.