F1 2021 Braking Point

F1 2021 Braking Point

REALTIME harnessed the power of PURE4D to drive the facial animation of five digital doubles and one digital character for the story mode cutscenes in Codemasters’ F1^® 2021. The finished game contained one hour of facial animation, making it the largest video game project processed by DI4D to date.

Breathing Life Into F1

F1 2021 features an immersive new story mode called ‘Braking Point’, allowing players to rise from the ranks of Formula 2 and take a shot at stardom in Formula 1^®. To create a truly authentic world away from the track, REALTIME needed to ensure that the in-game cutscenes featured believable characters that were as close to real-world people as is digitally possible. To achieve this, REALTIME chose DI4D’s new digital double animation pipeline, PURE4D, to track and recreate the exact facial performance of each character’s actor.

The six characters include two competing drivers as the main protagonists: one is at the end of his career and the other is a new rookie attempting to solidify his name on the world stage. Further characters include the team manager, a journalist, and a driver’s wife – all five of whom are exact digital doubles with actors cast for their likeness, animated using PURE4D. A fan favourite and long-time F1 character, Devon Butler, was also animated in part using DI4D’s 4D facial performance data.

PURE4D

The PURE4D pipeline works by combining accurate facial performance data captured with DI4D HMC systems with even higher fidelity expression data captured with the DI4D PRO system. Both sets of data are processed using advanced machine learning algorithms to increase efficiency, enabling its use on larger projects. F1 2021 is the first game to utilise the full capabilities of PURE4D. 

All of the facial and performance data was captured in a single week at Rebellion Film Studios in Didcot with Audiomotion – the studio is an old converted paper factory with a dedicated performance-capture stage.

DI4D PRO

Seated in front of the DI4D PRO system, the actors performed one minute of scripted facial expressions spanning 60 shapes. Additionally, phonetic pangrams – sentences designed to use as many consonances and sounds as possible – were captured from each actor. Actors were provided with a video of the pangrams for reference.

Each actor performed these facial performance sequences in character. For example, one of the American characters is played by an English actress, requiring her to use an authentic American accent. This is to ensure the most accurate data is captured, matching the final performance data recorded with the HMC. 

It took no more than 15 minutes of each actor’s time to capture the required DI4D PRO expression data, and all during breaks in the performance capture shoots to maximise efficiency on-set. 

DI4D HMC

The DI4D HMC (Head-mounted Camera) is a wireless headset fitted with a stereo pair of 4MP cameras, designed to track accurate facial performance data simultaneously with full-body motion capture, audio recording and reference video. All of the data streams are recorded with the same timecode to ensure that correct synchronisation is maintained. The full process is referred to as performance capture, enabling multiple actors to be captured as they roam and interact with each other freely. 

For the F1 2021 performance shoots, the Vicon optical marker-based motion capture system was used to obtain full-body motion data. Audio was captured by Codemasters using a “lav” mic fixed onto each DI4D HMC and connected to a recorder attached to the HMCs’ harnesses. With each session taking place on a large performance capture stage, multiple actors were recorded simultaneously.

Using the DI4D HMC, REALTIME captured the exact facial performance of each actor as it would appear in the final game’s cutscene. This revolutionary approach allowed the studio to drive the facial animation directly without building extensive animation rigs. The actors were free to perform their character’s dialogue and motion in the same way they would perform on-set for movie or television and REALTIME didn’t need to complete any further animation in post-production. Nevertheless, the final level of fidelity and realism was incredibly high. This was also the first time the HMC data had been used to track the exact movement of each actor’s eyes. Eye and jaw tracking are both new features included with PURE4D.

Machine Learning

PURE4D’s cutting-edge machine learning technology was used to efficiently process the DI4D HMC data. The resulting 4D facial performance data was then used to drive the DI4D PRO digital double assets, producing accurate facial animation and conveying authentic facial expressions in high fidelity.

The final F1 game required one hour of 4D facial capture data, which was significantly more than any previous project processed by DI4D. Nevertheless, by using PURE4D’s advanced machine learning algorithms, DI4D was able to complete the post-processing for the project in only two months – while also improving the quality of the final animation. 

PURE4D not only produces dense, high-fidelity 4D animation data, but it represents it in an incredibly lightweight form, meaning that it can run seamlessly in modern game engines. This allowed REALTIME to render the final cutscenes for F1 2021 completely from Unreal, and still achieve 120 frames per second, realistic digital human animation both in-game and during cinematics.

This is the first large-scale video game project to use PURE4D and it’s been an incredible success. We managed to both raise the level of fidelity and accuracy of the digital doubles appearing in F1 2021, and lower the amount of processing time required. We’ve also demonstrated an end result that runs seamlessly in the Unreal Engine, opening up new possibilities for game developers looking to drive realistic digital double animation in-game – PURE4D is a huge breakthrough for DI4D.

Douglas Green, Co-Founder & COO, DI4D

Game Changer for Facial Capture

 PURE4D transformed the creative process for REALTIME, providing them with one hour of accurate, high-fidelity facial performance data at incredible speed and at an unprecedented level of quality for a video game.

We knew from the beginning that F1 2021’s story mode would benefit hugely from the use of digital doubles based on real-world actors. After speaking with the team at DI4D, they introduced us to their new pipeline, PURE4D, which took a unique approach to tracking facial performance and driving facial animation. It has been a truly transformative process, speeding up our pipeline significantly and improving the final result.

Ian Jones CG Director at REALTIME