XBOX SERIES X

Creating Daniel Kaluuya’s Digital Double for Xbox Series X

Daniel Kaluuya’s performance was recreated in stunning CG without the need for extensive re-animation.

Power your dreams – the message underlying 215 McCann and Ian Pon’s Xbox Series X advertisement that sees a digital double of acclaimed actor, Daniel Kaluuya (Get Out, Black Panther), traverse imaginary space and visit distant planets containing characters from some of Microsoft’s biggest games. To ensure Daniel’s CG appearance was as realistic and consistent as possible, MPC (Moving Picture Company) – producing all of the VFX for the commercial – turned to DI4D to provide precise, 4D facial capture sessions, tracking Daniel’s facial performance to drive his digital double. 

Enter your favourite gaming worlds

To emphasize the visual experiences made possible with the new Xbox Series X console, the animated advertisement follows Daniel’s CG counterpart as he experiences the intersecting dreamworlds of various Xbox video games, including Halo, Assassins Creed and Medieval Vikings. MPC used a range of software and hardware solutions to achieve the most engaging result possible. 

For example, the cityscape – one of the dream worlds that stems from Daniel’s imagination – takes on the form of bismuth crystallization. MPC obtained bismuth crystals, took a range of photographs and made scans of them using photogrammetry software to make models and mimic them. But beyond the detail found in the environment, MPC turned their attention to Daniel’s on-screen digital double. 

Faithfully capturing Daniel Kaluuya’s facial performance

The accuracy of Daniel’s facial animation was vital to ensure that closeup shots of the face consistently matched up with the high level of fidelity in the environments and game characters. Much like using motion capture technology for full-body tracking, the most efficient and precise way of achieving lifelike accuracy in facial animation is through the use of 4D facial capture. 

Glasgow-based 4D facial capture company DI4D has worked with MPC on a number of projects, becoming the VFX studio’s primary facial capture solution and go-to for digital humans due to its consistency and accuracy. The DI4D team was on-hand again for this Xbox series X commercial. DI4D has two primary facial capture technologies, the DI4D PRO system and the HMC (head-mounted camera). 

4D Facial capture pipeline

For this project,Daniel’s facial expressions were first captured with the DI4D PRO system at DI4D’s studio in Playa Vista, Los Angeles, California. Nine, synchronised, 12-megapixel machine-vision cameras captured Daniel’s facial movements as he performed a wide range of different emotions and facial expressions under standard video lighting. DI4D’s advanced 4D software  was used to reconstruct a high fidelity 3D scan of Daniel’s face per frame of captured data. A dense mesh was then tracked  through this data with high precision and consistency, and without requiring any markers or makeup to be applied. This data was used by MPC to create a highly realistic animation rig for Daniel’s face.

Next, Daniel’s facial performance was captured with a HMC (head mounted camera) system fitted with two miniature synchronized machine vision cameras. By using an HMC system to capture Daniel’s performance, he was given the  freedom needed to act out the scenes convincingly. DI4D’s advanced 4D software was used again to process the stereo video data acquired by the HMC to produce pointcache animation.

The final animation was produced by applying the HMC pointcache animation to the rig created from the DI4D PRO data, achieving the highest fidelity possible while also remaining extremely faithful to Daniel’s original performance. The resulting data was used throughout the advertisement to drive the facial animation during all scenes with Daniel’s digital double.

Adapting workflows

With social distancing measures and restrictions in place during both the shoot and the production, MPC had to adapt from its traditional workflows and take advantage of new remote post-production techniques. Using the Unreal Engine to produce the advertisement, the director and clients could all connect remotely and make creative decisions in real-time. This included using virtual production techniques and real-time pre-visualisation to speed up the production process and provide more creative control. It was the first time that MPC had used a remote setup like this to produce an advertisement with such a high production value. 

Dan Seddon, VFX Supervisor MPC LA said, “DI4D provides high fidelity facial capture delivered in a tight time frame and an easily usable format. They are our go to solution for facial capture on digital human projects. Working with Colin is always a pleasure, and they are collaborative in pushing the boundaries of what is possible in digital human creation. It is no surprise when I read about new groundbreaking digital human work that DI4D are behind the facial capture”.

The final advertisement was a commercial success, demonstrating both the high level of graphical fidelity possible using modern real-time engines and the effectiveness of 4D facial capture as a reliable and accurate method for producing realistic facial animation for digital doubles. By using 4D facial capture, it was possible for MPC to recreate Daniel Kaluuya’s exact facial performance in CG, without the need for extensive hand animation. The advertisement also provides a stunning  vision of how these same advanced techniques could be used in future to create video games themselves for consoles such as Xbox Series X.