High-definition Face Tracking and Face Geometry Modelling for Kinect 2

February 14, 2014 — Leave a comment

My team at Microsoft shipped the high-definition real-time face tracking and face shape modelling tech as part of XBox One Kinect back in November 2013 (also as part of Kinect For Windows 2). I contributed a lot of algorithms/code to the 3D face tracker and face geometry computation. More information can be found in my paper.

Here are some facial expressions that can be tracked by Kinect 2 in real-time and animated in 3D (gray masks) with pretty good precision:


The system finds a large number of semantic points on a user’s face in 3D (3D model has ~1200 vertices with ~80 salient points). The developers can then use this information to animate avatars or do some other processing (like recognize expressions, etc.).

This video shows the early prototype of this technology that I made back in 2012:

This is a bit more technical video that demos the 3D mask painted over the video stream and computed facial animation units:

The algorithm that we created is a fusion of a 2D video feature tracker based on Active Appearance Models (AAM) and a 3D tracker based on Iterative Closest Point (ICP) algorithm, which aligns 3D face model based on depth data from Kinect. Both trackers are resolved together and linked by special 2D-3D constraints such that the alignment makes sense and looks natural. We are going to publish details as a paper some time soon. We also compute 3D face shapes for a given user from a set of input frames (RGB + depth).

We published this paper that describes our face tracking and face geometry modeling algorithms. The resulting 3D models are pretty realistic and look like this:




Now it is up to game developers to use this tech in their titles! The API is available as part of Xbox One XDK. For example you can produce NPC faces like this in your game:

Nikolai as a textured shape model

Nikolai as a textured shape model


The 3D model that we shipped in Xbox One Kinect + face capture system is very capable and flexible. The face tracking and face shape modelling is used in just released Kinect Sports Rivals game by RARE studio. You can see the face shape modelling session in this game video:


You can also do alien effects like this 🙂


No Comments

Be the first to start the conversation!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s