Archives For XBox

My team at Microsoft shipped the high definition face tracking and face shape modelling tech as part of XBox One Kinect back in November 2013. These are some facial expressions that can be tracked by Kinect in real time and animated in 3D (gray masks) with pretty good precision:

HDFaceTrackingSamples

 

The system finds a big number of semantic points on a user’s face in 3D. The developers can then use this information to animate avatars or do some other processing (like recognize expressions, etc.). This video shows the early prototype of this technology that I made back in 2012:     This is a bit more technical video that demos the 3D mask painted over the video stream (for those who like this stuff):     The algorithm that we created is a fusion of a 2D video feature tracker based on Active Appearance Model (AAM) and a 3D tracker based on Iterative Closest Point (ICP) algorithm, which aligns 3D face model based on depth data from Kinect. Both trackers are resolved together and linked by special 2D-3D constraints such that the alignment makes sense and looks natural. We are going to publish details as a paper some time soon. We also compute 3D face shapes for a given user from a set of input frames (RGB + depth).

We published this paper that describes the face tracking and modeling algorithms in details. The resulting 3D models are pretty realistic and look like this:

 

OnlineShapeFit_Nikolai_April23_2013

 

Now it is up to game developers to use this tech in their titles! The API is available as part of Xbox One XDK. For example you can produce NPC faces like this in your game:

Nikolai as a textured shape model

Nikolai as a textured shape model

 

The 3D model that we shipped in Xbox One Kinect + face capture system is very capable and flexible. The face tracking and face shape modelling is used in just released Kinect Sports Rivals game by RARE studio. You can see the face shape modelling demo in this video:

 

 

Advertisements

    Today, I saw the live broadcast of the demo of the new XBox interface (project “Natal”) at E3 conference. It stunned me! This is a groundbreaking event in the interface design and can be compared to the mouse invention (but it goes much much further). Microsoft did something revolutionary!

    This new interface is based on video cameras that watch your movements, calculate positioning of your body and then translate it to the movements of your avatar in a game. Also, it includes speech recognition, facial recognition and probably a set of other features. They demoed sport games where you control your avatar by literally moving in front of a TV and it does exactly what you do (kicks balls, move legs up and down, etc.). It is much better than Wii, since you actually don’t need any device/controller. Your body controls your avatar in the game.

    These demos were pretty impressive and showed that MS just outdid everyone else in this space (I think competition will have big problems catching up), BUT it got even better! Lionhead founder Peter Molyneux kicked off another demo where a woman was interacting with a virtual boy. It looked like a very real conversation where the virtual boy actually knew where the woman was and was talking to her and not to the space like in today’s games. At some point, the woman drew a picture and gave it to the boy (gave it to the front camera) and the boy took a virtual piece of paper with THIS picture on it! The other interesting thing was when the woman was making patterns on the water and you could see her reflected image on it. Oh yeah, and they were also talking in plain English. It seems like speech recognition also works really well.

Virtual boy Milo

Virtual boy Milo

project Natal in action

project Natal in action