In summer 2011, my group at Microsoft shipped an interesting Computer Vision app/mini game for XBox 360 called “Avatar Kinect”. I worked on the 2D/3D face tracking technology. You can pose in front of the Kinect camera and the application tracks movements of your head and facial features (lips, eyebrows) and renders you as an XBox avatar. Pretty cool app if you want to record little videos of yourself as an animated cartoon avatar and then post them to YouTube. Or if you want to talk to your friends as an avatar, the app allows multiparty avatar chat.
For example you can do:
Here is the internals of our face tracking technology:
We used a combination of Active Appearance Models on “steroids” plus few other things like neural network, face detector and various classifiers to make it stable and robust. You can read more about Active Appearance Models here, here and here . Off course the usage of Kinect camera improved precision and robustness a lot (due to its depth camera).
For more information about our face tracking tech, see my paper “Real-time 3D face tracking based on active appearance model constrained by depth data”.