Comprehensive Facial Performance Capture
Eurographics 2011
Graham Fyffe    Tim Hawkins1    Chris Watts2    Wan-Chun Ma    Paul Debevec   
USC Institute for Creative Technologies    Lightstage LLC1    Bake Visual Effects Inc.2   
Abstract

We present a system for recording a live dynamic facial performance, capturing highly detailed geometry and spatially varying diffuse and specular reflectance information for each frame of the performance. The result is a reproduction of the performance that can be rendered from novel viewpoints and novel lighting conditions, achieving photorealistic integration into any virtual environment. Dynamic performances are captured directly, without the need for any template geometry or static geometry scans, and processing is completely automatic, requiring no human input or guidance. Our key contributions are a heuristic for estimating facial reflectance information from gradient illumination photographs, and a geometry optimization framework that maximizes a principled likelihood function combining multi-view stereo correspondence and photometric stereo, using multi-resolution belief propagation. The output of our system is a sequence of geometries and reflectance maps, suitable for rendering in off-the-shelf software. We show results from our system rendered under novel viewpoints and lighting conditions, and validate our results by demonstrating a close match to ground truth photographs.

Downloads

Eurographics 2011 Paper
Face_Capture_Eurographics_2011.pdf, (52.3MB)

Eurographics 2011 Movie
Face_Capture_Eurographics_2011.mov, (36.2MB)