Comprehensive Facial Performance Capture

Eurographics 2011

Graham Fyffe     *Tim Hawkins     **Chris Watts     Wan-Chun Ma     Paul Debevec

USC Institute for Creative Technologies     *Lightstage LLC     **Bake Visual Effects Inc.

 

---

Abstract:

We present a system for recording a live dynamic facial performance, capturing highly detailed geometry and spatially varying diffuse and specular reflectance information for each frame of the performance. The result is a reproduction of the performance that can be rendered from novel viewpoints and novel lighting conditions, achieving photorealistic integration into any virtual environment. Dynamic performances are captured directly, without the need for any template geometry or static geometry scans, and processing is completely automatic, requiring no human input or guidance. Our key contributions are a heuristic for estimating facial reflectance information from gradient illumination photographs, and a geometry optimization framework that maximizes a principled likelihood function combining multi-view stereo correspondence and photometric stereo, using multi-resolution belief propagation. The output of our system is a sequence of geometries and reflectance maps, suitable for rendering in off-the-shelf software. We show results from our system rendered under novel viewpoints and lighting conditions, and validate our results by demonstrating a close match to ground truth photographs.

 

---

Material:

Eurographics 2011 Paper:

• Face_Capture_Eurographics_2011.pdf, 52.3 MB. (Adobe Acrobat)

Eurographics 2011 Movie:

• Face_Capture_Eurographics_2011.mov, 36.2 MB. ( Quicktime )

---

Related Projects:

• Light Stage 5:
  • Temporal Upsampling of Performance Geometry using Photometric Alignment, SIGGRAPH Technical Talks, 2008
  • A High-Resolution Geometry Capture System for Facial Performance, SIGGRAPH Technical Talks, 2008
  • Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination ,
        EGSR 2007