Post-production Facial Performance Relighting using Reflectance Transfer
ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007)
Pieter Peers1    Naoki Tamura2,3    Wojciech Matusik3    Paul Debevec3   
University of Southern California, Institute for Creative Technologies1    The University of Tokyo2    MERL3   

From left to right: A reference (source) subject relit with the desired illumination. A frame of the performance (target) actor captured under uniform illumination. The quotient image of the reference actor relit using the target illumination and the uniform performance illumination (warped into the pose of the performance actor). The performance actor with the desired shading transferred from the relit performance actor using the quotient image.
Abstract

We propose a novel post-production facial performance relighting system for human actors. Our system uses just a dataset of view-dependent facial appearances with a neutral expression, captured for a static subject using a Light Stage apparatus. For the actual performance, however, a potentially different actor is captured under known, but static, illumination. During post-production, the reflectance field of the reference dataset actor is transferred onto the dynamic performance, enabling image-based relighting of the entire sequence. Our approach makes post-production relighting more practical and could easily be incorporated in a traditional production pipeline since it does not require additional hardware during principal photography. Additionally, we show that our system is suitable for real-time post-production illumination editing.


Downloads

SIGGRAPH 2007 Paper
SIGGRAPH2007_PFPR_high.pdf, (17.9MB)
SIGGRAPH2007_PFPR_low.pdf, (1MB)

SIGGRAPH 2007 Video
SIG2007_PFPR_Hires.mp4, (57.1MB)
SIG2007_PFPR_Lores.mp4, (33MB)

Related Projects

Light Stage 1
Acquiring the Reflectance Field of a Human Face, SIGGRAPH 2000
Facial Reflectance Field Demo, SIGGRAPH 2000 Creative Applications Laboratory
Realistic Human Face Scanning and Rendering, ICT Graphics Lab 2001

Light Stage 2
A Photometric Approach to Digitizing Cultural Artifacts, VAST 2001
Animatable Facial Reflectance Fields, EGSR 2004
Reflectance Field Rendering of Human Faces for "Spider-Man 2", SIGGRAPH 2004 Sketch

Light Stage 3
A Lighting Reproduction Approach to Live-Action Compositing, SIGGRAPH 2002
Optimizing Color Matching in a Lighting Reproduction System for Complex Subject and Illuminant Spectra, EGSR 2003

Light Stage 5
Postproduction Re-Illumination of Live Action Using Interleaved Lighting, SIGGRAPH 2004 Poster
Performance Geometry Capture for Spatially Varying Relighting, SIGGRAPH 2005 Sketch

Light Stage 6
Relighting Human Locomotion with Flowed Reflectance Fields, EGSR 2006 Paper
Relighting Human Locomotion with Flowed Reflectance Fields, SIGGRAPH 2006 Sketch
Light Stage Data Gallery.