Performance Relighting and Reflectance Transformation with Time-Multiplexed Illumination
SIGGRAPH 2005 Paper
Andreas Wenger    Andrew Gardner    Chris Tchou   
Jonas Unger    Tim Hawkins    Paul Debevec   
USC Institute for Creative Technologies
Abstract

We present a technique for capturing an actor's live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-multiplexed basis lighting conditions, and to record these conditions with a highspeed video camera so that many conditions are recorded in the span of the desired output frame interval. We investigate several lighting bases for representing the sphere of incident illumination using a set of discrete LED light sources, and we estimate and compensate for subject motion using optical flow and image warping based on a set of tracking frames inserted into the lighting basis. To composite the illuminated performance into a new background, we include a time-multiplexed matte within the basis. We also show that the acquired data enables time-varying surface normals, albedo, and ambient occlusion to be estimated, which can be used to transform the actor's reflectance to produce both subtle and stylistic effects.



Actress Marissa Sellers performs in Light Stage 5 as a high speed camera films her lit by 152 lighting directions every 24th of a second. The images can be used to relight her in new lighting environments, or to estimate and transform the reflectance characteristics of her performance. Behind her is a gray board used for time-multiplexed matting.

Inset: Her performance virtually illuminated under two different real-world illumination environments.
Downloads

SIGGRAPH 2005 Paper
LightStage5_S2005.pdf (4.5MB)

SIGGRAPH 2005 Video
LS5_Siggraph_052005.avi (101MB)
LS5Siggraph2005_H264.mov (170MB)

High Def Clip from SIGGRAPH 2005 ET
LS5_ET_HD_H264.mov (88MB)

SIGGRAPH 2004 Poster using nine spherical harmonics for relighting
lightstage5_poster.pdf (610KB)

SIGGRAPH 2004 Poster Video using nine spherical harmonics for relighting
LS5-Poster-Video-2004.mov (29MB)

2004 Technical Report
ICT-TR-05.2004.pdf (1.45MB)

Related Projects

Light Stage 1
Acquiring the Reflectance Field of a Human Face, SIGGRAPH 2000
Facial Reflectance Field Demo, SIGGRAPH 2000 Creative Applications Laboratory
Realistic Human Face Scanning and Rendering, ICT Graphics Lab 2001

Light Stage 2
A Photometric Approach to Digitizing Cultural Artifacts, VAST 2001
Animatable Facial Reflectance Fields, EGSR 2004
Reflectance Field Rendering of Human Faces for "Spider-Man 2", SIGGRAPH 2004 Sketch

Light Stage 3
A Lighting Reproduction Approach to Live-Action Compositing, SIGGRAPH 2002
Optimizing Color Matching in a Lighting Reproduction System for Complex Subject and Illuminant Spectra, EGSR 2003

Light Stage 5
Postproduction Re-Illumination of Live Action Using Interleaved Lighting, SIGGRAPH 2004 Poster
Performance Geometry Capture for Spatially Varying Relighting, SIGGRAPH 2005 Sketch

Light Stage 6
Relighting Human Locomotion with Flowed Reflectance Fields, EGSR 2006 Paper
Relighting Human Locomotion with Flowed Reflectance Fields, SIGGRAPH 2006 Sketch
Light Stage Data Gallery.