Relighting Human Locomotion with Flowed Reflectance Fields
2006 Eurographics Symposium on Rendering
Per Einarsson    Charles-Felix Chabert    Andrew Jones    Wan-Chun Ma*    Bruce Lamond   
Tim Hawkins    Mark Bolas**    Sebastian Sylwan    Paul Debevec   
University of Southern California Centers for Creative Technologies    National Taiwan University*   
University of Southern California School of Cinematic Arts**
Abstract

We present an image-based approach for capturing the appearance of a walking or running person so they can be rendered realistically under variable viewpoint and illumination. In our approach, a person walks on a treadmill at a regular rate as a turntable slowly rotates the person's direction. As this happens, the person is filmed with a vertical array of high-speed cameras under a time-multiplexed lighting basis, acquiring a seven-dimensional dataset of the person under variable time, illumination, and viewing direction in approximately forty seconds. We process this data into a flowed reflectance field using an optical flow algorithm to correspond pixels in neighboring camera views and time samples to each other, and we use image compression to reduce the size of this data. We then use image-based relighting and a hardware-accelerated combination of view morphing and light field rendering to render the subject under user-specified viewpoint and lighting conditions. To composite the person into a scene, we use an alpha channel derived from back lighting and a retroreflective treadmill surface and a visual hull process to render the shadows the person would cast onto the ground. We demonstrate realistic composites of several subjects into real and virtual environments using our technique.


Multiple instances of a running subject are composited into an image-based lighting environment with matched viewpoint and illumination. The subject was captured running in a human-scale light stage apparatus under a time multiplexed lighting basis to allow for image-based relighting. A novel combination of light field rendering and view interpolation was used to render the subject from an arbitrary 3D viewpoint.

Relighting Human LocomotionSIGGRAPH 2006 Computer Animation Festival
Downloads

ESGR 2006 Paper
EGSR2006_RHL.pdf, (5.9MB)

ESGR 2006 Video
LS6_EGSR_062006.avi, (52.3MB)
LS6_EGSR_062006.mov, (96.1MB)

ESGR 2006 Sketch
SIGGRAPHsketch_RHL_0610.pdf, (456KB)
Presentation Slides, (18MB)

LS6 Video Presented at the Director's Guild of America's Digital Day Conference, July 29, 2006
LS6_DGA_864x480_DivX.avi, (52.3 MB)
LS6_DGA_864x480_H264.mov, (96.1MB)
Related Projects

Light Stage 1
Acquiring the Reflectance Field of a Human Face, SIGGRAPH 2000
Facial Reflectance Field Demo, SIGGRAPH 2000 Creative Applications Laboratory
Realistic Human Face Scanning and Rendering, ICT Graphics Lab 2001

Light Stage 2
A Photometric Approach to Digitizing Cultural Artifacts, VAST 2001
Animatable Facial Reflectance Fields, EGSR 2004
Reflectance Field Rendering of Human Faces for "Spider-Man 2", SIGGRAPH 2004 Sketch

Light Stage 3
A Lighting Reproduction Approach to Live-Action Compositing, SIGGRAPH 2002
Optimizing Color Matching in a Lighting Reproduction System for Complex Subject and Illuminant Spectra, EGSR 2003

Light Stage 5
Postproduction Re-Illumination of Live Action Using Interleaved Lighting, SIGGRAPH 2004 Poster
Performance Geometry Capture for Spatially Varying Relighting, SIGGRAPH 2005 Sketch