Practical Modeling and Acquisition of Layered Facial Reflectance
ACM SIGGRAPH Asia 2008
Abhijeet Ghosh    Tim Hawkins    Pieter Peers    Sune Frederiksen    Paul Debevec   
USC Institute for Creative Technologies

Figure 1: Layers of skin reflectance which are modeled by our technique and used to render faces for novel viewpoints and lighting.
Abstract

We present a practical method for modeling layered facial reflectance consisting of specular reflectance, single scattering, and shallow and deep subsurface scattering. We estimate parameters of appropriate reflectance models for each of these layers from just 20 photographs recorded in a few seconds from a single viewpoint. We extract spatially-varying specular reflectance and singlescattering parameters from polarization-difference images under spherical and point source illumination. Next, we employ direct-indirect separation to decompose the remaining multiple scattering observed under cross-polarization into shallow and deep scattering components to model the light transport through multiple layers of skin. Finally, we match appropriate diffusion models to the extracted shallow and deep scattering components for different regions on the face. We validate our technique by comparing renderings of subjects to reference photographs recorded from novel viewpoints and under novel illumination conditions.

Introduction

This paper aims to develop a practical appearance model which is in addition easy to incorporate in existing rendering systems. The detail in the facial appearance model should be such that full-screen close-ups can be faithfully reproduced. Additionally, working with live subjects requires fast acquisition to avoid registration problems, temporal changes in the appearance (e.g., due to sweat or blood flow), and to enable capture of facial appearance of natural expressions which are difficult to hold for more than a few seconds.

To achieve these goals, we model facial skin reflectance as a combination of different layers: specular reflectance, single scattering, and shallow and deep multiple scattering (Fig. 1). A suitable reflectance or scattering model is selected for each layer, and parameters are obtained using a single high-resolution still camera to capture a small set of 20 photographs under environmental and projected lighting conditions. For each reflectance component, we estimate or infer high-frequency details such as albedo and normals per pixel based on the environmental illumination patterns, while modeling lower-frequency BRDF and scattering behavior per region based on the projected patterns. This allows for fast acquisition and straightforward processing, while achieving a high level of realism in the resulting models.


Figure 2: Modeled skin reflectance components.

Measured Reflectance Components


Figure 3:(a) Polarization difference image under spherical illumination, used for estimating specular albedo. (b) Cross-polarized image under spherical illumination, used to measure total scattered albedo. (c) Polarization difference image under directional illumination, used for estimating the specular lobe shape per region. The image also includes some polarization preserving non-specular backscattering (which we model as mostly single-scattering), which can be seen to pick up color from the melanin in the epidermis. (d) Cross-polarized image under directional illumination, showing multiple scattering. (e) "direct" component of (d), showing shallow scattering. (f) "indirect" component of (d), showing deeply scattered light. Note that (d) = (e) + (f) and that (c) + (d) produces a typical front-lit photograph.

Downloads

ACM SIGGRAPH Asia 2008 Paper
facereflectance_sa2008.pdf, (18.5MB)

SIGGRAPH 2008 Talk
siggraph2008Talk.pdf, (972KB)

Related Projects

Surface Reflectomy
High-Resolution Face Scanning, Eurographics Symposium on Rendering 2007
Spatially Varying Subsurface Scattering, USC ICT Technical Report ICT-TR-01-2006

Light Stage 1
Acquiring the Reflectance Field of a Human Face, SIGGRAPH 2000
Facial Reflectance Field Demo, SIGGRAPH 2000 Creative Applications Laboratory
Realistic Human Face Scanning and Rendering, ICT Graphics Lab 2001

Light Stage 2
A Photometric Approach to Digitizing Cultural Artifacts, VAST 2001
Animatable Facial Reflectance Fields, EGSR 2004
Reflectance Field Rendering of Human Faces for "Spider-Man 2", SIGGRAPH 2004 Sketch

Light Stage 3
A Lighting Reproduction Approach to Live-Action Compositing, SIGGRAPH 2002
Optimizing Color Matching in a Lighting Reproduction System for Complex Subject and Illuminant Spectra, EGSR 2003

Light Stage 5
Postproduction Re-Illumination of Live Action Using Interleaved Lighting, SIGGRAPH 2004 Poster
Performance Geometry Capture for Spatially Varying Relighting, SIGGRAPH 2005 Sketch