|Real-time Geometry and Reflectance Capture for Digital Face Replacement||ICT Technical Report ICT-TR-04-2008
SIGGRAPH Technical Talk 2008
|Andrew Jones Jen-Yuan Chiang Abhijeet Ghosh Magnus Lang
Matthias Hullin* Jay Busch Paul Debevec
|University of Southern California Centers for Creative Technologies
Max-Planck-Institut fÜr Informatik *
The project aims to develop a real-time geometry capture approach to digital face replacement for a dynamic performance. Digital face replacement has major applications in visual effects for motion pictures as well as interactive applications such as video games and simulation and training environments. This project looks into extending the current 3D face scanning technology developed at the ICT Graphics Lab to support seamless face replacement along with separated diffuse and specular albedo textures and surface normals for high quality post-production relighting of the captured performance. Such an approach goes beyond the traditional scope of face replacement techniques that are either completely image based and hence view-dependent or typically capture a performance under a fixed lighting condition and hence cannot be re-used for other performances.
Our approach builds upon the high resolution 3D face scanning technology developed at the ICT
Graphics Lab for capturing dynamic facial performance. This includes capture of high resolution
performance geometry and textures with high speed cameras (stereo pair of Phantom V10) and active
illumination using a high speed MULE projector and spherical gradient illumination using Light Stage
5. Furthermore, our approach involves tracking of the facial pose during performance (with and without
markers) in order to be able to composite the digital face on the captured background plate (e.g., a
stunt performance) in a view-independent manner.
Another aspect of the project looks into techniques for obtaining separate specular and diffuse albedo
textures and surface normals for high quality relighting of the captured facial performance
to match the lighting of a given background plate. While the 3D face scanning technique for static
expressions exploits polarization of light for this purpose [Ma et al. 2007], such a technique cannot
be used while capturing a dynamic performance due to limitations in the hardware setup [Ma 2008a].
Thus, the project looks into alternative separation techniques of diffuse and specular reflectance
based on color-space analysis and computing separate diffuse and specular normals based on such separation
as a post-process.
Tracking and Alignment:
In order to align the dynamic geometry of the subject with the dynamic background plate, we perform a 3D tracking based on a small set of 15-25 painted marker points on both the subject and the background plate actor. We manually select a number of markers that can be reliably tracked automatically in both of the input videos. Using the tracking data of the subject, we stabilize its performance by inverting the rigid movement. The stabilized geometry is then transformed dynamically to the 3D face position in the background plate.
The captured facial performance is finally rendered with incident illumination from a light probe that is also captured on-site along with the background plate in order to match the appearance of the digital face for compositing with the background plate.
Here, we sample the captured light probe into a set of point lights using the Median Cut algorithm [Debevec 2005] and employ the hybrid normal map technique [Ma et al. 2007] for efficiently relighting the performance with a local shading model using the separated diffuse and specular normals and albedo textures.
ICT Technical Report 2008:
ICT-TR-04-2008-FaceReplacement.pdf, Adobe Acrobat Document, 2.54 MB.
SIGGRAPH Technical Talk 2008 Videos:
SIGGRAPH Technical Talk 2008 Videos:
- Final.mov, 28.5 MB. ( Quicktime )
- Cleanplate.mov, 25.1 MB. ( Quicktime )
- Mike.mov, 11.1 MB. ( Quicktime )
- FloatingHead.mov, 2.16 MB. ( Quicktime )
- Stereo_Patterns_H264.mov, 21.7 MB. ( Quicktime )
SIGGRAPH Technical Talk 2008 Slides:
GradientMocap_S2008Slides, Microsoft Powerpoint, 121 MB.
- Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination , EGSR 2007
- A Framework for Capture and Synthesis of High Resolution Facial Geometry and Performance, Wan-Chun Ma PhD Thesis
- Facial Performance Synthesis using Deformation-Driven Polynomial Displacement Maps, SIGGRAPH Asia 2008
- High Resolution Face Scanning for "Digital Emily", SIGGRAPH 2008 Technical Talk