|A Framework for Capture and Synthesis of |
High Resolution Facial Geometry and Performance
|PhD Thesis, National Taiwan University|
We present a framework that captures and synthesizes high resolution facial geometry and performance. In order to capture highly detailed surface structures, a theory of fast normal recovery using spherical gradient illumination patterns is presented to estimate surface normal maps of an object from either its diffuse or specular reflectance, simultaneously from any viewpoints. We show that the normal map from specular reflectance yields the best record of detailed surface shape, which can be used for geometry enhancement. Moreover, the normal map from the diffuse reflectance is able to produce a good approximation of subsurface scattering. Based on the theory, two systems are developed to capture high resolution facial geometry of a static face or dynamic facial performance.
We human beings are able to exhibit complex facial expressions. On the other hand, we are also evolved to be extremely sensitive to reading subtle changes in facial expressions. As an result, this leads to the concept of the Uncanny Valley theory, which holds that the closer something appears to human, the more its dissimilarities may stand out and create a negative reaction in viewers. We tend to become less tolerate to facial animations as they are getting closer to realism. Therefore, accurately reproducing facial expressions still remains one of the most difficult problem in computer graphics.
To make facial animation looks plausible, at least we need to consider the following aspects:
- Deformation. A human face contains more than fifty muscles. Facial muscles don't all attach directly to bone like they usually do in the rest of the body. Instead, many of them attach under the skin. This allows you precisely control your facial deformation. By controlling the muscles, we are able to demonstrate expressions with a large number of degrees of freedom. Motion capture is able to record a sparse resolution of the deformation.
- Skin meso-structure. Skin is the most exterior surface and it exhibits specular reflections. The appearance of pores and wrinkles on the skin is changed during deformation caused by expressions, mostly the change in specular highlights. For 9 examples, skin may exhibits anisotropic specular reflections under stretching. At this point, there are very few researches targeting on dynamic behaviors of the skin meso-structure.
- Reflectance. It is impossible to render realistic human faces without proper reflectance information of the skin. Due to the complex nature of the facial reflectance, high end rendering techniques often require a large amount of data (imagebased techniques) or massive computation (subsurface scattering simulation) to create vivid face renderings. The translucency property of the skin also varies with different wavelengthes. Light of longer wavelength can be scattered further into skin.
PhD Thesis Paper:
- Polarized Spherical Gradients, Eurographics Symposium on Rendering 2007
- Facial Performance Synthesis using Deformation-Driven PDMs , SIGGRAPH Asia 2008