|Skin Microstructure Deformation with Displacement Map Convolution|
|SIGGRAPH 2015 Computer Animation Festival
SIGGRAPH 2015 Technical Papers
|Koki Nagano Graham Fyffe Oleg Alexander Jernej Barbic1 Hao Li1
Abhijeet Ghosh2 Paul Debevec
|USC Institute for Creative Technologies University of Southern California 1 Imperial College London 2|
Simulating the appearance of human skin is important for rendering realistic digital human characters for simulation, education, and entertainment applications. Skin exhibits great variation in color, surface roughness, and translucency over different parts of the body, between different individuals, and when it's transformed by articulation and deformation. But as variable as skin can be, human perception is remarkably attuned to the subtleties of skin appearance, as attested to by the vast array of makeup products designed to enhance and embellish it.
Advances in measuring and simulating the scattering of light beneath the surface of the skin have made it possible to render convincingly realistic human characters whose skin appear to be fleshy and organic. Today's high-resolution facial scanning techniques (e.g. record facial geometry, surface coloration, and surface mesostructure details at the level of skin pores and fine creases to a resolution of up to a tenth of a millimeter. By recording a sequence of such scans or performing blendshape animation using scans of different high-res expressions, the effects of dynamic mesostructure - pore stretching and skin furrowing - can be recorded and reproduced on a digital character.
Dynamic skin microstructure results from the epidermal skin layers being stretched and compressed by motion of the tissues underneath. Since the skin surface is relatively stiff, it develops a rough microstructure to effectively store a reserve of surface area to prevent rupturing when extended. Thus, parts of the skin which stretch and compress significantly (such as the forehead and around the eyes) are typically rougher than parts which are mostly static, such as the tip of the nose or the top of the head. When skin stretches, the microstructure flattens out and the surface appears less rough as the reserves of tissue are called into action. Under compression, the microstructure bunches up, creating micro-furrows which exhibit anisotropic roughness. Often, stretching in one dimension is accompanied by compression in the perpendicular direction to maintain the area of the surface or the volume of tissues below. A balloon provides a clear example of roughness changes under deformation: the surface is diffuse at first, and becomes shiny when inflated.
While it would be desirable to simulate these changes in appearance during facial animation, curent techniques do not record or simulate dynamic surface microstructure for facial animation. One reason scale: taking the facial surface to be 25cm X 25cm, recording facial shape at 10 micron resolution would require real-time Gigapixel imaging beyond the capabilities of today's camera arrays. And simulating a billion triangles of skin surface, let alone several billion tetrahedra of volume underneath, would be computationally very expensive using finite element techniques.
In this work, we approximate the first-order effects of dynamic skin microstructure by performing fast image processing on a highresolution skin microstructure displacement map obtained as in. Then, as the skin surface deforms, we blur the displacement map along the direction of stretching, and sharpen the displacement map along the direction of compression. On a modern GPU, this can be performed at interactive rates, even for facial skin microstructure at ten micron resolution. We determine the degree of blurring and sharpening by measuring in vivo surface microstructure of several skin patches under a range of stretching and compression, tabulating the changes in their surface normal distributions. We then choose the amount of blurring or sharpening to affect a similar change in surface normal distribution on the microstructure displacement map. While our technique falls short of simulating all the observable effects of dynamic microstructure, it produces measurement-based changes in surface roughness and anisotropic changes in surface microstructure orientation consistent with real skin deformation. For validation, we compare renderings using our technique to real photographs of faces making similar expressions.
Fig. 5 shows frames from a sequence of a 1cm wide digitized skin patch being deformed by an invisible probe. It uses a relatively low-resolution finite element volumetric mesh with 25,000 tetrahedra to simulate the mesostructure which in turns drives dynamic microstructure convolution. The neutral microstructure was recorded using the system in Fig. 1 (left) at 10 micron resolution from the forehead of a young adult male, and its microstructure is convolved with parameters fit to match its own surface normal distribution changes under deformation as described in Sec. 5. The rendering was made using the V-Ray package to simualte subsurface scattering. As seen in the accompanying video, the skin microstructure bunches up and flattens out as the surface deforms at a resolution much greater than the FEM simulation.
Fig. 1 highlights the effect of using no microgeometry, static microgeometry, and dynamic microgeometry simulated using displacement map convolution with a real-time rendering. Rendering only with 4K resolution mesostructure from a standard facial scan produces too polished an appearance at this scale. Adding static microstructure computed at 16K resolution using a texture synthesis technique increases visual detail but produces conflicting surface strain cues in the compressed and stretched areas. Convolving the static microstructure according to the surface strain using normal distribution curves from a related skin patch produces anisotropic skin microstructure consistent with the expression deformation and a more convincing sense of skin under tension.
Questions and Answers:
What is microgeometry?
- Human skin features can be divided into roughly three scales: macro, meso, and micro. On a face, "macroscale" features define the overall shape of the facial features. This includes things like the jaw line, nose, and eyebrows. "Mesoscale" features are on the order of a millimeter (~0.1 mm) and include features such as pores and fine wrinkles. "Microscale" features are an order of magnitude smaller than pores and fine wrinkles (~10 microns) and describe the very fine deviations inside pores and along wrinkles. The following report extensively describes the anatomy of human skin: Link
Why is microgeometry important? Is this "microfacet"?
- In the theory of rough surface reflections, a surface is composed of microscopic faces called "microfacets", which are assumed to behave like perfect minuscule mirrors. Our scan, captured at sub 10 micron resolution, is still an order of magnitude larger than a "real" microfacet. However it provides noticeable effects on surface reflectance. When it is seen from far away such as in a portrait, it breaks up specular highlights, appearing as high frequency glints on the surface. Figure 1 (a) shows that rendering with only a 4K resolution mesostructure from a standard facial scan. This produces specular highlights that appear too dull in an extreme closeup. Adding the static microstructure computed at 16K resolution with the texture synthesis technique from [Graham et al. 2013] increases visual detail as in Figure 1 (b). For more details about static microgeometry, we encourage you to read our previous paper.
Is there a low budget way to approximate the static microgeometry?
- While [Graham et al. 2013] used measurement-based texture synthesis to synthesize 16K static microgeometry, [von der Pahlen et al. 2014] demonstrated that the effects of static microstructure can be approximated by a procedural noise in a real-time character demo Link.
What is the difference between this paper and the [Graham et al. 2013] paper?
- While [Graham et al. 2013] focuses on the effect of static microstructures, our work investigates the dynamic appearance of the skin microstructures. Though adding the static microstructure improves the skin like quality, it produces conflicting surface strain cues when the skin is deformed significantly (Figure 1 (b)). On the other hand, our technique produces anisotropic skin structures consistent with the expression, providing a more convincing sense of skin under tension (Figure 1 (c)).
What happens when the skin deforms?
- Generally speaking, the skin becomes rougher when it is compressed and smoother when it is stretched. Figure 4 shows that under deformation the skin surface normal distribution histogram (which can be viewed as a resulting BRDF) exhibits the roughness and anisotropy changes in a predictable manner.
How do you simulate dynamic microgeometry?
- We approximate the skin being flattened under stretching, and bunched up under compressions by convolving a 16K displacement map. We blur the microgeometry displacement map in the direction of stretching, and sharpen it in the direction of compression using the surface normal distribution histogram as a guide. This entire computation can be efficiently implemented on GPU shaders.
Can I use dynamic microgeometry for realtime/offline applications?
- The paper and the companion video include both realtime renders done with GPU shaders such as Figure 1, and offline renders such as Figure 5 skin slab.
How is the computed microstructure used together with the mesoscale structure?
- In this paper, we used unconstrained blending, meaning that we simply added the computed 16K dynamic microgeometry displacement to the existing 4K mesogeometry displacement. If desired, constrained blending may be done by first converting displacement into a tangent normal map, and leveraging a normal map blending technique as done in [von der Pahlen et al. 2014].
Can I use physics simulation with this technique?
- The microgeometry simulation framework allows deformation from any source, including physics simulation, keyframe animation, or captured facial performance as shown in this paper.
The authors would like to thank Randal Hill, Kimberly Lu, Ari Shapiro, Cary Peng, Bill Phelps, Emily O'Brien, Jay Busch, Xueming Yu, Etienne Danvoye, Javier von del Pahlen, the Digital Human League, Valerie Dauphin, and Kathleen Haase for their assistance and support. This research was sponsored by the U.S. Army Research Laboratory (ARL), the Funai Foundation for Information Technology, and in part by the National Science Foundation (CAREER-1055035, IIS-1422869) and the Sloan Foundation, and Royal Society Wolfson Research Merit Award. The content of the information does not necessarily reflect the position or the policy of the US Government, and no official endorsement should be inferred.
SIGGRAPH 2014 Talk:
SIGGRAPH 2015 Paper:
SIGGRAPH 2014 Video:
- SkinStretchTalk_SIGGRAPH2014.wmv, 167 MB.
SIGGRAPH 2015 Video:
- SkinStretchPaper_SIGGRAPH2015.mp4, 191 MB.
- SkinStretchPaper_SIGGRAPH2015.wmv, 302 MB.
- EmilyReunion_2015CAF_1920x1080_30fps_v16.mp4, 65.2 MB.
Sample Patch Data:
- SkinPatchData.zip, 502 MB.
- Measurement-Based Synthesis of Facial Microgeometry, SIGGRAPH 2012, Eurographics 2013