Although PRT can capture realistic global illumination effects for efficient run-time render-ing at real-time rates, it only considers spatially-uniform surface appearance. BRT [163] thus generalizs PRT with spatially-varying materials, which is called radiance transfer textures, in low-frequency lighting environments to improve image quality with detailed surface appear-ance. The main concept of BRT is to separate the light transport problem into macro-scale radiance transfer (coarsely-sampled global illumination data) and meso-scale radiance trans-fer (spatially-varying appearance models). This not only enriches PRT with illumination- and view-dependent meso-textures, but also prevents full PRT simulation all over the object sur-faces at the meso-scale. Nevertheless, previous BRT algorithms are limited to low-frequency light transport and bump-mapping-like surface appearance. The radiance transfer textures only model fine-scale lighting and shadowing effects, but neither contain shape information nor actu-ally modify the surface geometry. Therefore, meso-scale shape details and shadow boundaries owing to complex meso-structures can not be faithfully captured.
In this dissertation, we address the problem of all-frequency BRT with spatially-varying re-flectance and complex meso-structures. The proposed approach combines bidirectional texture functions(BTFs) and view-dependent occlusion texture functions (VOTFs) for meso-scale radi-ance transfer to model not only spatially-varying illumination effects but also view-dependent occlusions at the meso-scale. To obtain an accurate and compact representation, we apply ten-sor approximation algorithms to decompose BTFs and VOTFs into a few low-order factors and the reduced multi-dimensional core tensor(s).
As for macro-scale radiance transfer, a solution to global illumination based on all-frequency PRT is precomputed. The low-order factors extracted by tensor approximation algorithms espe-cially facilitate the computation of the radiance transfer matrix at each object vertex. Moreover,
Table 9.3: Feature comparisons of the proposed all-frequency BRT algorithm with various meso-scale appearance models, including bidirectional texture functions (BTFs) [28], gener-alized displacement maps (GDMs) [197], relief mapping (RM) [138, 139], radiance transfer textures(RTTs) [163], radiance transfer volume (RTV) [197], shell radiance texture functions (SRTFs) [166], shell texture functions (STFs) [21], and view-dependent displacement mapping (VDM) [194].
Appearance models BTFs GDMs Proposed BRT RM RTTs RTV SRTFs STFs VDM
All-frequency √ √ √ √ √ √
∗
For shell texture functions, meso-scale occlusions are determined by the ray tracing process at run-time.
†
To the best of our knowledge, although it is possible to extend these appearance models for meso-structure synthesis, no articles have conducted complete experiments on this issue.
‡
One of the acquisition methods for spatially-varying sub-surface scattering materials can be found in [135].
we also employ the scale occlusion information in VOTFs to cast shadows due to meso-structures.
Table 9.3 lists the feature comparisons of the proposed all-frequency BRT algorithm with several meso-scale appearance models. It can be shown from this table that the proposed BRT algorithm supports most features, including sub-surface scattering if the utilized BTFs contain this effect.
9.2.1 Problem Formulation
Consider a static scene, distant illumination, and spatially-varying appearance models. The rendering equation [73] over the unit sphere S2 can be rewritten as
Lo,p(ωv) =
Object3
Figure 9.6: The proposed all-frequency BRT algorithm determines visibility values at the meso-scale. Black solid lines indicate true object surfaces. Red dotted lines instead show that there are object surfaces at the macro-scale, but no occlusions at the meso-scale. From a surface point p in direction ωl, the true intersection point thus is not q(1)p,ωl but q(3)p,ωl, which is the first point whose exitant ray in direction −ωlis occluded at the meso-scale.
direct and indirect illumination terms of exitant radiance, Lin(ωl) ∈ R represents the incident radiance from distant illumination direction ωl ∈ S2, and np ∈ S2 specifies the surface normal at point p. The BTF Bp(ωl, ωv, tp) ∈ R and the VOTF Op(ωv, tp) ∈ {x ∈ R | 0 ≤ x ≤ 1} are the meso-scale reflectance distributions and view-dependent occlusions at point p, where the spatial coordinates of point p, denoted by tp, are obtained from the meso-structure synthesis.
Moreover, Qp,ωl =q(1)p,ωl, . . . , q(h)p,ωl represents the set of intersection points from point p in direction ωl. The elements of Qp,ωl are ordered according to their Euclidean distances to point p, from the nearest hit point q(1)p,ωl to the first point q(h)p,ωl whose exitant ray in direction −ωl is occluded at the scale (Figure 9.6). Note that visibility values are determined at the meso-scale, which is different from previous PRT methods that only model macro-scale visibility.
To obtain a compact representation for meso-scale radiance transfer, Bp(ωl, ωv, tp) and Op(ωv, tp) are respectively organized as a fourth order tensor A(Bp)∈ RIωl(Bp)×Iωv(Bp)×Ix(Bp)×Iy(Bp)
A(Op) ≈ ˆA(Op) =
Here, we omit a detailed description of the notation in Equations 9.18–9.21. As described in Sections 8.1.1 and 8.2.1, readers can refer to Equations 8.1, 8.2, 8.14, and 8.15 for the mathematical notation.
Equations 9.18 and 9.20 can be further rewritten in matrix forms as N -SVD:
From Equations 9.22 and 9.23, it is easy to identify that mode-ωl basis matrices resemble the illumination-dependent basis functions in BRDF factorization [105, 196]. By following the same approach as in the derivation of Equation 9.8, Equation 9.15 then becomes
N -SVD:
where lo,p denotes the exitance radiance vector at point p, Tp (or Tp,c) denotes the radiance transfer matrix at point p (for cluster c), and the interpolation matrix Apas well as the incident illumination vector lin are respectively defined in Equations 9.10 and 9.11.
From [196], Tp (or Tp,c) can be further modeled with the sum of direct radiance transfer matrix T(D)p (or T(D)p,c) and an infinite series of j-bounce radiance transfer matrices T(j)p
∞
Macro-Scale
Input Model Surface Atlas Coordinate Texture
Measured Meso-Structures
OR
(b) Meso-scale radiance transfer and synthesis
BRT
Figure 9.7: System diagrams of the proposed all-frequency BRT algorithm.
9.2.2 Algorithm
Overview
As illustrated in Figure 9.7(a), the proposed all-frequency BRT algorithm consists of off-line and run-time processes. In the off-line process, the surface meso-structures, namely BTFs and VOTFs, are either simulated by rendering three-dimensional scenes using global illumination techniques or measured from real-world objects. After that, the meso-structures are compressed using tensor approximation algorithms and then synthesized over object surfaces, as illustrated in Figure 9.7(b). For macro-scale radiance transfer (Figure 9.7(c)), the all-frequency PRT algo-rithm (Section 9.1) is extended to derive a global illumination solution that considers geometry details at both scales. In the run-time process, the compressed meso-structures and macro-scale BRT data, along with the synthesized results, are combined to render objects in all-frequency lighting environments.
Off-Line Process
Meso-scale radiance transfer: To reduce storage space, the meso-scale surface appearance models, including BTFs and VOTFs, are compressed using tensor approximation algorithms.
For VOTFs, view-dependent signed-distance representations are recommended instead of bi-nary visibility masks to preserve sharp geometric features at the meso-scale (Section 8.2.1). Af-ter compressing BTFs and VOTFs, we also employ the proposed resampling and meso-structure synthesis methods as described in Section 8.1.1 to prepare necessary data for macro-scale radi-ance transfer and run-time rendering.
Macro-scale radiance transfer: Based on the all-frequency PRT framework in Section 9.1, the raw radiance transfer matrices are computed at each object vertex, approximated with a set of uniform univariate SRBFs, and finally compressed using CTA (or K-CTA) to exploit inter-vertex coherence. During BRT computation, the meso-scale visibility at point p in direction ωv, namely Op(ωv, tp), is obtained by sampling ˆZ(Op) (or ˆZc(Op)) at spatial coordinates tp in direction ωv and performing construction on GPUs, where
N -SVD: Zˆ(Op) = Z(Op)×xU(Ox p)×yU(Oy p), (9.28) CTA or K-CTA: ∀c, ˆZc(Op) = Zc(Op)×xU(Ox,cp)×yU(Oy,cp). (9.29) For T(j)p
∞
j=1 (or T(j)p,c
∞
j=1), a triangle index texture, a barycentric coordinate texture, and a core transfer vector texture are rendered at each vertex on GPUs to identify the infor-mation of hit points. These textures are sampled at low resolutions as suggested by Sun et al.
[171] to decrease precomputation time, and then employed for the computation of T(j)p
∞ j=1
(orT(j)p,c
∞
j=1) on GPUs. The core transfer vector of point p in direction ωv is derived from
sampling ˆZ(Bp)(or ˆZc(Bp)) at spatial coordinates tpin direction ωv, where
N -SVD: Zˆ(Bp) = Z(Bp)×xU(Bx p)×yU(By p), (9.30) CTA or K-CTA: ∀c, ˆZc(Bp) = Zc(Bp)×xU(Bx,cp)×yU(By,cp). (9.31) Similar to the off-line process of the proposed all-frequency PRT algorithm (Section 9.1.2), the radiance transfer matrices of an object are approximated with a set of uniform univariate SRBFs, and further organized as a third order tensor A(T)∈ RIωv(T)×Iωl(T)×Ivert(T). Given the reduced ranks of each mode, namely Rω(T)v , R(T)ωl , and R(T)vert, A(T) is then compressed using CTA (or K-CTA) with total C(T)clusters for the vertex mode. Here, note that the utilized tensor approx-imation algorithm for macro-scale radiance transfer (CTA or K-CTA) is independent of that for meso-scale radiance transfer. One can always apply different tensor approximation algorithm to model the radiance transfer data at each scale.
Run-Time Process
The BRT rendering process is similar to the run-time process of the proposed all-frequency PRT algorithm and consists of the following steps:
1. Perform Steps 1–3 as described in the run-time process of the proposed all-frequency PRT algorithm (Section 9.1.2 on Page 132) to obtain the per-vertex radiance transfer vector rp. 2. In the pixel shader, sample the synthesized texture S for the BTF/VOTF spatial
coordi-nates tpof current pixel p.
3. If the meso-structure of pixel p does not contain VOTF, set pixel p as visible and go to Step 6. Otherwise, continue to execute Step 4.
4. Perform Steps 3 and 4 as described in the run-time process of VOTFs (Section 8.2.2 on Page 121) to obtain the meso-scale visibility value ˆO0p. Set pixel p to visible if ˆOp0 > 0.
5. If pixel p is visible, compute its shading color by Step 6. Otherwise, discard it.
6. Sample the texture of mode-ωv basis matrix for all the components of current novel view direction. The shading color of pixel p is then given by the dot product of the sampled results and rp.
Here, we omit a detailed description of the above rendering steps. Readers can refer to Sections 8.1.1, 8.2.2, and 9.1.2 for more details about the rendering issues of BTFs, VOTFs, and PRT.
Table 9.4: Statistics and timing measurements of the proposed all-frequency BRT algorithm.
In the row Frames per second, we list the rendering performance with/without visibility anti-aliasing when the viewpoint changes. For the configurations of each meso-scale material, please refer to Tables 8.1 and 8.6.
Model Bunny BunnyPlane Cloth Teapot
Material(s) Sponge Fiber + Sponge Wool + Sponge RoughHole + Sponge
Vertices: I
vert(T)36k 98k 55k 75k
SRBFs: I
ω(T)l642 2562 642 642
Raw BRT data (GB) 40.11 108.35 69.23 123.26
R
(T)ωl× R
(T)vert64×12 64×12 64×12 64×12
Clusters: C
(T)100 180 110 150
Compressed data (MB) 15.77 71.4 18.79 29.44
BRT computation time (hr.) 2.65 22.55 7.86 10.08
CTA compression time (hr.) 8.28 16.98 12.51 13.66
Frames per second (w/wo) -/50.45 21.38/28.13 -/31.83 19.46/24.21
9.2.3 Experimental Results
The experiments and simulation timings of the proposed all-frequency PRT algorithm were conducted and measured on a workstation with an Intel Core 2 Extreme QX6700 CPU, an NVIDIA GeForce 8800 Ultra graphics card, and 4 gigabytes main memory under Microsoft Windows Vista operating system. We employed NVIDIA CUDA [128] to accelerate the off-line process and Microsoft Direct3D 9 for run-time rendering on GPUs. In our experiments, the univariate Gaussian SRBFs were adopted to represent the radiance transfer functions and the lighting environments. The SRBF center and bandwidth sets were also constrained to be the same for red, green, and blue channels of a radiance transfer function or a lighting environment.
To improve rendering performance, the super-clustering technique [161] was also applied to decrease the number of redrawn triangles.
Table 9.4 lists the experimental statistics of the proposed all-frequency BRT algorithm in various configurations. For macro-scale radiance transfer, we simulated light paths with at most two inter-reflections to precompute a 6×32×32 radiance transfer matrix at each vertex and approximated the raw BRT data with a set of uniform univariate SRBFs. Due to enormous amount of BRT data, we did not directly employ CTA (or K-CTA) but first applied clustered principal component analysis [161] to classify vertices, as suggested by Sun et al. [171], and fine-tuned cluster membership using CTA (or K-CTA). The reduction of the view mode was omitted to accelerate the compression process, and the reduced ranks of the light and vertex modes were respectively set to 64 and 12.
Figure 9.8 compares the rendered images with different tensor approximation algorithms for meso-structures based on the proposed approach. It shows that K-CTA achieves image quality comparable to N -SVD, while providing almost the same rendering performance as CTA.
Al-(a) Raw BRT data (b) N -SVD (6.34 FPS)
(c) CTA (22.69 FPS) (d) K-CTA (21.38 FPS)
Figure 9.8: Comparisons of rendered images with different tensor approximation algorithms for meso-structures based on the proposed all-frequency BRT algorithm for the model BunnyPlane with the materials Fiber and Sponge. The rendering performance in frames per second (FPS) are shown in parentheses. The configurations of macro- and meso-scale radiance transfer are listed in Tables 8.1, 8.6, and 9.4.
though both CTA and K-CTA allow real-time rendering performance, CTA sometimes produces noticeable seams when the viewpoint changes. This result is consistent with our experiments of BTF and VOTF compression in Chapter 8.
Figure 9.9 compares the rendered images with/without indirect illumination based on the proposed approach for the model Bunny. Without indirect illumination, as shown in Figure 9.9(a), inter-reflections between object surfaces can not be faithfully captured. This produces some false hard and soft shadows that result in many over dark regions, especially around the neck and forelegs of Bunny. In Figure 9.10, we further present more rendered images based on the proposed approach. From these images, the proposed BRT algorithm certainly provides more reflectance and geometric details of meso-scale surface appearance than previous PRT methods. Futhermore, VOTFs particularly allow rendering complex geometric features without consuming hundreds of thousands of polygons to explicitly model the surface micro-geometry.
(a) Direct illumination (b) Direct and indirect illumination
Figure 9.9: Comparisons of rendered images with/without indirect illumination based on the proposed all-frequency BRT algorithm for the model Bunny with the material Sponge. From top to bottom: raw BRT data; compressed BRT data (SRBF); compressed BRT data (SRBF + CTA).
The configurations of macro- and meso-scale radiance transfer are listed in Tables 8.1 and 9.4.
(a) Cloth with Wool and Sponge (b) Teapot with RoughHole and Sponge
Figure 9.10: Rendered images based on the proposed all-frequency BRT algorithm. From top to bottom: raw BRT data; compressed BRT data (SRBF); compressed BRT data (SRBF + CTA).The configurations of macro- and meso-scale radiance transfer are listed in Tables 8.1, 8.6, and 9.4.
Chapter 10
Conclusions and Future Work
10.1 Conclusions
In this dissertation, we have introduced two new data representations, univariate and multivari-ate spherical radial basis functions (SRBFs), and two novel compression algorithms, clustered tensor approximation(CTA) and K-clustered tensor approximation (K-CTA), for real-time data-driven rendering. While SRBFs provide an intrinsic and efficient description of univariate or multivariate spherical functions, CTA and K-CTA are powerful compression algorithms that allow sophisticated data analysis, compact storage space, and fast data reconstruction. Further-more, we have also described several important applications in computer graphics and vision.
Experimental results reveal that the proposed representations and approximation algorithms can be seamlessly integrated to obtain practical solutions of photo-realistic rendering at real-time rates.
10.1.1 Spherical Radial Basis Functions
Based on univariate/multivariate SRBFs, illumination and radiance functions can be modeled in their intrinsic spherical domain to avoid artifacts that result from false boundaries, distortions, and unnecessary parameterization. Most existing methods were not originally developed to handle spherical functions and deficient in this important feature. Therefore, they frequently have to model a spherical function in an inappropriate domain other than the unit hyper-sphere.
By contrast, the proposed SRBFs naturally bear the spherical domain in mind and are indeed suitable for directional data analysis.
Furthermore, all-frequency signals can be efficiently modeled by adjusting the bandwidth parameter of a SRBF. Previous articles have demonstrated that some high-frequency features, such as sharp shadow boundaries and specular highlights, have significant contributions to hu-man visual perception. Unfortunately, data representations based on the popular spherical har-monics frequently rely on discarding high-frequency parts to achieve compact results. This makes spherical harmonics difficult to adapt to various kinds of real-world illumination and
radiance functions. On the contrary, the proposed SRBFs allow localized high-frequency sig-nals to be easily described by their bandwidth parameters. This spatial localization property particularly distinguishes SRBFs from spherical harmonics. We thus expect that univariate and multivariate SRBFs will have a great impact on data representations for illumination and radi-ance functions in the future.
10.1.2 Tensor Approximation Algorithms
As for approximation algorithms, one major disadvantage of previous tensor-based approaches for real-time applications is that the decomposed results are not compact enough to be employed for efficient reconstruction on GPUs. We thus aim at overcoming this drawback by introducing the concept of sparse representation into multi-linear models, and effectively integrate cluster-ing, sparse codcluster-ing, and tensor approximation into a unified framework. As a result, the pro-posed CTA and K-CTA algorithms are especially appropriate for analyzing multi-dimensional data sets in real-time applications. When compared to traditional tensor approximation, the rendering performance of CTA and K-CTA is frequently improved by a substantial factor with only slight increases in approximation errors and amounts of compressed data. Moreover, K-CTA can also exploit inter-cluster coherence for smooth transitions across different physical conditions by mixing the decomposed results of multiple clusters.
At first glance, CTA and K-CTA may seem to have similarities with previous matrix factor-ization methods [90, 124]. Nayar et al. [124] introduced two-stage singular value decomposi-tion to exploit inter-block coherence, but their approach does not allow sparse representadecomposi-tions.
Lawrence et al. [90] instead proposed to include sparsity penalty in the objective function so that the final results tend to be sparse. Nevertheless, the weighting parameter of sparsity penalty should be tuned for different data sets in order to obtain a pre-defined number of non-zero terms.
By contrast, the proposed CTA and K-CTA algorithms permit sparse representations in which the number of non-zero terms is inherently enforced. For K-CTA, the sparsity can even be controlled by a user-defined threshold. In this way, an upper bound on run-time reconstruction costs is guaranteed. We believe that this property of CTA and K-CTA is significant for real-time applications, since run-time performance is predictable and totally under user control.
10.1.3 Summary
Additionally, we can further combine the advantages of SRBFs and CTA/K-CTA to provide more compact compressed data, while still achieving real-time rendering rates at run-time. The proposed all-frequency radiance transfer algorithms in Chapter 9 especially demonstrate that tight corporation between SRBFs and tensor approximation algorithms is possible and often favorable for practical real-time applications in computer graphics and vision. The successful radiance transfer frameworks also indicate that parametric representations, such as univariate
and multivariate SRBFs, and non-parametric models, for example CTA and K-CTA, may not actually conflict with each other.
In brief, we suggest that it is often preferable to employ univariate or multivariate SRBFs for modeling spherical functions, as they usually lead to more compact storage space and faster rendering performance on GPUs. Nevertheless, a slight quality loss and long computation time are inevitable consequences. By contrast, CTA and K-CTA indeed provide a more general, flexible, and accurate data analysis tool for various kinds of multi-dimensional visual data sets.
They are not restricted to a pre-defined functional form and only need efficient linear algebra computation. However, CTA and K-CTA have higher reconstruction costs and often require additional auxiliary data for efficient run-time rendering on GPUs.