8 The Basics
This chapter lays the land for physically modeling light-matter interactions in rendering, discussing the different levels at which the modeling can occur. We will then introduce one such model, an extremely high-level model that abstracts away almost all the underlying physics and models an object as its apparent reflection and transmission spectra. This simple modeling is functionally very useful, as it is very commonly used in practice, and serves as a curious teaser for the remaining chapters: how and when can this simple model be a good approximation of the sophisticated light-matter interactions? We will conclude with a brief overview of radiometry, which provides the necessary analytical tools we will use in physical modeling.
\[ \def\oi{{\omega_i}} \def\os{{\omega_s}} \def\Oi{{\Omega_i}} \def\Os{{\Omega_s}} \def\d{{\text{d}}} \def\D{{\Delta}} \def\do{{\d\omega}} \def\Do{{\Delta\omega}} \def\doi{{\d\omega_i}} \def\dos{{\d\omega_s}} \def\Doi{{\D\omega_i}} \def\Dos{{\D\omega_s}} \def\H{{\mathbf{H}}} \newcommand{\cM}{\mathcal{M}} \newcommand{\cL}{\mathcal{L}} \]
8.1 Overview
When a beam of photons hits a material surface, some of the photons will be scattered directly back to your eyes, and others will penetrate into the material. These surface phenomena are governed by surface scattering. We use the word “scattering” here to generally refer to lights coming back from the surface. Depending on the material, some of the scattered photons are along the perfect mirror-reflection directions, and others might be more diffuse. You might sometimes see the word “reflection” used. Reflection is sometimes used in the same way as scattering, which will be our use, but other times is reserved for the perfect, mirror-like reflection. Usually what the word means is self-evident given the context, but we will err on the side of verbosity when we want to mean a specific form of reflection.
Photons that penetrate the surface will further interact with particles in the material, which absorb, scatter, or might even emit photons. This is called subsurface scattering (SSS) in computer graphics. Even though we use the term “scattering”, you should know that the actual SSS processes involve not only scattering but also absorption and emission. It turns out that the principles that govern SSS are exactly the same as those that govern the interactions between photons and particles in the so-called “participating media”, such as clouds, fogs, and smokes. In computer graphics, light transport in participating media is called volume scattering, and again, even though we use the term “scattering”, absorption and emission are usually involved in the most general cases.
The way to model SSS/volume scattering is different from the way to model surface scattering: we no longer consider the material as a continuous surface and the light-matter interaction as photons bouncing off of the surface; instead, we break a material down into small particles and model how photons interact with individual particles.
Very importantly, the difference in the modeling methodology does not imply that there somehow is a fundamental difference between surface scattering and volume scattering. Ultimately, both are caused by the light, an oscillating electromagnetic field, exciting discrete electric charges. The differences lie in how the charges are arranged in space and in relation to one another. The laws that govern how photons interact with the charges are described by the electromagnetic theories in the classical regime and, in the quantum regime, by the quantum electrodynamics (QED) 1. In fact, using the electromagnetic theories, we can show that surface reflection/refraction is nothing more than the coherent scattering of incident light waves by the surface particles.
Since there is no fundamental differences in the underlying physics, the only meaningful distinction is one between different phenomenological approximations, or “models”, of the same underlying physics. We totally could invoke the electromagnetic theories or QED, and if we did, we would have one single unified model that explains both surface scattering (reflection and refraction) and volume scattering. Doing so, however, is not only unnecessary (because many, not all, real-world material color phenomena could be modeled without them) and too computationally expensive, but also, perhaps more importantly, blinds us from the relatively simple intuitions in each scenario. Instead, each phenomenological model is based on a set of high-level guiding principles, which are approximations of the underlying physical process but are sufficient to quantitatively describe light-matter interactions in each scenario.
Johnsen (2012) is a great reference, which has some equations but generally focuses on building intuitions and mostly uses the electromagnetic language rather than the quantum language. If you want to get to the nuts and bolts of the mathematical modeling, Bohren and Clothiaux (2006) is a phenomenal text whose models are also built in the electromagnetic land. Feynman (1985) has an accessible and breathtaking introduction to QED that I highly recommend. Dorsey, Rushmeier, and Sillion (2010) is a classic text on material appearance modeling in graphics that covers a range of topics, including modeling, measurements, and various implementation issues in practice. Johnston-Feller (2001) is specifically concerned with paintings; it has many interesting discussions of pigments and pigment mixtures and has many real-world data and insights that are rarely found elsewhere.
8.2 Observed Reflection and Transmission
Regardless of the details of surface scattering and volume scattering, a material appears to have some color because some photons leaving the material enter our eye. If we observe the material from the same side of the light source, it is the lights reflected from the material that matter. If we observe the material from the other side of the light source, it is the light transmitted through the material that matter. At the highest level of abstraction, we can model the material color in the real world by modeling the observed reflection and transmission apparent to an outside observer: how much of the incident power is reflected/transmitted back to the eye?
We can quantify the observed reflection and transmission using the spectral reflectance function \(r(\lambda)\) and the spectral transmittance function \(t(\lambda)\), respectively. These two functions spare us the details of how lights interact with a material but describe, at each wavelength \(\lambda\), the percentage of optical power that is reflected back to the eye or transmitted through the material and enters the eye, respectively.
Figure 8.1 (a) illustrates this modeling at work using the famous The Astronomer by Johannes Vermeer. We will proceed with our discussion using reflectance, but the case idea can be easily extended to transmittance. Vermeer paints an astronomer looking at a globe. Given the illumination coming from the window \(\Phi(\lambda)\) and the spectral reflectance of the point on the globe under gaze \(r(\lambda)\), the light reflected toward the eye is then \(\Phi(\lambda)r(\lambda)\). We can then calculate the color of these lights using the cone fundamentals or some set of CMFs, the same way as if the lights were directly emitted from the globe.
As another example, Figure 8.1 (b) is a photo of Acadia Redfish I took when visiting the Ripley’s Aquarium of Canada. The fish ordinarily looks red-ish under a white-ish light, which suggests that its spectral reflectance \(r(\lambda)\) peaks at longer wavelengths: it scatters more long-wavelength, i.e., red-ish, lights than short-wavelength lights. But the fish appears colorless in the aquarium, which simulates the lighting environment in the deep sea where lights \(\Phi(\lambda)\) are predominately blue/violet 2. As a result, the scattered lights have a rather uniform spectral power distribution, resulting in a gray-ish appearance.
Figure 8.1 makes an important simplification: the reflectance of a point \(p\) on the material is simplified to only a single spectrum. In reality, the reflectance of a point \(p\) depends on both \(\oi\), the direction of the light incident on \(p\), and \(\os\), the outgoing direction (leaving \(p\)) through which one observes the material. In certain materials where SSS contributes to the material appearance (e.g., translucent materials like jade), the reflectance can also depend on light incident on other points of the material surface. So when we use a single reflectance spectrum to model material colors, what we have implicitly assumed is that the reflectance spectrum has been calculated in such a way that when you multiply it with the incident illumination, you get the scattered light power that is actually observed.
How such a reflectance spectrum can be obtained in measurement (to the extent that it is a useful high-level abstraction) will be discussed in Section 9.6. The reflectance is a “quick-and-dirty” abstraction that we often use to give a rough estimation/explanation of a material’s color, but it is so high-level that it hides lots of the low-level details: what exactly are the light-matter interactions that cause the surface and subsurface scattering behaviors that eventually give rise to the apparent reflectance and transmittance spectra? The remaining chapters in this part essentially answer this question.
8.3 Key Concepts in Radiometry
To be more formal about surface and volume scattering, we need to scientifically define a few physical properties pertaining to light propagation spatially and angularly. This is called radiometry, which operates completely at the geometric optics level, so we will be describing light as a collection of photons, each of which can travel along a particular direction with certain energy associated with it. Reinhard et al. (2008, Chpt. 6) and Bohren and Clothiaux (2006, Chpt. 4) have more rigorous treatments of radiometry. Here, we introduce the language and a few important radiometric quantities that are relevant to our discussion.
8.3.1 Energy and Power
Each photon carries a certain amount of energy that is determined by its wavelength governed by:
\[ \begin{align} Q = \frac{hc}{\lambda}, \end{align} \]
where \(c\) is the speed of light, \(\lambda\) is the photon wavelength, and \(h\) is the Planck’s constant.
Power, or more formally in radiometry, radiant flux (or simply flux) is the total amount of energy passing through some surface in space per unit time. Or, taking a calculus perspective, power \(\Phi\) is defined as:
\[ \begin{align} \Phi = \lim_{\Delta t \rightarrow 0}\frac{\Delta Q}{\Delta t} = \frac{\text{d}Q}{\text{d}t}. \end{align} \]
The way to think about this is that each photon carries a certain amount of energy so if you monitor photons passing across a surface over a period of time \(\Delta t\), you can calculate the average power of that period by dividing the total energy passed by by \(\Delta t\). As \(\Delta t\) approaches 0, we get the instantaneous power.
Of course, energy/power is a function of wavelength, so more rigorously we should be talking about spectral power \(\Phi(\lambda)\), which has a unit of \(\text{W}/\text{nm}\):
\[ \begin{align} \Phi(\lambda) = \lim_{\Delta \lambda \rightarrow 0}\frac{\Delta \Phi}{\Delta \lambda} = \frac{\d \Phi}{\d \lambda}, \end{align} \]
where \(\Delta \Phi\) is the total power within a wavelength interval \(\Delta \lambda\).
8.3.2 Irradiance
Our power calculation is done with respect to a surface area, but how about the power at each point on the surface area? You can imagine that some points get more photons and others get fewer, so it is useful to characterize the power at any given point. Technically, the answer to the question “how many photons hit a particular point” is zero, since the area of a single point is 0. The meaningful question is: what is the power density of a particular point \(p\)? Irradiance is such a quantity.
Imagine again that you are monitoring photons crossing a surface for a period \(\Delta t\); you can calculate the average power received per unit area by dividing the average power by the surface area, and when you shrink the surface area to an infinitesimal point \(p\), we can calculate the power density, i.e., the irradiance, of \(p\) by:
\[ \begin{align} E(p) = \lim_{\Delta A \rightarrow 0}\frac{\Delta \Phi(p)}{\Delta A} = \frac{\text{d}\Phi(p)}{\text{d}A}. \end{align} \tag{8.1}\]
Irradiance is a more primitive measure than power: in calculus terms, irradiance is a power density function, which means we can derive the power of a surface by integrating the irradiance over the surface area:
\[ \begin{align} \Phi = \int^{A}E(p)dA. \end{align} \]
Irradiance has a unit of \(\text{W}/\text{m}^\text{2}\), and spectral irradiance has a unit of \(\text{W}/(\text{m}^\text{2} \cdot \text{nm})\).
8.3.3 Solid Angle
Irradiance is concerned with the power of all the photons incident on a point, but photons hit a point from all directions, so how do we quantify the amount of light coming from a direction?
A direction is a vector, which is invariant to translational transformations, so the two parallel “arrows” \(r_1\) and \(r_2\) in Figure 8.2 (left) represent the same vector/direction. Therefore, conceptually it is easier if we translate all the arrows so that they start from the same origin when we want to reason about a collection of directions.
How do we count the number of directions? In 2D, we use a planar angle to measure the amount of directions. Given an origin \(O\) and a vector, we rotate it to generate an arc. The angle subtended by the arc and \(O\) is a measure of the amount of directions we have just covered. The angle can also be mathematically given by the ratio \(s/r\), where \(s\) is the arc length and \(r\) is the radius of the circle. This matches our intuition that if we increase the radius of the circle, we would get a longer arc but the same angle. A full circle has a planar angle of \(2\pi\).
We can similarly define the size of a set of directions in 3D. We draw a sphere around \(O\), and imagine that we have some area on the spherical surface. Connecting \(O\) to every point on that area represents a direction in 3D. So the spherical surface area is a measure of the amount of 3D directions. Like in the 2D case, we want the measure to be invariant to the spherical radius, so we define solid angle, a measure of the size of a set of 3D directions, as:
\[ \begin{align} \Omega = \frac{A}{r^2}, \end{align} \tag{8.2}\]
where \(A\) is an area on a spherical surface and \(r\) is the radius. The unit of a solid angle is the steradian (\(sr\)), and the entire sphere subtends a solid angle of \(4\pi\).
Sometimes we want to know the size of the set of directions from a point \(O\) to an arbitrary surface. We would project that surface to a sphere and get a projected spherical area \(A\), using which we can invoke Equation 8.2 to estimate the solid angle subtended by the surface. One useful trick that might help sometimes is to project the surface to the unit sphere (i.e., \(r=1\)), and the solid angle is mathematically equivalent to the projected area on the unit sphere. But the most useful intuition I use whenever I am confused about what a particular solid angle means is to always think of the set of directions/vectors that are represented by that solid angle.
8.3.4 Radiance
We can now ask, what is the amount of flux received by a point from a particular direction? Photons travel in all sorts of directions. Let’s assume that we place an imaginary flux detector with an area \(A\) in the field. The detector is able to receive light from only one direction \(\omega\), as illustrated in Figure 8.2 (c). We can then read out the total flux \(\Phi\) received by the detector, from which we know that the power per unit area along the direction \(\omega\) is simply \(\frac{\Phi}{A}\).
Now imagine that we orient the detector so that its normal subtends an angle \(\theta\) with respect to the light direction \(\omega\). Figure 8.2 (b) explicitly illustrates this angle, where the tilted detector lies in the \(xy\)-plane, and the \(z\) direction is the normal \(n\). In a spherical coordinate system, a direction \(\omega\) can be parameterized by two angles: a polar angle \(\theta\) and an azimuthal angle \(\phi\).
The total flux received by the detector has changed to \(\Phi\cos\theta\), because the area that is available to receive photons is now \(A\cos\theta\). We call this the “effective area”. As a result, the power per area at the direction \(\omega\) remains the same, i.e., \(\frac{\Phi}{A}\). This is not surprising, because we are not changing the radiation field, only how we measure it. When the effective area reaches 0 (i.e., the detector is completely parallel to the light direction), the detector collects no photons, but it certainly does not mean that there is no light in the field.
If we now want to measure light power coming from another direction, we would change the detector so that it receives light from only that direction. In reality, this is, of course, not possible. No detector can screen lights only from one direction. If we place a detector in a radiation field, it is going to receive photons from all sorts of directions. We can limit the directions of photons that the detector collects by placing a baffle that allows only certain directions to hit the detector.
This setup is illustrated in Figure 8.3 (left). The total flux collected by the detector is \(\Delta \Phi\), the detector size is \(\D A\), and the solid angle subtended by the baffle is \(\D \omega\). The average power collected per unit “effective area” per unit direction by the detector is then:
\[ \begin{align} \frac{\D \Phi}{\D A \cos\theta \D \omega}. \end{align} \]
The baffle does a good job of rejecting many directions that are outside \(\D \omega\), but unless it is infinitely long, the detector will still collect some photons traveling through directions outside \(\D \omega\). But as we reduce the detector size and the baffle size, the baffle becomes a very thin cylinder over a very small detector, which collects light from a very small area along a very small solid angle, visualized in Figure 8.3 (right) 3. In calculus terms, when we let the detector size and baffle’s solid angle approach 0, we obtain the quantity called radiance:
\[ \begin{align} L(p, \omega) = \lim_{\D \omega \rightarrow 0} \lim_{\D A \rightarrow 0} \frac{\D \Phi}{\D A \cos\theta \D \omega} = \frac{\text{d}}{\text{d}\omega}\frac{\text{d}\Phi(p)}{\text{d}A \cos\theta} = \frac{\text{d}^2\Phi(p)}{\text{d}\omega\text{d}A \cos\theta}. \end{align} \tag{8.3}\]
Equation 8.3 is the definition of radiance, and it can be rewritten to Equation 8.4 given the definition of irradiance (see Equation 8.1).
\[ \begin{align} L(p, \omega) = \frac{\text{d}E(p)}{\text{d}\omega\cos\theta}. \end{align} \tag{8.4}\]
Radiance is an intrinsic property of the radiation field, and the reason we have the \(\cos\theta\) term in the definition is merely due to the way we have chosen to measure the property (using a detector that is \(\theta\)-oriented). Radiance has a unit of \(\text{W}/(\text{m}^\text{2}\cdot \text{sr})\), and spectral radiance has a unit of \(\text{W}/(\text{m}^\text{2}\cdot \text{sr} \cdot \text{nm})\).
Looking at the effective area in Figure 8.3, if the irradiance at the infinitesimal area \(p\) is \(\d E(p)\), the irradiance at the (infinitesimal) effective area (projected from \(\d A\) along \(\omega\)) is \(\frac{\d E(p)}{\cos\theta}\), which we denote \(\d E_\bot(p)\). Combining this with Equation 8.4, radiance \(L(p, \omega)\) can also be defined as:
\[ \begin{align} L(p, \omega) = \frac{\text{d}E_\bot(p)}{\text{d}\omega}. \end{align} \tag{8.5}\]
Equation 8.5 and Equation 8.4 each corresponds to a concrete way of measuring the radiance. Equation 8.5 places the detector perpendicular to the direction of light that we care to measure, and the detector used by Equation 8.4 is \(\theta\)-oriented with respect to the direction of interest. They give us an identical radiance result because radiance, again, is an inherent property of the radiation field invariant to how we measure it.
Radiance is a density function: the density of power at a point along a direction. As with any density function, it is useful when it gets integrated to compute some other quantities. For instance, given the radiance \(L(p, \omega)\), the irradiance at \(p\) is given by:
\[ \begin{align} E(p, \Omega) = \int^{\Omega}L(p, \omega)\cos\theta\text{d}\omega. \end{align} \tag{8.6}\]
Here we write the irradiance as \(E(p, \Omega)\) to explicitly signify that the irradiance depends not only on the specific position \(p\) but also the solid angle \(\Omega\) over which the lights are coming.
Using the interpretation of radiance in Equation 8.5, we can also give a more operational interpretation of Equation 8.6: we first calculate the infinitesimal irradiance \(\d E_\bot(p) = L(p, \omega)\do\) made by lights at the direction \(\omega\), then “transfer” that to the infinitesimal irradiance at the detector surface through the \(\cos\theta\) factor, and then repeat this for all the directions to accumulate the contributions from all directions.
8.4 Lambert’s Cosine Law
A Lambertian emitter or an ideal diffuse emitter is a flux-emitting point whose emitted radiance is constant regardless of the outgoing direction. A related concept is a Lambertian scatterer or an ideal diffuse surface, which is a surface point where the scattered radiance is independent of the scattering direction.
It might come as a surprise that the flux emitted by a Lambertian emitter through a fixed solid angle is different for different emission directions. Consider a setup where a Lambertian emitter has an infinitesimal area \(\d A\). The power emitted by \(\d A\) toward its normal direction in an infinitesimal solid angle of \(\do\) is \(\d\Phi_0 = L\do\d A\), where \(L\) is the radiance. The power emitted toward an oblique direction \(\omega\) through the same solid angle is \(\d\Phi_\theta = L \do \cos\theta \d A\).
In radiometry, the ratio of infinitesimal power and infinitesimal solid angle is called the radiant intensity4, denoted \(I\):
\[ \begin{align} I(\omega) = \frac{\d\Phi}{\do}. \end{align} \tag{8.7}\]
\(I\) is a meaningful measure only for a point source (e.g., our infinitesimal Lambertian emitter here). We can see that for a Lambertian emitter, the radiant intensity decays by a factor of \(\cos\theta\): \(\frac{\d\Phi_\theta}{\d\omega} = \frac{\d\Phi_0}{\d\omega}\cos\theta\). This is usually called the Lambert’s cosine law, named after Johann Heinrich Lambert, from his Photometria (Lambert 1760). Similarly, if we have a Lambertian scatterer, its scattered radiant intensity will also decay by \(\cos\theta\) as the polar angle \(\theta\) of the viewing direction \(\omega\) increases.
Figure 8.4 compares the radiance distribution and radiant intensity distribution of a Lambertian emitter/scatterer. Both distributions are over the entire hemisphere, but we show only a cross section. The distributions are visualized as two lobes, and the distance of a point on the lobe to the origin is proportional to the value at that point. The radiance distribution is constant regardless of \(\omega\) but the radiant intensity is proportional to \(\cos\theta\). This difference stems from the fact that intensity is defined with respect to the power at the detector/emission area (\(\d A\)) while radiance is defined with respect to power at the effective area (\(\d A \cos\theta\)).
8.5 The Measurement Equation
Given that we have the basic understanding of radiometry, now seems like a good time to show how radiometry is of fundamental importance to computer graphics and imaging. For simplicity, let’s just consider one single pixel with a setup illustrated in Figure 8.5.
Each pixel is very small, but it has a finite area, say \(A_p\). Each pixel is constantly being bombarded by lights that enter the aperture, which has an area \(V\). The raw pixel value is roughly proportional to the energy it receives during the exposure time5. So using the basic radiometry, we can write the total energy received by a pixel during the exposure time \(T\) as:
\[ \begin{align} Q = \int^{T} \int^{A_p} \int^{\Omega(p, V)} L(p, \omega) \cos\theta~\text{d}\omega~\text{d}p~\text{d}t, \end{align} \tag{8.8}\]
where \(\Omega(p, V)\) explicitly expresses that a solid angle is determined by the aperture \(V\) and a point \(p\) on the pixel surface. Of course this quantity changes with \(p\). We sometimes omit \(p\) and \(V\) when it is clear what \(\Omega\) refers to, but here, since the solid angle changes with the dummy variable \(p\) in the integral equation, we express it explicitly. In graphics literature, this equation is sometimes called the measurement equation of an image sensor (Kolb, Mitchell, and Hanrahan 1995; Reinhard et al. 2008, Chpt. 6.8.1; Pharr, Jakob, and Humphreys 2023, Chpt. 5.4).
The inner integral in Equation 8.8 is expressed over the solid angle, which varies with \(p\). A more common, but equivalent, formulation of the measurement equation is to re-express the inner integral over the aperture area \(V\):
\[ \begin{align} Q = \frac{1}{d^2} \int^{T} \int^{A_p} \int^{V} L(p, \omega) |\cos^4\theta|~\text{d}p'~\text{d}p~\text{d}t, \end{align} \tag{8.9}\]
where \(d\) is the distance between the aperture plane and the sensor plane, and \(p'\) is a point on the aperture plane. The derivation is available in standard texts (Pharr, Jakob, and Humphreys 2023, Chpt. 5.4.1) and is omitted here.
The measurement equation is concerned with the radiance distribution inside a camera, but the only reason there is a radiance distribution inside the camera is because there is an external radiance distribution in the scene impinging upon the camera optics, which act as a transfer function that turns external radiance into internal radiance. The transfer function is determined by the material properties of the camera optics (e.g., lenses, filters, etc.), whose effects are nothing more than surface scattering and volume scattering, topics of the next two chapters.
Using Figure 8.5 as a concrete example, to know the radiance \(L(p, \omega)\) inside the camera, we need to know \(L(p', \omega')\), the corresponding radiance in the scene and how the latter is transferred to the former. If the camera is an ideal pinhole, we have \(\omega = \omega'\) and \(L(p, \omega) = L(p', \omega')\) (ignoring diffraction). If the camera uses an ideal convex lens, the relationship between the two rays is governed by the Gauss lens equation (Section 11.3.1) and, with some simplifications, \(L(p, \omega) = L(p', \omega')\) still holds (Section 11.3.7). The transfer function is more complicated when as the camera optics become more complicated. Imaging we replace the lenses with a duck tape — how would the radiance be transferred?
The measurement equation is important because it fundamentally allows us to, in theory, synthesize/render any image taken by any camera at any viewpoint — given that we know the radiance distribution of the scene. Using Figure 8.6 as an example, let us simulate a new camera where the sensor is moved closer to the lens. To calculate the pixel value \(p_c\) of this new camera imaging the scene, it requires nothing more than invoking the measurement equation Equation 8.8 at \(p_c\), integrating over all the incident rays, which is a portion of the overall radiance distribution. This is why having access to the underlying radiance field allows us to synthesize new images.
Critically, observe that two of the rays that \(p_c\) needs are already captured by \(p_a\) and \(p_b\) in the current camera. So it is only natural to ask: can we synthesize new images from images taken from the same scene? How do we systematically reason about this? Read on.
8.6 Light Field and Radiance Field
There is a name for the distribution of the radiance in the space — it is called the light field, which refers to the complete set of all the possible radiances flowing through every possible direction. The light field is thus a function \(L(p, \omega, \lambda, t)\), describing the energy of a ray passing the position \(p\), along the direction \(\omega\), at time \(t\) and wavelength \(\lambda\). This function is also called the plenoptic function (Bergen and Adelson 1991; Gortler et al. 1996; Levoy and Hanrahan 1996). Figure 8.6 shows a tiny portion of the light field — six rays in fact; three inside the camera and three outside the camera.
8.6.1 Light-Field Imaging
The field of light-field imaging is concerned with measuring the light field of a scene, which is a task impossible — we cannot possibly measure the radiance of every single ray. There are some simplifications we can make. For instance, we can assume that a ray’s energy does not change in free space during propagation, so the plenoptic function is invariant along the ray traversal direction; we can also assume that the light field is time-invariant during the period of interest. But still, the task of measuring the entire field is a daunting one.
The next best thing is to sample the light field. A lens-based camera does a poor job of sampling the light field. The pixel \(p_a\) integrates a bundle of rays, two of which are shown. Even assuming that the ray’s radiance remains unchanged as it passes through the lens, the inherent integration by the pixel (i.e., the measurement equation in Equation 8.8) still means from the pixel value itself we could not decouple the radiance of the incident rays. Therefore, the ray that \(p_c\) wants cannot be easily extracted from \(p_a\). Using an ideal pinhole helps, but pinhole imaging comes with its own limitations that make it infeasible in practice (Section 11.2).
A vast literature exists in effective light-field sampling (Lam 2015, sec. 3). A good trade-off in practice is to insert a lenticular array or a microlens array between the main imaging lens and the sensor plane (Ng 2006; Adelson and Wang 1992). The idea was first conceptualized by Gabriel Lippmann (Lippmann 1908)6 Figure 8.7 shows one such example. Without the microlens array, a pixel (e.g., \(q\)) would integrate over all the rays that are subtended by the main lens, which is relatively large. Now we insert a microlens array and move the sensor plane a little farther back; each pixel (e.g., \(q\)) now integrates over a much smaller portion of the light field (rays subtended by a microlens), providing a higher angular resolution in light-field measurement.
8.6.2 Light-Field Rendering and Radiance Field
The main reason we want to measure the light field is so that we can render new images. Light-field rendering is concerned with rendering a new image at a novel perspective (or by a novel camera configuration) given a set of images from other perspectives/configurations. It is a form of image-based rendering. In this sense, many familiar tasks such as interpolating between video frames, panoramic photography, and (stereoscopic) 360\(^\circ\) video rendering are all light-field rendering in disguise.
Given that each image is a sample of a portion of the light field followed by a low-pass filter (i.e., the integration in Equation 8.8), rendering an image at a new perspective is nothing more than estimating another sample of the light field. As with any signal resampling task, the ideal solution to light-field rendering is to first estimate the underlying light field from a set of samples and then re-sample the light field given the new perspective. Signal filtering is necessary for both signal reconstruction and anti-aliasing, and the name of the game is to design good filters that are practically useful and computationally tractable.
Of course, modern image-based rendering, known under the name (neural) radiance-field rendering (Mildenhall et al. 2021; Kerbl et al. 2023), approaches the whole problem through machine learning and learns to reconstruct from massive amounts of data. To be precise, these methods do not reconstruct the light field; they reconstruct the radiance field.
Radiance Field
Radiance field, popularized by Mildenhall et al. (2021), applies a simplification and an addition to a light field. A radiance field is described by a function \(R(p, \omega, r, g, b, \sigma)\), describing the \((r, g, b)\) color and the density \(\sigma\) of a ray passing through a position \(p\) along the direction \(\omega\). Compare that with the plenoptic function, we can see that the radiance field function simplifies the the energy spectrum into just the tristimulus color values and assumes that the energy is time-invariant.
Importantly, the radiance field incorporates a new quantity, density, that is absent in the light field. Density has nothing to do with the energy of a ray; rather, it is/models an intrinsic property of the material (at position \(p\) along ray direction \(\omega\)). Materials are important for imaging and rendering, because they change the light field of the scene — through surface scattering and volume scattering. After all, rendering is a process of simulating the light-matter interactions.
In essence, a radiance field combines both a (simplified) light field, a property of the light, and a density field, a property of the materials. This simple extension from light to materials allows radiance-field methods to model (in fact, learn) material properties, which in turn enables more effective light-field rendering. Conventional light-field rendering, in contrast, does not attempt to decouple the light field from the material properties.
Radiance-field methods learn, from offline captured images (hence image-based rendering), to predict the tristimulus color values and density of a given point along a given direction:
\[ f: (p, \omega) \mapsto r, g, b, \sigma. \]
The function \(f\) can be parameterized in many ways. Two of most popular parameterizations are to use either a neural network (Mildenhall et al. 2021) or a mixture of Gaussians (Kerbl et al. 2023). With \(f\), we can then synthesize/rendering any image — by co-opting the classic volume rendering. We will study density and radiance field in much greater detail in Section 10.4.
8.6.3 Light-Field Display
Light-field display is a 3D display technology that attempts to reproduce the light field of a scene (Jones et al. 2007; Wetzstein et al. 2012; Lanman and Luebke 2013). Reproducing the light field provides the depth information of a scene that is missing in conventional 2D displays and can, thus, accurately drive the accommodation of eye lens in immersive (AR/VR) environment (Wann, Rushton, and Mon-Williams 1995; Hoffman et al. 2008). Other technologies include varifocal displays, multi-focal displays, and holographic displays.
Figure 8.8 shows a usual two-stage process of displaying a light-field. The first step is to capture the light field using some form of light-field imaging technique discussed in Section 8.6.1; here we use a pinhole array placed in front of the sensor plane. Each pinhole covers a small group of the pixels on the sensor; the image captured by the group of pixels under each pinhole is called an elemental image. Once we have recorded the light field, we can then reproduce it. This is done by displaying the elemental images, each with a display placed on the other side of the pinhole array. Note that the relative positions of the display pixels are reversed from that of the the image pixels during light-field recording.
8.7 Photometric Quantities
Spectral radiant flux (power), irradiance, radiant intensity, and radiance are all radiometric quantities. They all have a photometric counterpart, which weighs the radiometric quantity by the luminous efficiency function (LEF). The LEF, as we have discussed in Section 4.3.2, at a particular wavelength is inversely proportional to the radiometric quantity at each wavelength needed to produce the same level of perceptual brightness.
For instance, given a spectral radiant flux \(\Phi(\lambda)\), the corresponding photometric counterpart is then:
\[ \begin{align} \Phi_v(\lambda) = K \Phi(\lambda) V(\lambda), \end{align} \]
where \(\Phi_v(\lambda)\) is the spectral luminous flux, \(V(\lambda)\) is the LEF, and \(K\) is a constant that, for historical reasons, takes the value of 683.002. The total luminous flux is then:
\[ \begin{align} \Phi_v = \int_\lambda K \Phi(\lambda) V(\lambda) \d \lambda. \end{align} \]
Luminous flux has a unit of lumen (\(\text{lm}\)), so K has a unit of \(\text{lm}/\text{W}\). We can also weigh the radiant power by the scotopic LEF, in which case \(\text{K} = 1700\) (\(\text{lm}/\text{W}\)).
Other radiometric quantities can be similarly converted to the photometric counterparts. Specifically:
- the photometric counterpart of irradiance is illumination, which has a unit of \(\text{lx} = \text{lm}/\text{m}^2\), which is also called the lux;
- the photometric counterpart of radiance intensity is luminous intensity, which has a unit of \(\text{cd} = \text{lm}/\text{sr}\), which is called the candela;
- the photometric counterpart of radiance is luminance, which has a unit of \(\text{lm}/(\text{m}^2\text{sr}) = \text{cd}/(\text{m}^2)\), which is also called the nit.
Sometimes radiometric vs. photometric quantities are also called the radiant vs. luminous quantities. The way to interpret the photometric quantities is that they take into account the spectral sensitivity of a particular photodetector, which in our case is the photoreceptors on the retina. But if we use other detectors, such as an image sensor, we will have a different spectral sensitivity, and the corresponding photometric measurements will be different. We will study the spectral sensitivity of image sensors in Section 12.5.
A radiometer measures the absolute radiometric quantities, whereas a photometer reports photometric quantities. An image sensor and our retina can both be thought of as a photometer but the spectral sensitivities in the two cases are different, so the raw pixel readings and the photoreceptor responses are different even under an identical illumination.
The electromagnetic theories do not explain everything in light-matter interactions. Famously, they do not explain how the interference pattern in the double-slit experiment still arises even if the photons are delivered sequentially.↩︎
which results from a combination of water selectively absorbing medium-to-long wavelengths of light and increasing scattering of short wavelengths in the Rayleigh regime (Section 10.3.4).↩︎
It is just a visualization convention, but visualizing \(\do\) as a cylinder rather than a cone makes it easier to imagine what \(\d A \cos\theta\) is like.↩︎
Or simply, the “intensity”, which is an extremely overloaded term, so we will be verbose and use “radiant intensity” when we mean it.↩︎
Assuming there is no noise and there is no quantization error in converting analog signals to digital signals.↩︎
Lippmann did not get to implement the idea. He won the Nobel Prize in Physics in 1908 for inventing, for the first time, a method for color photography.↩︎