Rendering

So far in the book, our focus has been on how our visual system encodes light spectrum coming from the scene into perception. We have, however, not cared much about how an object in the scene could produce light in the first place. An object could, of course, emit light itself. In the real world, however, the vast majority of objects have colors not because they emit light but because they interact with light that impinges upon them. The light-matter interaction modifies the energy spectrum of the incident light, and the modified light is scattered back to our eyes, giving rise to (color) vision.

A great deal of computer graphics is concerned with rendering the color of objects, and the name of the game is to model the light-matter interactions in a physically accurate manner so that the colors in the generated imagery look real. This part of the book focuses on the physical principles that govern light-matter interactions insofar as they are used in rendering photorealistic color images. We will not cover implementation-specific topics such as how these principles are supported in modern graphics programming models (e.g., OpenGL, Vulkan, and OptiX); nor will we cover how these programming models are supported on modern GPU hardware correctly and efficiently.

Any coverage of physics in rendering is necessarily an approximation — based on phenomenological models that abstract away unimportant details of the underlying physics while maintaining what is relevant for image synthesis. Deep learning and AI techniques push this kind of approximation to the extreme. With these techniques, rendering is re-branded as novel view synthesis, the prime example of which is the increasingly popular class of (neural) radiance-field rendering methods such as NeRF and 3DGS.

These methods are fundamentally image-based rendering: they sample, reconstruct, and re-sample the light field — using modern learning methods such as (stochastic) gradient descent. Interestingly, even though they do not exactly model the physics governing the light-matter interactions, their learning model is parameterized with physics-inspired formulations. By understanding the governing physics, we can better interpret these learning-based methods, understand their limits, and reason about potential opportunities for improvement.