What are Radiance Fields
Radiance Fields represent an innovative solution to challenges in inverse rendering and novel view synthesis, where the goal is to generate highly realistic 3D scenes from a set of 2D images.
This allows you to view a scene or object from any arbitrary angle. Additionally, Radiance Fields model view-dependent effects, such as reflections and highlights, which change based on the viewer's perspective, resulting in a more lifelike rendering.
"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."
How Do Radiance Fields Work?
Radiance Fields create a volumetric representation of a scene using a deep neural network. This network takes as input a 5D coordinate—comprised of the spatial location and the viewing direction—and outputs the scene's volume density and emitted radiance at that point. By synthesizing views from different angles, the model generates new perspectives that capture fine visual details and dynamic lighting interactions.
The two most widely adopted radiance field techniques are Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), both excelling in capturing complex lighting and geometry. Other methods, such as Trilinear Point Splatting, 3D Gaussian Ray Tracing, and Plenoxels, offer alternative approaches that prioritize computational efficiency or rendering speed.
Structure from Motion (SfM) and Radiance Fields
Many radiance field methods begin by processing standard 2D images through a Structure from Motion (SfM) algorithm. SfM aligns the images and generates sparse 3D point clouds that guide the training of the radiance field model. While the core principles of radiance field methods are similar, the training process can vary depending on the specific technique employed.
Radiance Fields vs. Photogrammetry
Radiance Fields often require fewer input images and can produce high-quality reconstructions more quickly than traditional methods like photogrammetry. However, while Radiance Fields excel in visual realism, they currently fall short in achieving the precise measurement accuracy required for certain industrial applications. By combining Radiance Fields with technologies like LiDAR or photogrammetry, users can achieve both visual realism and high accuracy for professional use cases.
Dynamic Radiance Fields
Beyond static scenes, Radiance Fields are also being extended to dynamic content, enabling the reconstruction of moving objects or evolving environments. This opens up possibilities for applications in video production, virtual reality, and simulations, where capturing dynamic light interactions is crucial. The ability to model dynamic content pushes the technology closer to a future where lifelike digital environments are created with ease, mirroring real-world events in real-time.
Who discovered Radiance Fields?
Introduced by Mildenhall et al. from Berkeley University, they unveiled Neural Radiance Fields in early 2020. The first author of NeRF, Ben Mildenhall recently recounted the first couple of weeks and initial tests at his keynote address at 3DV.
The Progress of Radiance Fields
What's Next for Radiance Fields?
The potential ceiling for Radiance Fields remains unknown. It is likely that new forms of radiance fields will continue to emerge. Another milestone was the introduction of RadSplat by Google, the first NeRF/3DGS hybrid that combines the strengths of both techniques. RadSplat achieves an astonishing 900 fps while maintaining the fidelity expected from NeRFs.
As new algorithms and optimization methods develop, the possibilities of Radiance Fields will continue to expand. Looking ahead, we can envision a time where photography and video are no longer the dominant imaging mediums. Instead, Radiance Fields promise a world where documenting our lives, businesses, and society will be done in hyper-realistic 3D, just as we experience the world every day.
From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, Radiance Fields stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.
Radiance Fields promise a time not too far off where we will no longer be using photography and videos as the dominant imaging medium and be able to routinely and with minimal effort be able to document our lives, business, and society in a hyper realistic way, similar to how we experience everyday life.
What are
Neural Radiance Fields
Neural Radiance Fields, abbreviated as NeRFs, are an emerging state-of-the-art solution to the problems of inverse rendering and novel view synthesis.
In these problems, the goal is to take a set of images of a subject from multiple angles and generate the most realistic 3D representation possible using a neural network. This allows you to look at the scene or object from any arbitrary angle. In addition, they also model view-dependent lighting effects, which means NeRFs can capture details like reflections that change depending on your viewing angle.
"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."
"We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis."
In other words, a deep neural network learns from a sparse set of input images, and the network's ability to predict radiance values for every point in the 3D space is honed over time. Ray casting, a technique in which rays are sent from the camera's position into the scene to calculate radiance values, plays a crucial role here. Rendering loss, an important metric in this method, is optimized to ensure that the rendered outputs closely resemble the training images. The continuous nature of the neural radiance field means it doesn’t rely on traditional voxel grids or meshes, contrasting with older methods of representing scenes.
Despite the complexity and intricacies of how neural radiance fields work, the technology's basic steps are straightforward. First, acquire data, usually photographs from different angles using standard cameras or even photogrammetry software. Next, feed this data into deep learning algorithms where the system is trained. Once trained, the NeRF can then synthesize or render new views of the scene, providing a new perspective previously unimagined.
Who created Neural Radiance Fields?
Introduced by Mildenhall et al. from Berkeley University, the NeRF representation creates a model in the form of a continuous volumetric scene function. This function, when queried with a ray casting from a particular viewing direction, returns the color and opacity values that, when combined, generate images that offer new perspectives of static scenes. These generated images are remarkably accurate and provide fine details, even from viewpoints vastly different from the input views.
Neural Radiance Fields were originally proposed and the full author list includes: Matthew Tancik, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng in 2020. Since then, a number of breakthroughs have pushed this field of research to the cutting edge. One such advancement was Instant-NGP, the project released alongside the research paper by Mueller, et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. Prior to the ingenious introduction of the multi-resolution hash encoding, it would take hours or even days to produce a high-quality NeRF. Now, the training process takes less than 15 minutes to produce photorealistic details. Instant-NGP went on to win one of top inventions of 2022 from Time Magazine.
The Progress of Neural Radiance Fields
NeRFs are almost something out of science fiction. They build upon the technology of light fields using concepts from artificial intelligence, machine learning, and neural networks. They mark a stark divergence from conventional triangle-mesh-based ray tracing. The key difference between NeRF and traditional photo-scanning is the potential for highly accurate reconstructions that look realistic to the human eye. Like the name implies, NeRFs are able to achieve such quality through the clever use of a type of neural network called an MLP (Multi-Layer Perceptron). By training the MLP, NeRFs are able to approximate the shape and color of reality through a process called differentiable rendering. It’s amazing, but NeRFs are also not limited to just one object, and can generate novel views of complex scenes and produce amazing results. We are still in the early days of NeRF and generative AI, but the amount of progress made so far has been staggering. It seems like every day, there’s a new NeRF paper capable of pushing computer graphics to the next level. Start ups such as Luma AI have greatly reduced the barrier to entry, allowing people with just an iPhone to capture incredible results. Now anyone can make a NeRF, so go grab your digital cameras and start nerfing!
This publication exists to help highlight the spectacular work that is being done across the world with NeRFs, and provoke the imagination of readers to embrace what once seemed impossible.
What's Next for NeRFs?
While NeRFs offer amazing performance in rendering static scenes, their ability to handle dynamic or moving objects in real-world scenarios is still an area for future exploration. The concept of viewing direction is paramount in NeRFs, as the representation needs to accurately capture how light, from different directions, interacts with the scene geometry, be it buildings, a person, or even a simple table. The reflection, shading, and bounce of light off objects and materials are vital to the synthesis process.This article just touches on the tip of the iceberg when it comes to understanding NeRFs. The depth and breadth of this topic are immense, and every section, from ray casting to deep learning, is a deep dive into the exciting world of neural rendering.
From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, NeRFs stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.
What is 3D Gaussian
Splatting?
3D Gaussian Splatting (3DGS) is a Radiance Field reconstruction method with an explicit rasterization-based representation rather than using a implicit one like in Neural Radiance Fields (NeRFs).
Who created 3D Gaussian Splatting?
3DGS emerges from the seminal paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.
3D Gaussian Splatting or 3DGS has quickly spread across the world and onto a variety of platforms, including Unreal Engine, Unity, Nerfstudio, Polycam, Luma AI, Volinga, Kiri Engine, and more. Basically, if a company offers a radiance field solution, they are also offering an implementation of Gaussian Splatting at this point.
3DGS was released in late April of 2023 and quickly became inordinately popular, winning best paper at SIGGRAPH 2023 in August.
The Progress of 3D Gaussian Splatting
Gaussian Splatting has had a meteoric rise since it was published in early 2023. The pure volume of papers building and exploring in the space has been staggering, with large progress being made.
Additionally, we have seen several companies, building both publicly and in stealth mode begin utilizing 3D Gaussian Splatting.
There has also been a surge of using 3D Gaussian Splatting in both text or image to 3D Generative AI models, despite them only being created recently. Its explicit representation makes it easy to work with and thus we have seen people take advantage of it.
People have also been creating experiential and educational content from Gaussian Splatting and using the host of viewers and distributors, such as Spline, PlayCanvas's Super Splat, and Antimatter15's web viewer.