What are Radiance Fields
Radiance Fields, are an emerging state-of-the-art solution to the problems of inverse rendering and novel view synthesis.
In addition to being able to model hyper realistic static content, the promise of radiance fields also extends towards dynamic or moving content. This means that we will be able to experience not only the photography equivalent of 3D life, but videography as well.
Most of the radiance field methods utilize the same initial intake process, whereby standard 2D images are run through a alignment process named Structure from Motion (SfM) and then trains the resulting data. Depending on the radiance field method, the training implementation can be vastly different. The two most common radiance field methods to date are Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS).
Who created Radiance Fields?
Introduced by Mildenhall et al. from Berkeley University, they unveiled Neural Radiance Fields in early 2020. The first author of NeRF, Ben Mildenhall recently recounted the first couple of weeks and initial tests at his keynote address at 3DV.
The NeRF representation creates a model in the form of a continuous volumetric scene function. This function, when queried with a ray casting from a particular viewing direction, returns the color and opacity values that, when combined, generate images that offer new perspectives of static scenes. These generated images are remarkably accurate and provide fine details, even from viewpoints vastly different from the input views.
Now that there are multiple radiance field methods, we can only speculate what the future holds for them. The progression rate has been astounding to follow along within computer vision and I believe its use cases will extend far beyond the industry. You can follow along the progression on the website or through our discord server!
The Progress of Radiance Fields
What's Next for Radiance Fields?
The ceilings of radiance fields continues to be unknown. Additionally, it is likely that more forms of radiance fields emerge. Already in the beginning of 2024, we saw the introduction of a third type of radiance field, Trilinear Point Splatting for Real Time Radiance Field Rendering.
Additionally, we only recently saw the first NeRF/3DGS hybrid paper with RadSplat emerge from Google. This took the best of both worlds from each method and combines them into a unified power. RadSplat results in nearly 900 fps, with the fidelity that you expect from NeRFs.
From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, Radiance Fields stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.
Radiance Fields promise a time not too far off where we will no longer be using photography and videos as the dominant imaging medium and be able to routinely and with minimal effort be able to document our lives, business, and society in a hyper realistic way, similar to how we experience everyday life.
What are
Neural Radiance Fields
Neural Radiance Fields, abbreviated as NeRFs, are an emerging state-of-the-art solution to the problems of inverse rendering and novel view synthesis.
In these problems, the goal is to take a set of images of a subject from multiple angles and generate the most realistic 3D representation possible using a neural network. This allows you to look at the scene or object from any arbitrary angle. In addition, they also model view-dependent lighting effects, which means NeRFs can capture details like reflections that change depending on your viewing angle.
"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."
"We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis."
In other words, a deep neural network learns from a sparse set of input images, and the network's ability to predict radiance values for every point in the 3D space is honed over time. Ray casting, a technique in which rays are sent from the camera's position into the scene to calculate radiance values, plays a crucial role here. Rendering loss, an important metric in this method, is optimized to ensure that the rendered outputs closely resemble the training images. The continuous nature of the neural radiance field means it doesn’t rely on traditional voxel grids or meshes, contrasting with older methods of representing scenes.
Despite the complexity and intricacies of how neural radiance fields work, the technology's basic steps are straightforward. First, acquire data, usually photographs from different angles using standard cameras or even photogrammetry software. Next, feed this data into deep learning algorithms where the system is trained. Once trained, the NeRF can then synthesize or render new views of the scene, providing a new perspective previously unimagined.
Who created Neural Radiance Fields?
Introduced by Mildenhall et al. from Berkeley University, the NeRF representation creates a model in the form of a continuous volumetric scene function. This function, when queried with a ray casting from a particular viewing direction, returns the color and opacity values that, when combined, generate images that offer new perspectives of static scenes. These generated images are remarkably accurate and provide fine details, even from viewpoints vastly different from the input views.
Neural Radiance Fields were originally proposed and the full author list includes: Matthew Tancik, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng in 2020. Since then, a number of breakthroughs have pushed this field of research to the cutting edge. One such advancement was Instant-NGP, the project released alongside the research paper by Mueller, et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. Prior to the ingenious introduction of the multi-resolution hash encoding, it would take hours or even days to produce a high-quality NeRF. Now, the training process takes less than 15 minutes to produce photorealistic details. Instant-NGP went on to win one of top inventions of 2022 from Time Magazine.
The Progress of Neural Radiance Fields
NeRFs are almost something out of science fiction. They build upon the technology of light fields using concepts from artificial intelligence, machine learning, and neural networks. They mark a stark divergence from conventional triangle-mesh-based ray tracing. The key difference between NeRF and traditional photo-scanning is the potential for highly accurate reconstructions that look realistic to the human eye. Like the name implies, NeRFs are able to achieve such quality through the clever use of a type of neural network called an MLP (Multi-Layer Perceptron). By training the MLP, NeRFs are able to approximate the shape and color of reality through a process called differentiable rendering. It’s amazing, but NeRFs are also not limited to just one object, and can generate novel views of complex scenes and produce amazing results. We are still in the early days of NeRF and generative AI, but the amount of progress made so far has been staggering. It seems like every day, there’s a new NeRF paper capable of pushing computer graphics to the next level. Start ups such as Luma AI have greatly reduced the barrier to entry, allowing people with just an iPhone to capture incredible results. Now anyone can make a NeRF, so go grab your digital cameras and start nerfing!
This publication exists to help highlight the spectacular work that is being done across the world with NeRFs, and provoke the imagination of readers to embrace what once seemed impossible.
What's Next for NeRFs?
While NeRFs offer amazing performance in rendering static scenes, their ability to handle dynamic or moving objects in real-world scenarios is still an area for future exploration. The concept of viewing direction is paramount in NeRFs, as the representation needs to accurately capture how light, from different directions, interacts with the scene geometry, be it buildings, a person, or even a simple table. The reflection, shading, and bounce of light off objects and materials are vital to the synthesis process.This article just touches on the tip of the iceberg when it comes to understanding NeRFs. The depth and breadth of this topic are immense, and every section, from ray casting to deep learning, is a deep dive into the exciting world of neural rendering.
From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, NeRFs stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.
What is 3D Gaussian
Splatting?
3D Gaussian Splatting is a radiance field reconstruction method that is rasterization-based rather than using a neural network like in Neural Radiance Fields (NeRFs).
Who created 3D Gaussian Splatting?
3DGS emerges from the seminal paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.
3D Gaussian Splatting or 3DGS has quickly spread across the world and onto a variety of platforms, including Unreal Engine, Unity, Nerfstudio, Polycam, Luma AI, Volinga, Kiri Engine, and more. Basically, if a company offers a radiance field solution, they are also offering an implementation of Gaussian Splatting at this point.
3DGS was released in late April of 2023 and quickly became inordinately popular, winning best paper at SIGGRAPH 2023 in August.
The Progress of 3D Gaussian Splatting
Gaussian Splatting has had a meteoric rise since it was released less than a year ago. The pure volume of papers building and exploring in the space has been staggering, with large progress being made.
Additionally, we have seen several companies, building both publicly and in stealth mode begin utilizing 3D Gaussian Splatting.
There has also been a surge of using 3D Gaussian Splatting in both text or image to 3D Generative AI models, despite them only being created recently.
Its explicit representation makes it easy to work with and thus we have seen people take advantage of it.
People have also been creating experiential and educational content from Gaussian Splatting and using the host of viewers and distributors, such as Spline, PlayCanvas's Super Splat, and Antimatter15's web viewer.
What's Next for Gaussian Splatting?
With all the advancements in such a short period of time, it’s crazy to think where 3D Gaussian Splatting might be headed. I expect to see more implementations across various industries that are looking to utilize lifelike 3D content in their offerings,
On the research side, Generative AI using Gaussians have remained popular and I believe that we will see faster, more complex, and more hyper realistic outputs. Recently, we have begun to see work to pull high quality meshes out from Gaussian Splatting outputs, such as SuGaR, its follow up paper Gaussian Frosting, and Gaustudio.
There is currently no publicly known ceiling on the technology and will surely be one of the most exciting topics to follow along with over the coming years.
In its current state, 3D Gaussian Splatting struggles a bit with fine details, but another real time radiance field method, Trilinear Point Splatting for Real Time Radiance Field Rendering has proposed a solution.