Trending

Trending

Recents

Recents

Advertisement

Recent Jobs

Recent Headlines

About RADIANCE FIELDS

About RADIANCE FIELDS

What are Radiance Fields

Radiance Fields represent an innovative solution to challenges in inverse rendering and novel view synthesis, where the goal is to generate highly realistic 3D scenes from a set of 2D images.

This allows you to view a scene or object from any arbitrary angle. Additionally, Radiance Fields model view-dependent effects, such as reflections and highlights, which change based on the viewer's perspective, resulting in a more lifelike rendering.

"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."

How Do Radiance Fields Work?

Radiance Fields create a volumetric representation of a scene using a deep neural network. This network takes as input a 5D coordinate—comprised of the spatial location and the viewing direction—and outputs the scene's volume density and emitted radiance at that point. By synthesizing views from different angles, the model generates new perspectives that capture fine visual details and dynamic lighting interactions.

The two most widely adopted radiance field techniques are Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), both excelling in capturing complex lighting and geometry. Other methods, such as Trilinear Point Splatting, 3D Gaussian Ray Tracing, and Plenoxels, offer alternative approaches that prioritize computational efficiency or rendering speed.

Structure from Motion (SfM) and Radiance Fields

Many radiance field methods begin by processing standard 2D images through a Structure from Motion (SfM) algorithm. SfM aligns the images and generates sparse 3D point clouds that guide the training of the radiance field model. While the core principles of radiance field methods are similar, the training process can vary depending on the specific technique employed.

Radiance Fields vs. Photogrammetry

Radiance Fields often require fewer input images and can produce high-quality reconstructions more quickly than traditional methods like photogrammetry. However, while Radiance Fields excel in visual realism, they currently fall short in achieving the precise measurement accuracy required for certain industrial applications. By combining Radiance Fields with technologies like LiDAR or photogrammetry, users can achieve both visual realism and high accuracy for professional use cases.

Dynamic Radiance Fields

Beyond static scenes, Radiance Fields are also being extended to dynamic content, enabling the reconstruction of moving objects or evolving environments. This opens up possibilities for applications in video production, virtual reality, and simulations, where capturing dynamic light interactions is crucial. The ability to model dynamic content pushes the technology closer to a future where lifelike digital environments are created with ease, mirroring real-world events in real-time.

Who discovered Radiance Fields?

Introduced by Mildenhall et al. from Berkeley University, they unveiled Neural Radiance Fields in early 2020. The first author of NeRF, Ben Mildenhall recently recounted the first couple of weeks and initial tests at his keynote address at 3DV.

The Progress of Radiance Fields

Radiance Fields have been skyrocketing in popularity over the last year. It seems like there are daily 10+ papers that are being released and pushing boundaries. Additionally, the rate that people are solving problems and lowering barriers to entry also continues to fall. Whether that is through more startups building in the space or weaknesses of the methods being addressed, it seems like there is something exciting happening daily.

Moreover, there are quite a few companies that are building in stealth mode, actively building on top of the existing technology across such a wide variety of industries. We, as consumers will directly benefit from the creation of these companies and will push us forwards as a society to reap the benefits of a lifelike imaging medium.

This publication exists to help highlight the spectacular work that is being done across the world with Radiance Fields, and provoke the imagination of readers to embrace what once seemed impossible, but now is an ever shortening period of time.

Radiance Fields have been skyrocketing in popularity over the last few years.

It seems like there are daily 10+ papers that are being released and pushing boundaries. Additionally, the rate that people are solving problems and lowering barriers to entry also continues to fall. Whether that is through more startups building in the space or weaknesses of the methods being addressed, it seems like there is something exciting happening daily.

Moreover, there are quite a few companies that are building in stealth mode, actively building on top of the existing technology across such a wide variety of industries. We, as consumers will directly benefit from the creation of these companies and will push us forwards as a society to reap the benefits of a lifelike imaging medium.

This publication exists to help highlight the spectacular work that is being done across the world with Radiance Fields, and provoke the imagination of readers to embrace what once seemed impossible, but now is an ever shortening period of time.

What's Next for Radiance Fields?

The potential ceiling for Radiance Fields remains unknown. It is likely that new forms of radiance fields will continue to emerge. Another milestone was the introduction of RadSplat by Google, the first NeRF/3DGS hybrid that combines the strengths of both techniques. RadSplat achieves an astonishing 900 fps while maintaining the fidelity expected from NeRFs.

As new algorithms and optimization methods develop, the possibilities of Radiance Fields will continue to expand. Looking ahead, we can envision a time where photography and video are no longer the dominant imaging mediums. Instead, Radiance Fields promise a world where documenting our lives, businesses, and society will be done in hyper-realistic 3D, just as we experience the world every day.

From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, Radiance Fields stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.

Radiance Fields promise a time not too far off where we will no longer be using photography and videos as the dominant imaging medium and be able to routinely and with minimal effort be able to document our lives, business, and society in a hyper realistic way, similar to how we experience everyday life.

About Neural RADIANCE FIELDS

About Neural RADIANCE FIELDS

What are
Neural Radiance Fields

Neural Radiance Fields, abbreviated as NeRFs, are an emerging state-of-the-art solution to the problems of inverse rendering and novel view synthesis.

In these problems, the goal is to take a set of images of a subject from multiple angles and generate the most realistic 3D representation possible using a neural network. This allows you to look at the scene or object from any arbitrary angle. In addition, they also model view-dependent lighting effects, which means NeRFs can capture details like reflections that change depending on your viewing angle.

"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."

"We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis."

In other words, a deep neural network learns from a sparse set of input images, and the network's ability to predict radiance values for every point in the 3D space is honed over time. Ray casting, a technique in which rays are sent from the camera's position into the scene to calculate radiance values, plays a crucial role here. Rendering loss, an important metric in this method, is optimized to ensure that the rendered outputs closely resemble the training images. The continuous nature of the neural radiance field means it doesn’t rely on traditional voxel grids or meshes, contrasting with older methods of representing scenes.

Despite the complexity and intricacies of how neural radiance fields work, the technology's basic steps are straightforward. First, acquire data, usually photographs from different angles using standard cameras or even photogrammetry software. Next, feed this data into deep learning algorithms where the system is trained. Once trained, the NeRF can then synthesize or render new views of the scene, providing a new perspective previously unimagined.

Who created Neural Radiance Fields?

Introduced by Mildenhall et al. from Berkeley University, the NeRF representation creates a model in the form of a continuous volumetric scene function. This function, when queried with a ray casting from a particular viewing direction, returns the color and opacity values that, when combined, generate images that offer new perspectives of static scenes. These generated images are remarkably accurate and provide fine details, even from viewpoints vastly different from the input views.

Neural Radiance Fields were originally proposed and the full author list includes: Matthew Tancik, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng in 2020. Since then, a number of breakthroughs have pushed this field of research to the cutting edge. One such advancement was Instant-NGP, the project released alongside the research paper by Mueller, et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. Prior to the ingenious introduction of the multi-resolution hash encoding, it would take hours or even days to produce a high-quality NeRF. Now, the training process takes less than 15 minutes to produce photorealistic details. Instant-NGP went on to win one of top inventions of 2022 from Time Magazine.

The Progress of Neural Radiance Fields

NeRFs are almost something out of science fiction. They build upon the technology of light fields using concepts from artificial intelligence, machine learning, and neural networks. They mark a stark divergence from conventional triangle-mesh-based ray tracing. The key difference between NeRF and traditional photo-scanning is the potential for highly accurate reconstructions that look realistic to the human eye. Like the name implies, NeRFs are able to achieve such quality through the clever use of a type of neural network called an MLP (Multi-Layer Perceptron). By training the MLP, NeRFs are able to approximate the shape and color of reality through a process called differentiable rendering. It’s amazing, but NeRFs are also not limited to just one object, and can generate novel views of complex scenes and produce amazing results. We are still in the early days of NeRF and generative AI, but the amount of progress made so far has been staggering. It seems like every day, there’s a new NeRF paper capable of pushing computer graphics to the next level. Start ups such as Luma AI have greatly reduced the barrier to entry, allowing people with just an iPhone to capture incredible results. Now anyone can make a NeRF, so go grab your digital cameras and start nerfing!

This publication exists to help highlight the spectacular work that is being done across the world with NeRFs, and provoke the imagination of readers to embrace what once seemed impossible.

What's Next for NeRFs?

While NeRFs offer amazing performance in rendering static scenes, their ability to handle dynamic or moving objects in real-world scenarios is still an area for future exploration. The concept of viewing direction is paramount in NeRFs, as the representation needs to accurately capture how light, from different directions, interacts with the scene geometry, be it buildings, a person, or even a simple table. The reflection, shading, and bounce of light off objects and materials are vital to the synthesis process.This article just touches on the tip of the iceberg when it comes to understanding NeRFs. The depth and breadth of this topic are immense, and every section, from ray casting to deep learning, is a deep dive into the exciting world of neural rendering.

From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, NeRFs stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.

About 3D Gaussian Splatting

About 3D Gaussian Splatting

What is 3D Gaussian
Splatting?

3D Gaussian Splatting (3DGS) is a Radiance Field reconstruction method with an explicit rasterization-based representation rather than using a implicit one like in Neural Radiance Fields (NeRFs).

This allows for Gaussian Splatting to maintain the high visual fidelity and view dependent effects that Radiance Fields are known for, but also allows for real time rendering rates and performance, including on mobile devices!

There has been tremendous excitement about 3D Gaussian Splatting for Real Time Radiance Field Rendering. As its name implies, 3DGS is able provide high quality results while still rendering in real time.

Instead of relying on a neural network to model the radiance field, 3DGS uses 3D Gaussians to represent the scene. A Gaussian in this context is essentially a smooth, bell-shaped curve that can vary in width, height, and orientation, offering a flexible way to model the density and appearance of different parts of the scene.

Each 3D Gaussian models a volumetric "splat" in the scene, with properties like position, size, orientation, and color. This representation is both compact and expressive, allowing for detailed scenes to be modeled with fewer parameters than a dense voxel grid or a neural network would require.

To render a new view of the scene, the positions and properties of the 3D Gaussians are projected onto the 2D image plane of the camera viewpoint. This projection translates the 3D Gaussians into 2D "splats" on the image, where their contributions are blended together based on their properties and distances to the camera.

This allows for it to maintain the high visual fidelity and view dependent effects that NeRFs are known for, but also allows for real time rendering rates and performance, including on mobile devices!

There has been tremendous excitement about another radiance field method, 3D Gaussian Splatting for Real Time Radiance Field Rendering. As its name implies, 3DGS is able provide high quality results while still rendering in real time.

Instead of relying on a dense neural network to model the radiance field, 3DGS uses 3D Gaussians to represent the scene. A Gaussian in this context is essentially a smooth, bell-shaped curve that can vary in width, height, and orientation, offering a flexible way to model the density and appearance of different parts of the scene.

Each 3D Gaussian models a volumetric "splat" in the scene, with properties like position, size, orientation, and color. This representation is both compact and expressive, allowing for detailed scenes to be modeled with fewer parameters than a dense voxel grid or a neural network would require.

The rendering algorithm accounts for the visibility and orientation of each Gaussian, ensuring that only the visible parts contribute to the final image. This includes handling occlusions and leveraging anisotropic (directionally varying) properties to accurately render surfaces and edges.

By optimizing the properties of the 3D Gaussians directly, the method achieves fast training times. The renderer, designed for efficiency, leverages modern GPU architectures to achieve real-time rendering speeds while maintaining high visual quality.

Similar to NeRFs, the input data used are multiple 2D photos from various camera perspectives. Gaussian Splatting is able to create a highly accurate representation of a 3D space. Both NeRFs and 3D Gaussian Splatting use structure from motion (SfM) to begin aligning the images, but Gaussian Splatting generates a sparse point cloud.

Additionally, because of 3DGS’s structure, there have also been integrations with some web based platforms, such as Three.js, React Three Fiber, and 3D Design tool, Spline. These tools help make it possible to bring Gaussian Splatting into no code web design platforms such as Framer and Webflow, making it easy to show off your creations.

To render a new view of the scene, the positions and properties of the 3D Gaussians are projected onto the 2D image plane of the camera viewpoint. This projection translates the 3D Gaussians into 2D "splats" on the image, where their contributions are blended together based on their properties and distances to the camera.

The rendering algorithm accounts for the visibility and orientation of each Gaussian, ensuring that only the visible parts contribute to the final image. This includes handling occlusions and leveraging anisotropic (directionally varying) properties to accurately render surfaces and edges.

By optimizing the properties of the 3D Gaussians directly, the method achieves fast training times. The renderer, designed for efficiency, leverages modern GPU architectures to achieve real-time rendering speeds while maintaining high visual quality.

Similar to NeRFs, the input data used are multiple 2D photos from various camera perspectives. Gaussian Splatting is able to create a highly accurate representation of a 3D space. Both NeRFs and 3D Gaussian Splatting use structure from motion SfM to begin aligning the images, but Gaussian Splatting generates a sparse point cloud.

Gaussian Splatting
Gaussian Splatting
Gaussian Splatting
Gaussian Splatting

Who created 3D Gaussian Splatting?

3DGS emerges from the seminal paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.

3D Gaussian Splatting or 3DGS has quickly spread across the world and onto a variety of platforms, including Unreal Engine, Unity, Nerfstudio, Polycam, Luma AI, Volinga, Kiri Engine, and more. Basically, if a company offers a radiance field solution, they are also offering an implementation of Gaussian Splatting at this point.

3DGS was released in late April of 2023 and quickly became inordinately popular, winning best paper at SIGGRAPH 2023 in August.

The Progress of 3D Gaussian Splatting

Gaussian Splatting has had a meteoric rise since it was published in early 2023. The pure volume of papers building and exploring in the space has been staggering, with large progress being made.

Additionally, we have seen several companies, building both publicly and in stealth mode begin utilizing 3D Gaussian Splatting.

There has also been a surge of using 3D Gaussian Splatting in both text or image to 3D Generative AI models, despite them only being created recently. Its explicit representation makes it easy to work with and thus we have seen people take advantage of it.

People have also been creating experiential and educational content from Gaussian Splatting and using the host of viewers and distributors, such as Spline, PlayCanvas's Super Splat, and Antimatter15's web viewer.

What's Next for Gaussian Splatting?

On July 10th, 2024, NVIDIA Research unveiled 3D Gaussian Ray Tracing, as a departure from Splatting. As the name implies, it utilizes ray tracing rather than the rasterization that has traditionally been used with Gaussian Splatting.

There are several exciting advancements and functionalities that become readily available. It addresses one of the restrictions of Gaussian Splatting and allows for the reintroduction of fisheye lenses to captures. Additionally, secondary lighting effects such as refractions, shadows, depth of field, and mirrors.

This is a brand new line of research and it appears likely that it will attract appropriate research attention to push boundaries forwards.

With all the advancements in such a short period of time, it’s crazy to think where 3D Gaussian Splatting might be headed. I expect to see more implementations across various industries that are looking to utilize lifelike 3D content in their offerings,

On the research side, Generative AI using Gaussians have remained popular and I believe that we will see faster, more complex, and more hyper realistic outputs. Recently, we have begun to see work to pull high quality meshes out from Gaussian Splatting outputs, such as SuGaR, its follow up paper Gaussian Frosting, and Gaustudio.

There is currently no publicly known ceiling on the technology and will surely be one of the most exciting topics to follow along with over the coming years.

In its current state, 3D Gaussian Splatting struggles a bit with fine details, but another real time radiance field method, Trilinear Point Splatting for Real Time Radiance Field Rendering has proposed a solution.

Additionally, on July 10th, 2024, NVIDIA Research unveiled 3D Gaussian Ray Tracing, as a departure from Splatting. As the name implies, it utilizes ray tracing rather than the rasterization that has traditionally been used with Gaussian Splatting.

There are several exciting advancements and functionalities that become readily available. It addresses one of the restrictions of Gaussian Splatting and allows for the reintroduction of fisheye lenses to captures. Additionally, secondary lighting effects such as refractions, shadows, depth of field, and mirrors.

This is a brand new line of research and it appears likely that it will attract appropriate research attention to push boundaries forwards.