Trending

Trending

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

RadSplat

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

RadSplat

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

RadSplat

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

About RADIANCE FIELDS

About RADIANCE FIELDS

What are Radiance Fields

Radiance Fields, are an emerging state-of-the-art solution to the problems of inverse rendering and novel view synthesis.

In these problems, the goal is to take a set of images of a subject from multiple angles and generate the most realistic 3D representation possible using a neural network. This allows you to look at the scene or object from any arbitrary angle. In addition, they also model view-dependent lighting effects, which means NeRFs can capture details like reflections that change depending on your viewing angle.

"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."

In conclusion, radiance fields are for computer graphic simplification, in order to simplify the complex electromagnetic field (physical principles) into the radiance fields, which discards details of reflection, refraction, transmission et. al. So, in this case, we could easily apply neural rendering or volume rendering (e.g. NeRF), such simple rendering methods to render the 3D representation.

In these problems, the goal is to take a set of images of a subject from multiple angles and generate the most realistic 3D representation possible using a neural network. This allows you to look at the scene or object from any arbitrary angle. In addition, they also model view-dependent lighting effects, which means Radiance Fields can capture details like reflections that change depending on your viewing angle.

They additionally enable these view dependent effects from brand new viewpoints, also called novel view synthesis.

In conclusion, radiance fields are for computer graphic simplification, in order to simplify the complex electromagnetic field (physical principles) into the radiance fields, which discards details of reflection, refraction, transmission et. al. So, in this case, we could easily apply neural rendering or volume rendering (e.g. NeRF), such simple rendering methods to render the 3D representation.

In addition to being able to model hyper realistic static content, the promise of radiance fields also extends towards dynamic or moving content. This means that we will be able to experience not only the photography equivalent of 3D life, but videography as well.

Most of the radiance field methods utilize the same initial intake process, whereby standard 2D images are run through a alignment process named Structure from Motion (SfM) and then trains the resulting data. Depending on the radiance field method, the training implementation can be vastly different. The two most common radiance field methods to date are Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS).

Who created Radiance Fields?

Introduced by Mildenhall et al. from Berkeley University, they unveiled Neural Radiance Fields in early 2020. The first author of NeRF, Ben Mildenhall recently recounted the first couple of weeks and initial tests at his keynote address at 3DV.

The NeRF representation creates a model in the form of a continuous volumetric scene function. This function, when queried with a ray casting from a particular viewing direction, returns the color and opacity values that, when combined, generate images that offer new perspectives of static scenes. These generated images are remarkably accurate and provide fine details, even from viewpoints vastly different from the input views.

Now that there are multiple radiance field methods, we can only speculate what the future holds for them. The progression rate has been astounding to follow along within computer vision and I believe its use cases will extend far beyond the industry. You can follow along the progression on the website or through our discord server!

The Progress of Radiance Fields

Radiance Fields have been skyrocketing in popularity over the last year. It seems like there are daily 10+ papers that are being released and pushing boundaries. Additionally, the rate that people are solving problems and lowering barriers to entry also continues to fall. Whether that is through more startups building in the space or weaknesses of the methods being addressed, it seems like there is something exciting happening daily.

Moreover, there are quite a few companies that are building in stealth mode, actively building on top of the existing technology across such a wide variety of industries. We, as consumers will directly benefit from the creation of these companies and will push us forwards as a society to reap the benefits of a lifelike imaging medium.

This publication exists to help highlight the spectacular work that is being done across the world with Radiance Fields, and provoke the imagination of readers to embrace what once seemed impossible, but now is an ever shortening period of time.

Radiance Fields have been skyrocketing in popularity over the last few years.

It seems like there are daily 10+ papers that are being released and pushing boundaries. Additionally, the rate that people are solving problems and lowering barriers to entry also continues to fall. Whether that is through more startups building in the space or weaknesses of the methods being addressed, it seems like there is something exciting happening daily.

Moreover, there are quite a few companies that are building in stealth mode, actively building on top of the existing technology across such a wide variety of industries. We, as consumers will directly benefit from the creation of these companies and will push us forwards as a society to reap the benefits of a lifelike imaging medium.

This publication exists to help highlight the spectacular work that is being done across the world with Radiance Fields, and provoke the imagination of readers to embrace what once seemed impossible, but now is an ever shortening period of time.

What's Next for Radiance Fields?

The ceilings of radiance fields continues to be unknown. Additionally, it is likely that more forms of radiance fields emerge. Already in the beginning of 2024, we saw the introduction of a third type of radiance field, Trilinear Point Splatting for Real Time Radiance Field Rendering.

Additionally, we only recently saw the first NeRF/3DGS hybrid paper with RadSplat emerge from Google. This took the best of both worlds from each method and combines them into a unified power. RadSplat results in nearly 900 fps, with the fidelity that you expect from NeRFs.

From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, Radiance Fields stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.

Radiance Fields promise a time not too far off where we will no longer be using photography and videos as the dominant imaging medium and be able to routinely and with minimal effort be able to document our lives, business, and society in a hyper realistic way, similar to how we experience everyday life.

About Neural RADIANCE FIELDS

About Neural RADIANCE FIELDS

What are
Neural Radiance Fields

Neural Radiance Fields, abbreviated as NeRFs, are an emerging state-of-the-art solution to the problems of inverse rendering and novel view synthesis.

In these problems, the goal is to take a set of images of a subject from multiple angles and generate the most realistic 3D representation possible using a neural network. This allows you to look at the scene or object from any arbitrary angle. In addition, they also model view-dependent lighting effects, which means NeRFs can capture details like reflections that change depending on your viewing angle.

"Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x y z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location."

"We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis."

In other words, a deep neural network learns from a sparse set of input images, and the network's ability to predict radiance values for every point in the 3D space is honed over time. Ray casting, a technique in which rays are sent from the camera's position into the scene to calculate radiance values, plays a crucial role here. Rendering loss, an important metric in this method, is optimized to ensure that the rendered outputs closely resemble the training images. The continuous nature of the neural radiance field means it doesn’t rely on traditional voxel grids or meshes, contrasting with older methods of representing scenes.

Despite the complexity and intricacies of how neural radiance fields work, the technology's basic steps are straightforward. First, acquire data, usually photographs from different angles using standard cameras or even photogrammetry software. Next, feed this data into deep learning algorithms where the system is trained. Once trained, the NeRF can then synthesize or render new views of the scene, providing a new perspective previously unimagined.

Who created Neural Radiance Fields?

Introduced by Mildenhall et al. from Berkeley University, the NeRF representation creates a model in the form of a continuous volumetric scene function. This function, when queried with a ray casting from a particular viewing direction, returns the color and opacity values that, when combined, generate images that offer new perspectives of static scenes. These generated images are remarkably accurate and provide fine details, even from viewpoints vastly different from the input views.

Neural Radiance Fields were originally proposed and the full author list includes: Matthew Tancik, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng in 2020. Since then, a number of breakthroughs have pushed this field of research to the cutting edge. One such advancement was Instant-NGP, the project released alongside the research paper by Mueller, et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. Prior to the ingenious introduction of the multi-resolution hash encoding, it would take hours or even days to produce a high-quality NeRF. Now, the training process takes less than 15 minutes to produce photorealistic details. Instant-NGP went on to win one of top inventions of 2022 from Time Magazine.

The Progress of Neural Radiance Fields

NeRFs are almost something out of science fiction. They build upon the technology of light fields using concepts from artificial intelligence, machine learning, and neural networks. They mark a stark divergence from conventional triangle-mesh-based ray tracing. The key difference between NeRF and traditional photo-scanning is the potential for highly accurate reconstructions that look realistic to the human eye. Like the name implies, NeRFs are able to achieve such quality through the clever use of a type of neural network called an MLP (Multi-Layer Perceptron). By training the MLP, NeRFs are able to approximate the shape and color of reality through a process called differentiable rendering. It’s amazing, but NeRFs are also not limited to just one object, and can generate novel views of complex scenes and produce amazing results. We are still in the early days of NeRF and generative AI, but the amount of progress made so far has been staggering. It seems like every day, there’s a new NeRF paper capable of pushing computer graphics to the next level. Start ups such as Luma AI have greatly reduced the barrier to entry, allowing people with just an iPhone to capture incredible results. Now anyone can make a NeRF, so go grab your digital cameras and start nerfing!

This publication exists to help highlight the spectacular work that is being done across the world with NeRFs, and provoke the imagination of readers to embrace what once seemed impossible.

What's Next for NeRFs?

While NeRFs offer amazing performance in rendering static scenes, their ability to handle dynamic or moving objects in real-world scenarios is still an area for future exploration. The concept of viewing direction is paramount in NeRFs, as the representation needs to accurately capture how light, from different directions, interacts with the scene geometry, be it buildings, a person, or even a simple table. The reflection, shading, and bounce of light off objects and materials are vital to the synthesis process.This article just touches on the tip of the iceberg when it comes to understanding NeRFs. The depth and breadth of this topic are immense, and every section, from ray casting to deep learning, is a deep dive into the exciting world of neural rendering.

From its beginnings, introduced by pioneers like Mildenhall et al., to its present applications and future potential, NeRFs stand as a testament to the ever-evolving nature of technology. Importantly, as new algorithms and optimization methods are developed, we can only look ahead with anticipation at the myriad of possibilities that NeRF technology presents.

About 3D Gaussian Splatting

About 3D Gaussian Splatting

What is 3D Gaussian
Splatting?

3D Gaussian Splatting is a radiance field reconstruction method that is rasterization-based rather than using a neural network like in Neural Radiance Fields (NeRFs).

This allows for it to maintain the high visual fidelity and view dependent effects that NeRFs are known for, but also allows for real time rendering rates and performance, including on mobile devices!

There has been tremendous excitement about another radiance field method, 3D Gaussian Splatting for Real Time Radiance Field Rendering. As its name implies, 3DGS is able provide high quality results while still rendering in real time.

Instead of relying on a dense neural network to model the radiance field, 3DGS uses 3D Gaussians to represent the scene. A Gaussian in this context is essentially a smooth, bell-shaped curve that can vary in width, height, and orientation, offering a flexible way to model the density and appearance of different parts of the scene.

Each 3D Gaussian models a volumetric "splat" in the scene, with properties like position, size, orientation, and color. This representation is both compact and expressive, allowing for detailed scenes to be modeled with fewer parameters than a dense voxel grid or a neural network would require.

To render a new view of the scene, the positions and properties of the 3D Gaussians are projected onto the 2D image plane of the camera viewpoint. This projection translates the 3D Gaussians into 2D "splats" on the image, where their contributions are blended together based on their properties and distances to the camera.

The rendering algorithm accounts for the visibility and orientation of each Gaussian, ensuring that only the visible parts contribute to the final image. This includes handling occlusions and leveraging anisotropic (directionally varying) properties to accurately render surfaces and edges.

By optimizing the properties of the 3D Gaussians directly, the method achieves fast training times. The renderer, designed for efficiency, leverages modern GPU architectures to achieve real-time rendering speeds while maintaining high visual quality.

Similar to NeRFs, the input data used are multiple 2D photos from various camera perspectives. Gaussian Splatting is able to create a highly accurate representation of a 3D space. Both NeRFs and 3D Gaussian Splatting use structure from motion SfM to begin aligning the images, but Gaussian Splatting generates a sparse point cloud.

Additionally, because of 3DGS’s setup, there have also been integrations with some web based platforms, such as Three.js, React Three Fiber, and 3D Design tool, Spline. These tools help make it possible to bring Gaussian Splatting into no code web design platforms such as Framer and Webflow, making it easy to show off your creations.

Gaussian Splatting
Gaussian Splatting
Gaussian Splatting
Gaussian Splatting

Who created 3D Gaussian Splatting?

3DGS emerges from the seminal paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.

3D Gaussian Splatting or 3DGS has quickly spread across the world and onto a variety of platforms, including Unreal Engine, Unity, Nerfstudio, Polycam, Luma AI, Volinga, Kiri Engine, and more. Basically, if a company offers a radiance field solution, they are also offering an implementation of Gaussian Splatting at this point.

3DGS was released in late April of 2023 and quickly became inordinately popular, winning best paper at SIGGRAPH 2023 in August.

The Progress of 3D Gaussian Splatting

Gaussian Splatting has had a meteoric rise since it was released less than a year ago. The pure volume of papers building and exploring in the space has been staggering, with large progress being made.

Additionally, we have seen several companies, building both publicly and in stealth mode begin utilizing 3D Gaussian Splatting.

There has also been a surge of using 3D Gaussian Splatting in both text or image to 3D Generative AI models, despite them only being created recently.

It's explicit representation makes it easy to work with and thus we have seen people take advantage of it.

People have also been creating experiential and educational content from Gaussian Splatting and using the host of viewers and distributors, such as Spline, PlayCanvas's Super Splat, and Antimatter15's web viewer.

What's Next for Gaussian Splatting?

With all the advancements in such a short period of time, it’s crazy to think where 3D Gaussian Splatting might be headed. I expect to see more implementations across various industries that are looking to utilize lifelike 3D content in their offerings,

On the research side, Generative AI using Gaussians have remained popular and I believe that we will see faster, more complex, and more hyper realistic outputs. Recently, we have begun to see work to pull high quality meshes out from Gaussian Splatting outputs, such as SuGaR, its follow up paper Gaussian Frosting, and Gaustudio.

There is currently no publicly known ceiling on the technology and will surely be one of the most exciting topics to follow along with over the coming years.

In its current state, 3D Gaussian Splatting struggles a bit with fine details, but another real time radiance field method, Trilinear Point Splatting for Real Time Radiance Field Rendering has proposed a solution.