Platforms|Research

Language Embedded Radiance Field (LERF) Announced

Michael Rubloff

Mar 17, 2023

LERF
LERF

With GTC and GDC in the immediate future, we have seen an increase of consequential NeRF papers get published. Language Embedded Radiance Fields or LERF, represent a new emergence of the technology. LERF allows for computers to understand in a given NeRF what the individual items contained in a scene are from a text prompt.

LERF Teaser Video

This unlocks a tremendous amounts of use cases ranging from assembly instructions, to building inspections, to fire escape plans. For me, I am reliant on watching YouTube tutorials on how to assemble furniture and pretty much anything that doesn't come pre-assembled. Now it will be possible for retailers such as IKEA to train these models to what piece corresponds to each step of the assembly process and allow the end user to query where each piece goes. For instance, what are the all pieces I will be using for steps 1-3 of assembly?

Additionally, it allows for asynchronous customer support and troubleshooting.

Furthermore, these retailers will also have the ability to generate NeRFs of what the completed product is supposed to look like in 3D and allow users to make sure they followed the instructions correctly - something that I have continued to struggle with.

Given that the paper authors, Matthew Tancik and Angjoo Kanazawa, are also founders of NeRFstudio, it's not a surprise that they announced an integration within NeRFstudio is coming. This integration capability is what transforms the technology into a realistic use case for short term impact on businesses.

This is further inline with some of the recent developments as more and more features begin to be compatible with NeRFstudio. The ecosystem that is developing around NeRF is extremely exciting and one to closely watch as the year continues.

This advance also gives us a glimpse into the future where it will be possible to utilize a text prompt to delete or transform an object. Perhaps it will also build upon NeRFshop to make some of these changes to a scene in a more efficient manner, similar to how content aware has emerged in Photoshop. It also represents the possibility of providing a command to a robot to fulfill. Immediately my mind goes towards how distribution such as Amazon would be able to utilize NeRFs with its fulfillment network to more efficiently and accurately transport items around a warehouse to be fulfilled.

LERF enables pixel-aligned queries of the distilled 3D CLIP embeddings without relying on region proposals, masks, or fine-tuning, supporting long-tail open-vocabulary queries hierarchically across the volume.

LERF Paper

In the paper, it is demonstrated how it is possible to now pair algorithms that work with natural language to interact with the scene.

Once the items that ChatGPT has recommended to clean up the area has been defined, it is easy to establish a workflow to complete the goal, all while being confirmed that the resources exist in the live scenario.

However, there are other examples included on the product page to help show how LERF can help transform businesses. For instance, a florist can demonstrate to a client how a bouquet will look and show live examples of different types of flowers will pair.

The amount of detail that is able to be drilled down is surprising as demonstrated by the above video; not only is LERF able to understand the larger bouquet of flowers, but it further able to isolate the types of flowers contained within it. As this technology progresses, the documentation of everyday items for educational purposes will rapidly increase. For instance, I would have been hopeless attempting to figure out what rosemary was, but now it's just about asking what flowers are contained in this bouquet. This tool will immediately be a boon for so many industries and it's easy to imagine how florists can use it to help upsell variety of flowers to clients.

Given that the paper authors, Matthew Tancik and Angjoo Kanazawa, are also founders of NeRFstudio, it's not a surprise that they announced an integration within NeRFstudio is coming. This integration capability is what transforms the technology into a realistic use case for short term impact on businesses.

This is further inline with some of the recent developments as more and more features begin to be compatible with NeRFstudio. The ecosystem that is developing around NeRF is extremely exciting and one to closely watch as the year continues.

Professor Angjoo Kanazawa will also be giving a presentation on NeRFstudio next week at NVIDIA's GTC. Check out the article here with the NeRF schedule and make sure to add it to your list!

Featured

Featured

Featured

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Featured

Featured

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Recent articles

Recent articles