Research

Nuvo: Revolutionizing UV Mapping with Google’s Latest Neural Field-Based Method

Michael Rubloff

Dec 13, 2023

Nuvo
Nuvo

Google Research has been on fire lately. They just recently came out with Reconfusion and now joining a very busy year for the Google team is Nuvo.

Nuvo includes three of the original authors of NeRF on it (Jon Barron, Ben Mildenhall, and Pratul Srinivasan). This paper is of particular note, as it might be the last one to feature Ben Mildenhall as a Google employee. His last day was this past Friday at the tech giant. From my understanding, a new superstar has also joined the Google team.

But enough about personnel; this paper is awesome. Historically, many meshes had been carefully modeled and designed by professionals to be clean and easy to work with. However, NeRFs very quickly reconstruct real life, where you get real world outputs, which are not clean. That doesn't even mention generative, which gives us outputs that are only limited by our minds. NeRFs further represent geometry as volumetric fields instead of meshes.

The existing UV mapping methods, just were not meant to be used with these specific technological advancements, which is fair, but still leaves a problem. This is where Nuvo begins to come in.

It very quickly can become apparent the amount of creative flexibility for the manipulation and addition of assets into a space, while still holding the scene's original shape.

Unlike conventional methods that depend on the vertices of a mesh, Nuvo uses a neural field to represent a continuous UV mapping. This neural field is optimized specifically for the visible points of the scene, providing a robust solution to the challenges posed by complex geometries. In order to function Nuvo employs multiple MLPs for different aspects of UV mapping:

A "chart assignment" MLP assigns points on the surface to various charts. This MLP maps a 3D point to a probability mass function over the charts. The number of charts is an arbitrary number that is chosen by the user. It assesses the 3D landscape and probabilistically assigns each point to a chart, ensuring that the mapping process begins correctly.

Then Nuvo begins to implement a set of "texture coordinate" MLPs, one for each chart, mapping 3D points to corresponding 2D UV texture coordinates. These MLPs take the 3D points and transform them into the 2D realm of UV texture coordinates. They act as translators, converting the complex language of 3D geometry into the understandable dialect of 2D textures. Complementing this process are the Surface Coordinate MLPs. These function as the reverse translators, taking 2D UV coordinates back to the 3D world, ensuring that the mapping accurately reflects the original geometry.

Nuvo's journey through the optimization process is akin to a quest for perfection. It's not just about creating a UV map, but crafting one that is bijective, precise, with the 3D model. Nuvo's optimization involves minimizing a set of loss functions that collectively encourage a well-behaved and effective UV mapping.

The first challenge in this quest is achieving bijectivity. Nuvo embarks on a meticulous path, ensuring that each 3D point uniquely corresponds to a 2D point and vice versa. This bijective mapping is crucial – it ensures that the texture adheres perfectly to the model, with each point on the model uniquely mapped to the texture space. This is achieved through two cycle consistency losses: L3→2→3Minimizes the squared distance for 3D points after mapping to 2D and back. L2→3→2 Focuses on ensuring that the entire texture space is utilized effectively by the mapping.

As Nuvo navigates through this process, it encounters the need for precise chart assignment. Here, the Chart Assignment MLP plays a pivotal role, confidently partitioning the 3D scene into distinct charts. The optimization process fine-tunes this MLP, ensuring that its decisions are not just accurate but also definitive. Essentially it's saying what goes where.

In parallel, the surface coordinate accuracy is refined. Nuvo ensures that the MLPs mapping the 2D UV coordinates back to 3D points are in perfect alignment with the model's surface.

Lastly, Nuvo addresses the challenge of minimizing distortion. This part of the journey focuses on preserving the natural look of the textures when mapped onto the model. By regularizing the mappings to maintain angles and uniform stretching, Nuvo ensures that the textures appear realistic and unwarped, maintaining the integrity of the original capture.

I strongly hope that this code is released, so that people can begin building it into existing methods. I believe that it offers the ability to dramatically lower the barrier to editable NeRFs and generative models. It truly is an exciting paper!

All of this can be extremely technical a bit difficult to understand the implications of what is being presented, but this paper is significant. This will make it much easier to begin editing these 3D reconstructions. I have to imagine that the team over at Luma saw this paper and immediately began to think through how Genie can leverage Nuvo. There are so many possibilities!

By employing neural fields and optimizing for visible points, Nuvo overcomes the complexities of modern 3D geometries, crafting UV maps that are functional with the intricate details of NeRF based models.

Featured

Featured

Featured

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

4/11/24

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

4/10/24

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

4/9/24

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

4/11/24

Trending articles

Trending articles

Trending articles

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Recent articles

Recent articles

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024