Research

Nuvo: Revolutionizing UV Mapping with Google’s Latest Neural Field-Based Method

Michael Rubloff

Michael Rubloff

Dec 13, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Nuvo
Nuvo

Google Research has been on fire lately. They just recently came out with Reconfusion and now joining a very busy year for the Google team is Nuvo.

Nuvo includes three of the original authors of NeRF on it (Jon Barron, Ben Mildenhall, and Pratul Srinivasan). This paper is of particular note, as it might be the last one to feature Ben Mildenhall as a Google employee. His last day was this past Friday at the tech giant. From my understanding, a new superstar has also joined the Google team.

But enough about personnel; this paper is awesome. Historically, many meshes had been carefully modeled and designed by professionals to be clean and easy to work with. However, NeRFs very quickly reconstruct real life, where you get real world outputs, which are not clean. That doesn't even mention generative, which gives us outputs that are only limited by our minds. NeRFs further represent geometry as volumetric fields instead of meshes.

The existing UV mapping methods, just were not meant to be used with these specific technological advancements, which is fair, but still leaves a problem. This is where Nuvo begins to come in.

It very quickly can become apparent the amount of creative flexibility for the manipulation and addition of assets into a space, while still holding the scene's original shape.

Unlike conventional methods that depend on the vertices of a mesh, Nuvo uses a neural field to represent a continuous UV mapping. This neural field is optimized specifically for the visible points of the scene, providing a robust solution to the challenges posed by complex geometries. In order to function Nuvo employs multiple MLPs for different aspects of UV mapping:

A "chart assignment" MLP assigns points on the surface to various charts. This MLP maps a 3D point to a probability mass function over the charts. The number of charts is an arbitrary number that is chosen by the user. It assesses the 3D landscape and probabilistically assigns each point to a chart, ensuring that the mapping process begins correctly.

Then Nuvo begins to implement a set of "texture coordinate" MLPs, one for each chart, mapping 3D points to corresponding 2D UV texture coordinates. These MLPs take the 3D points and transform them into the 2D realm of UV texture coordinates. They act as translators, converting the complex language of 3D geometry into the understandable dialect of 2D textures. Complementing this process are the Surface Coordinate MLPs. These function as the reverse translators, taking 2D UV coordinates back to the 3D world, ensuring that the mapping accurately reflects the original geometry.

Nuvo's journey through the optimization process is akin to a quest for perfection. It's not just about creating a UV map, but crafting one that is bijective, precise, with the 3D model. Nuvo's optimization involves minimizing a set of loss functions that collectively encourage a well-behaved and effective UV mapping.

The first challenge in this quest is achieving bijectivity. Nuvo embarks on a meticulous path, ensuring that each 3D point uniquely corresponds to a 2D point and vice versa. This bijective mapping is crucial – it ensures that the texture adheres perfectly to the model, with each point on the model uniquely mapped to the texture space. This is achieved through two cycle consistency losses: L3→2→3Minimizes the squared distance for 3D points after mapping to 2D and back. L2→3→2 Focuses on ensuring that the entire texture space is utilized effectively by the mapping.

As Nuvo navigates through this process, it encounters the need for precise chart assignment. Here, the Chart Assignment MLP plays a pivotal role, confidently partitioning the 3D scene into distinct charts. The optimization process fine-tunes this MLP, ensuring that its decisions are not just accurate but also definitive. Essentially it's saying what goes where.

In parallel, the surface coordinate accuracy is refined. Nuvo ensures that the MLPs mapping the 2D UV coordinates back to 3D points are in perfect alignment with the model's surface.

Lastly, Nuvo addresses the challenge of minimizing distortion. This part of the journey focuses on preserving the natural look of the textures when mapped onto the model. By regularizing the mappings to maintain angles and uniform stretching, Nuvo ensures that the textures appear realistic and unwarped, maintaining the integrity of the original capture.

I strongly hope that this code is released, so that people can begin building it into existing methods. I believe that it offers the ability to dramatically lower the barrier to editable NeRFs and generative models. It truly is an exciting paper!

All of this can be extremely technical a bit difficult to understand the implications of what is being presented, but this paper is significant. This will make it much easier to begin editing these 3D reconstructions. I have to imagine that the team over at Luma saw this paper and immediately began to think through how Genie can leverage Nuvo. There are so many possibilities!

By employing neural fields and optimizing for visible points, Nuvo overcomes the complexities of modern 3D geometries, crafting UV maps that are functional with the intricate details of NeRF based models.

Featured

Featured

Featured

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV