Google Research has been on fire lately. They just recently came out with Reconfusion and now joining a very busy year for the Google team is Nuvo.
Nuvo includes three of the original authors of NeRF on it (Jon Barron, Ben Mildenhall, and Pratul Srinivasan). This paper is of particular note, as it might be the last one to feature Ben Mildenhall as a Google employee. His last day was this past Friday at the tech giant. From my understanding, a new superstar has also joined the Google team.
But enough about personnel; this paper is awesome. Historically, many meshes had been carefully modeled and designed by professionals to be clean and easy to work with. However, NeRFs very quickly reconstruct real life, where you get real world outputs, which are not clean. That doesn't even mention generative, which gives us outputs that are only limited by our minds. NeRFs further represent geometry as volumetric fields instead of meshes.
The existing UV mapping methods, just were not meant to be used with these specific technological advancements, which is fair, but still leaves a problem. This is where Nuvo begins to come in.
It very quickly can become apparent the amount of creative flexibility for the manipulation and addition of assets into a space, while still holding the scene's original shape.
Unlike conventional methods that depend on the vertices of a mesh, Nuvo uses a neural field to represent a continuous UV mapping. This neural field is optimized specifically for the visible points of the scene, providing a robust solution to the challenges posed by complex geometries. In order to function Nuvo employs multiple MLPs for different aspects of UV mapping:
A "chart assignment" MLP assigns points on the surface to various charts. This MLP maps a 3D point to a probability mass function over the charts. The number of charts is an arbitrary number that is chosen by the user. It assesses the 3D landscape and probabilistically assigns each point to a chart, ensuring that the mapping process begins correctly.
Then Nuvo begins to implement a set of "texture coordinate" MLPs, one for each chart, mapping 3D points to corresponding 2D UV texture coordinates. These MLPs take the 3D points and transform them into the 2D realm of UV texture coordinates. They act as translators, converting the complex language of 3D geometry into the understandable dialect of 2D textures. Complementing this process are the Surface Coordinate MLPs. These function as the reverse translators, taking 2D UV coordinates back to the 3D world, ensuring that the mapping accurately reflects the original geometry.
Nuvo's journey through the optimization process is akin to a quest for perfection. It's not just about creating a UV map, but crafting one that is bijective, precise, with the 3D model. Nuvo's optimization involves minimizing a set of loss functions that collectively encourage a well-behaved and effective UV mapping.
The first challenge in this quest is achieving bijectivity. Nuvo embarks on a meticulous path, ensuring that each 3D point uniquely corresponds to a 2D point and vice versa. This bijective mapping is crucial – it ensures that the texture adheres perfectly to the model, with each point on the model uniquely mapped to the texture space. This is achieved through two cycle consistency losses: L3→2→3Minimizes the squared distance for 3D points after mapping to 2D and back. L2→3→2 Focuses on ensuring that the entire texture space is utilized effectively by the mapping.
As Nuvo navigates through this process, it encounters the need for precise chart assignment. Here, the Chart Assignment MLP plays a pivotal role, confidently partitioning the 3D scene into distinct charts. The optimization process fine-tunes this MLP, ensuring that its decisions are not just accurate but also definitive. Essentially it's saying what goes where.
In parallel, the surface coordinate accuracy is refined. Nuvo ensures that the MLPs mapping the 2D UV coordinates back to 3D points are in perfect alignment with the model's surface.
Lastly, Nuvo addresses the challenge of minimizing distortion. This part of the journey focuses on preserving the natural look of the textures when mapped onto the model. By regularizing the mappings to maintain angles and uniform stretching, Nuvo ensures that the textures appear realistic and unwarped, maintaining the integrity of the original capture.
I strongly hope that this code is released, so that people can begin building it into existing methods. I believe that it offers the ability to dramatically lower the barrier to editable NeRFs and generative models. It truly is an exciting paper!
All of this can be extremely technical a bit difficult to understand the implications of what is being presented, but this paper is significant. This will make it much easier to begin editing these 3D reconstructions. I have to imagine that the team over at Luma saw this paper and immediately began to think through how Genie can leverage Nuvo. There are so many possibilities!
By employing neural fields and optimizing for visible points, Nuvo overcomes the complexities of modern 3D geometries, crafting UV maps that are functional with the intricate details of NeRF based models.
