Research

Stylized Radiance Fields with Geometry Transfer (Geo-SRF)

Michael Rubloff

Michael Rubloff

Feb 5, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Geometry Transfer for Stylizing Radiance Fields
Geometry Transfer for Stylizing Radiance Fields


Diving into a world where reality can take on the aesthetics of imagination, Geometry Transfer for stylizing radiance fields (Geo-SRF) showcases a glimpse, offering artists and technologists alike the whimsical power to reshape the very fabric of 3D spaces.



Authored by Seoul National University and Meta Reality Labs, this work introduces a novel method, Geo-SRF, that employs depth maps as a style guide, enabling the stylization of both the scene's appearance and shape in a coherent manner to better reflect the intended style.





There have been amazing companies such as Kaiber AI, EbSynth, and RunwayML that have created the most fantastical outputs. However, those are not editing the actual underlying three dimensional space. The significance of this research lies in its departure from existing 3D style transfer methods that primarily focus on transferring aesthetic qualities such as colors, textures, and brushstrokes, while largely neglecting the geometric aspects.



By incorporating geometric deformation into the style transfer process, the authors propose a method that significantly enhances the expressiveness and accuracy of 3D stylizations, offering a more comprehensive approach to capturing the stylistic essence of a reference style. Style Transfer is very simply put by the authors as "rendering one image in the stylistic manner of another, producing a new image that combines the foundational structure of the former with the aesthetic qualities of the latter."



Geometry Transfer is introduced as a novel method that leverages geometric deformation to stylize not just the color and texture but also the shape of 3D scenes. This approach uses depth maps to extract geometric styles from a style guide, which are then applied to the geometry of radiance fields. Instead of solely relying on RGB images, Geometry Transfer uses depth maps as style guides to capture the geometric style. Depth maps provide detailed information about the shape and structure of the style guide, which is crucial for geometric stylization.



https://www.youtube.com/watch?v=QpFQjE9sDg8



The technique uses deformation fields to predict the offset vector for each 3D point, allowing for the accurate stylization of both shape and appearance. This ensures that as the shape of the scene is modified to match the style guide, the appearance (colors and textures) is also appropriately adjusted to align with the new geometry.



Geometry Transfer further enhances the stylization process by using an RGB-D pair (an RGB image and a depth map) as the style guide. This approach allows for a more expressive stylization that accurately reflects the given style in terms of both shape and appearance. The method introduces geometry-aware matching to enhance the diversity of stylization while preserving local geometry. Patch-wise optimization is utilized to effectively transfer the style of geometry, considering the spatial relationships within the scene.



To bring a richer sense of scene depth, Geometry Transfer employs perspective style augmentation, varying the sizes of patterns applied to surfaces based on their distance from the viewer. This technique enhances the perception of depth in the stylized scene, making the stylization more immersive and realistic.



However, this isn't an all or nothing approach. They're also able to create partial stylization of a scene through Panoptic Lifting. This flexibility allows for selective stylization of scene elements, further expanding the creative possibilities of 3D style transfer. I would also be curious how this might work with something like GARField.






The results of extensive experiments demonstrate the ability of Geometry Transfer to achieve a broader and more expressive range of stylizations compared to existing methods, thereby expanding the potential applications of 3D style transfer in content creation for virtual and augmented reality. However, it goes even further than that! Like a dimmer on a light switch, you're also able to interpolate between the strength of the original scene and the new style. This can allow for transition effects to fade into the scene or showcase some kind of evolution within your subject.





There are so many fun and cool applications that can be done with this. I can't imagine this being overly difficult to incorporating into existing platforms and allow people to explore creating fantastical radiance fields.



This research not only opens new avenues for artistic expression within the digital domain but also sets a new benchmark for future explorations in the intersection of art and technology. The integration of geometric considerations into the style transfer process represents a significant advancement in the field, promising to inspire further innovations and applications in the creation and stylization of 3D content.



The code has not yet been published, but we'll be keeping an eye on their Github page.


Featured

Featured

Featured

Platforms

Varjo Unveils Teleport

Finnish VR company Varjo has announced 3DGS platform, Teleport.

Michael Rubloff

Jun 18, 2024

Platforms

Varjo Unveils Teleport

Finnish VR company Varjo has announced 3DGS platform, Teleport.

Michael Rubloff

Jun 18, 2024

Platforms

Varjo Unveils Teleport

Finnish VR company Varjo has announced 3DGS platform, Teleport.

Michael Rubloff

Platforms

Wayve Announces PRISM-1

Wayve, has announced the launch of PRISM-1, an innovative 4D reconstruction model designed to significantly enhance the testing and training of advanced driver assistance systems and autonomous driving technology.

Michael Rubloff

Jun 17, 2024

Platforms

Wayve Announces PRISM-1

Wayve, has announced the launch of PRISM-1, an innovative 4D reconstruction model designed to significantly enhance the testing and training of advanced driver assistance systems and autonomous driving technology.

Michael Rubloff

Jun 17, 2024

Platforms

Wayve Announces PRISM-1

Wayve, has announced the launch of PRISM-1, an innovative 4D reconstruction model designed to significantly enhance the testing and training of advanced driver assistance systems and autonomous driving technology.

Michael Rubloff

Platforms

Nerfstudio Gradio UI Added

Nerfstudio now includes a Gradio UI, making it easy and intuitive to set commands and run scripts.

Michael Rubloff

Jun 14, 2024

Platforms

Nerfstudio Gradio UI Added

Nerfstudio now includes a Gradio UI, making it easy and intuitive to set commands and run scripts.

Michael Rubloff

Jun 14, 2024

Platforms

Nerfstudio Gradio UI Added

Nerfstudio now includes a Gradio UI, making it easy and intuitive to set commands and run scripts.

Michael Rubloff

Platforms

Graswald.ai: 3DGS and Ecommerce Platform

Graswald is a 3D product visualization platform using Gaussian Splatting, aimed primarily at the Ecommerce industry to offer hyper realistic detail at a fraction of current time and cost. 

Michael Rubloff

Jun 13, 2024

Platforms

Graswald.ai: 3DGS and Ecommerce Platform

Graswald is a 3D product visualization platform using Gaussian Splatting, aimed primarily at the Ecommerce industry to offer hyper realistic detail at a fraction of current time and cost. 

Michael Rubloff

Jun 13, 2024

Platforms

Graswald.ai: 3DGS and Ecommerce Platform

Graswald is a 3D product visualization platform using Gaussian Splatting, aimed primarily at the Ecommerce industry to offer hyper realistic detail at a fraction of current time and cost. 

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Featured

Featured

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Nerfstudio

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Nerfstudio

Platforms

Nerfstudio Releases gsplat 1.0

Michael Rubloff

Jun 7, 2024

Nerfstudio

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio