TripoAI partners with Stability AI for TripoSR Release

Michael Rubloff

Michael Rubloff

Mar 5, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
TripoSR
TripoSR

Tripo AI, the company behind Triplane Gaussians has announced a partnership with Stability AI on the release of their newest state of the art image to 3D model, TripoSR.

You could already generate decently high fidelity 3D objects using Triplane Gaussians, but they've now stepped it up a notch. Built on inspiration from LRM: Large Reconstruction Model for Single Image to 3D, which uses NeRFs as part of their backbone, they're able to generate a 3D object within .5 seconds on a NVIDIA A100. However, it's not necessary to use such a powerful GPU to power TripoSR.

The initial phase involves an image encoder that employs a pre-trained vision transformer model to convert an image into latent vectors. These vectors encapsulate the critical global and local features of the image necessary for the reconstruction process.

TripoSR's transformation from a two-dimensional image to a three-dimensional model is facilitated by its image-to-triplane decoder. This innovative decoder converts the latent vectors into a triplane-NeRF representation, a method capable of capturing complex object shapes and textures. The decoder comprises transformer layers with self-attention and cross-attention mechanisms, enabling it to interpret the relationships within the triplane representation and integrate the image features encoded by the latent vectors effectively.

The triplane-based neural radiance field (NeRF) is where TripoSR’s process culminates, tasked with determining the color and density of points within the 3D space. Through this, the abstract triplane representation is materialized into the 3D mesh. This component utilizes multilayer perceptrons to infuse the final model with realistic textures and colors, closely mirroring the physical objects.

Notably, it adopts an unconventional approach to camera parameter estimation, enhancing the model's flexibility and adaptability to various imaging scenarios. The model also introduces triplane channel optimization and a unique mask loss function during training. These advancements serve to refine the reconstruction quality, optimizing the balance between computational demand and the accuracy of the generated models.

TripoSR already has made a Hugging Face demo already available for people to try out for themselves and yes, it's very fast.

With the model, source code, and an interactive demo readily available under the MIT license, TripoSR is poised to empower a wide range of users—from researchers and developers to creatives—eager to explore the latest advancements in 3D generative AI. We'll have to see if it ends up making an appearance in Comfy3D.

Featured

Recents

Featured

Platforms

Gracia Launches Dynamic Gaussian Splatting

Gracia launches dynamic captures of Radiance Fields, using Gaussian Splatting.

Michael Rubloff

Oct 17, 2024

Platforms

Gracia Launches Dynamic Gaussian Splatting

Gracia launches dynamic captures of Radiance Fields, using Gaussian Splatting.

Michael Rubloff

Oct 17, 2024

Platforms

Gracia Launches Dynamic Gaussian Splatting

Gracia launches dynamic captures of Radiance Fields, using Gaussian Splatting.

Michael Rubloff

Research

Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency

Rendering speeds are getting a big boost, with some pesky features addressed.

Michael Rubloff

Oct 16, 2024

Research

Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency

Rendering speeds are getting a big boost, with some pesky features addressed.

Michael Rubloff

Oct 16, 2024

Research

Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency

Rendering speeds are getting a big boost, with some pesky features addressed.

Michael Rubloff

Platforms

Postshot Announces Unreal Engine Beta

Postshot adds a UE5 plugin in its latest pre-release build.

Michael Rubloff

Oct 14, 2024

Platforms

Postshot Announces Unreal Engine Beta

Postshot adds a UE5 plugin in its latest pre-release build.

Michael Rubloff

Oct 14, 2024

Platforms

Postshot Announces Unreal Engine Beta

Postshot adds a UE5 plugin in its latest pre-release build.

Michael Rubloff

News

ITS Launches with Radiance Fields

The International Talent Support 2025 launch video arrives full of Radiance Fields.

Michael Rubloff

Oct 9, 2024

News

ITS Launches with Radiance Fields

The International Talent Support 2025 launch video arrives full of Radiance Fields.

Michael Rubloff

Oct 9, 2024

News

ITS Launches with Radiance Fields

The International Talent Support 2025 launch video arrives full of Radiance Fields.

Michael Rubloff