News

OpenAI Launches Sora and the World

Michael Rubloff

Feb 15, 2024

Sora
Sora

Just earlier today OpenAI unveiled Sora, arguably the most consequential announcement in the past twelve months

Sora is a text to video model announced just after midday on the East coast. To be blunt, it's significantly stronger than any existing public method, surpassing others by orders of magnitude.

That's a large claim, given the progression of technology. On its face, generating that level of high fidelity of video is extremely impressive, but where it becomes incredible is the generating of view consistent, parallax filled large areas. This unlocks the ability of large scale generative radiance fields.

What does this mean? It signifies that generating high-fidelity, large-scale radiance fields will soon be a reality for the public. The most challenging part is now behind us. It is now possible to generate view consistent locations and lifelike video output through text. That is so difficult. That is so completely difficult.

Like the vast majority of people, I only have access to the publicly available videos they've published, but that's all I need. According to the Sora announcement, they're able to generate 60 seconds of footage per prompt. I took the demo video of Santorini as my base. It's only 9 seconds, but that's sufficient.

From here, I simply slice the video up the same way I would with any existing video and run it through COLMAP and then into nerfstudio, but you can use any radiance field company. It works on the first try, with all camera poses found and trains like any other radiance field. Here is the resulting NeRF from Nerfacto.

This was taken from a 9 second video. Imagine what you could do with the full 60 seconds. Imagine what you can do with two minutes. This means, not only is it possible, they've already achieved it. I am very curious to see what OpenAI will do from here on out, but as this stands, they can generate large scale radiance fields from a text prompt. I hadn't anticipated seeing large-scale generative radiance fields until much later.

Further, there is absolutely zero reason why this cannot be automated from here on out; in fact it would be trivial. When Sora is unleashed to the world at large, it will allow for the rapid creation of hyper-realistic three dimensional worlds.

Before my thoughts around radiance fields were limited to capturing the existing physical world and generating smaller objects, but what this means expands possibilities to an almost unfathomable level. You can generate hyperrealistic large scale three dimensional radiance field worlds based off of text.

This also has significant implications for existing radiance field companies with generative capabilities, such as Luma and CSM. Given this is now proven to be possible, I can only wonder what's already been created behind closed doors. If they have not been pursuing this already, I imagine they now are.

Featured

Featured

Featured

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

4/11/24

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

4/10/24

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

4/9/24

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

4/11/24

Recent articles

Recent articles

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024