t3d.js is a lightweight and versatile 3D rendering library designed for web-first experiences. It draws inspiration from three.js but takes a distinct approach, focusing on modularity, extendability, and high-performance rendering. While three.js is great for developers with minimal 3D knowledge, t3d aims to cater to those who seek more control and flexibility, particularly in achieving optimized graphical effects.
In this interview, I speak to the developer of t3d, Shawn, and we explore t3d.js's journey—from its origins as a learning project to its role in pushing the boundaries of web-based 3D graphics. We also delve into the unique challenges of implementing Gaussian Splatting in t3d, the optimizations made, and how the library is evolving to meet new technological demands. Whether you're curious about extending web rendering frameworks or are keen to learn about practical solutions for complex 3D problems, this conversation sheds light on the capabilities and vision of t3d.js.
Could you explain what t3d is and how it differs from other game engines or 3D frameworks?
t3d.js is a lightweight, web-first, and extendable 3D rendering library.
t3d.js is largely inspired by three.js. I have extensively used three.js for projects, and t3d started as a learning and imitation project. However, I found that early versions of three.js were difficult to extend, especially for achieving high-performance post-processing with the renderer. Therefore, I decided to incorporate my design philosophy into this project. The encapsulation is more modular and closer to the underlying WebGL or WebGPU, making it easier for developers to extend. Now I hope the functionality encapsulation level of t3d lies between tinyGL and three.js: offering more graphical features than tinyGL, but more flexibility than three.js. Three.js is great because it allows people with minimal 3D knowledge to develop graphical applications. However, I want t3d to be a more handy tool for programmers.
Since its open-source release in early 2023, we have added many features to t3d, such as clustered lighting, volume rendering, occlusion queries, and various practical post-processing effects.
https://shawn0326.github.io/t3d-scene-demo/
More than a hundred real-time point lights and spotlights were used in the scene, and it can still be rendered on mobile phones at a frame rate of 60 frames per second.
https://uinosoft.github.io/t3d-demos/effect2024/
In summary, the characteristics of t3d are:
As simple and elegant as three.js
Smaller core package size, currently only half of the three.js core package
Provides more atomic rendering commands, enabling higher performance post-processing
Designed for both WebGL and WebGPU architectures
More features, like clustered lighting etc
Offers common extension libraries such as Gaussian splatting, post-processing, and particles
Extensions:
t3d-effect-composer - Post Effects extension for t3d.js.
t3d-particle - This is a particle system developed based on t3d.js.
t3d-pano - Panorama extension for t3d.
t3d-3dtiles - A 3dtile extension based on t3d.js.
t3d-dynamic-sky - Dynamic sky addon for t3d.
t3d-gaussian-splatting - A t3d-based implementation of 3D Gaussian Splatting.
Links:
Github: github.com/uinosoft/t3d.js
Examples: uinosoft.github.io/t3d.js/examples/
Twitter: x.com/shawn03261
Discord: discord.com/invite/urB54PPXc4
What challenges did you face when implementing Gaussian Splatting within t3d?
The main challenges faced when implementing Gaussian Splatting within t3d include:
Point Sorting Algorithm Overhead: In the current rendering process, the rendering order of Gaussian Splatting points needs to be sorted to achieve correct blending results. When there are many points, the sorting algorithm becomes a bottleneck.
Rendering Overhead for Large Numbers of Points: Real-time rendering of a large number of points often requires rendering tens of millions of points. Additionally, the rendering order of these points needs to be adjusted every frame, which adds to the complexity.
How did you optimize Gaussian Splatting for performance in t3d, especially with regard to memory management and rendering speed?
To optimize Gaussian Splatting for performance in t3d, especially with regard to memory management and rendering speed, we implemented the following strategies:
Sorting Algorithm Overhead:
Sorting: We used WebWorker to implement the sorting algorithm in a separate thread, effectively alleviating the rendering pressure on the main thread. By replacing JavaScript's native array sorting algorithm with a counting sort algorithm, we significantly improved sorting performance, reducing rendering errors caused by sorting lag during viewpoint changes.
Rendering:
Instanced Rendering: We adopted instanced rendering for points and stored the position and color information of the points in a fixed texture. This allowed us to extract the indices into an Instanced Attribute separately, reducing the overhead of uploading indices during updates.
How does t3d handle the unique characteristics of Gaussian Splatting, such as splatting large numbers of points in real time?
t3d handles the unique characteristics of Gaussian Splatting, such as splatting large numbers of points in real time, by using instanced rendering for points. The positions and colors of the points are stored in a fixed texture. This allows the indices to be extracted into an Instanced Attribute separately, reducing the overhead of uploading indices during updates.
What changes or customizations did you make to t3d to accommodate Gaussian Splatting?
Since Gaussian Splatting in t3d is implemented in an extension library and the t3d core library exposes low-level rendering functionalities, no modifications were made to the core library during the implementation process. Support for inputting floating-point textures in the vertex shader is a necessary feature.
In what specific scenarios do you think t3d is particularly suited for Gaussian Splatting?
t3d is particularly suited for scenarios where Gaussian Splatting models or scenes need to be combined with existing traditional models or 3D scenes. Since we have encapsulated Gaussian Splatting into a standard 3D node, you can freely arrange and use it within 3D scenes and integrate it with other 3D elements.
How does the t3d implementation compare to other platforms in terms of rendering quality and scalability?
Since Gaussian Splatting does not require real-time lighting calculations, its rendering quality is relatively consistent across different platforms. Due to t3d's strong scalability, you can freely modify the existing Gaussian Splatting rendering, such as shaders, etc. However, this requires some background knowledge in 3D technology.
Are there any planned features or improvements you are considering for the t3d Gaussian Splatting implementation?
Yes, there are several planned features and improvements:
Rendering Order Improvement: Whether using WebWorker or WebAssembly to accelerate sorting, we still need to re-upload the updated index information to the GPU every frame. Tests have shown that this overhead is significant. While using counting sort, I realized that this sorting algorithm might be implemented in the shader. This could allow the GPU to handle index sorting, enhancing sorting performance and avoiding the overhead of transferring index information from the CPU to the GPU. This work is still in the preparation stage, and I cannot yet confirm that WebGL can achieve this, but WebGPU certainly can.
Rendering Performance: Instanced rendering is still not fast enough. It might be better to use traditional merged rendering. This requires further testing.
How do you see this implementation evolving as both t3d and Gaussian Splatting technologies advance?
Currently, Gaussian Splatting technology primarily shortens the time from model scanning to rendering by eliminating the need for topology reconstruction, thus offering higher real-time performance. In the future, this technology could enable sports live streaming where viewers can observe the content from any angle or project 3D images using new display devices.
However, in terms of rendering alone, I don't believe it brings revolutionary improvements (perhaps it is more beneficial for hair rendering). It also introduces new challenges such as ray tracing accuracy and depth sorting, which need to be addressed in the future. For large-scale rendering scenes, combining 3D tiles with Gaussian Splatting could be a good idea, but currently, there doesn't seem to be a unified standard in the industry.
Can you describe how you handle the rendering pipeline in t3d for Gaussian Splatting?
Here is the architecture:
What hardware setups work best for running Gaussian Splatting in t3d?
Since t3d is a web-based rendering library, it has very flexible hardware requirements. As long as the browser supports WebGL2, it can run. The better the graphics card, the larger the scale of scenes it can render.
Are there any collaborations within the t3d community to enhance or support Gaussian Splatting implementations?
Currently, t3d-gaussian-splatting has not received many PR contributions from the community, but we welcome them.
Has anyone created anything really cool using t3d’s Gaussian Splatting implementation?
Currently, t3d-gaussian-splatting is primarily used by me in some commercial projects, such as digital twin projects, but it is still a work in progress. We are also exploring interactive solutions that combine 3D Gaussian Splatting scenes with traditional rendering scenes.
How can developers interested in t3d benefit from using your Gaussian Splatting implementation?
t3d-gaussian-splatting is a completely open-source library, and it is entirely free and available for commercial use.
We strive to maintain its performance and capabilities at the forefront of web technologies.
We have encapsulated Gaussian Splatting into a standard 3D node, allowing you to freely arrange and use it within 3D scenes and integrate it with other 3D elements.
t3d is BSD 3.0 Licensed and can be accessed through GitHub or their demo.
