Neural Rendering for Unity Muse

Unity Muse is the AI creator toolkit Unity rolled out as part of their generative-AI push. Behind the demos was a question: how do you take research-grade neural rendering models and make them shippable inside a real-time engine, on the machines real creators use?

This work lived at the boundary between research and the engine:

• Custom GPU kernels & ML operators for the model paths Muse cared about - the ones where stock framework ops left throughput on the table.
• Sentis integration so models could run client-side inside Unity rather than rounding through a cloud endpoint.
• R&D direction for the team exploring what else neural rendering could do for game lighting and post effects - a research backlog that fed several Muse and Sentis showcases.

What shipped

• Visual-transformation features in the Unity Muse beta.
• Demo content in Unity Sentis launch material.
• A baseline of ML-op performance that the rest of the team could build on.