This is a lil riff on this demo using real-time diffusion to generate a starting image that can then be explored as a 3D Gaussian splat.

In this demo, we use a real-time video generation model (LongLive + RIFE for frame interpolation) to animate starting images to "preview" what's behind each portal. I can conjure a portal anywhere in my environment and then peer into what's on the other side via a video stream generated by the model. I can then step through a portal of my choice into the actual explorable world, a 3D Gaussian splat generated using Apple's ml-sharp.

A simple image is now a portal that starts off as a window into what's happening in another world. And when you step through you're not just looking at the world you're in it.

The code for the demo is at https://github.com/yondonfu/scope-portal-demo