[World Model] Runway ML Solaris First Look

youtu.be/c2yCePPnrSA

I want to try keeping track of big AI announcements as they break because it could be fun to look back at our thoughts one day.

So Runway ML shows off Solaris - a World Model - I highly suggest you go check out their press release - https://runway.com/news/research/introducing-solaris - and you can even sign up for Early Access.

I want to share a bunch of thoughts I had, specifically about what the model still struggles with

from runway.com/news/research/introducing-solaris

What It Can't Do Yet

Solaris is strongest at ambient motion, click-and-drag interactions and scene transitions. Several important challenges remain:

Text. Stable, legible text remains one of the hardest problems in video generation, yet interfaces depend on it more than almost any other visual domain. One practical path is a hybrid system in which image models render text-heavy views whenever a brief pause is acceptable, while video models handle continuous interaction. Fully real-time generated text remains an open challenge.

Isn't that interesting? Why can't the Ai hold text in place? Why can't it visualize it as a physical object?

^ I guess that's the million-dollar question

from Runway's Solaris announcement

It is interesting that despite all the advances, time still seems to be a heavy cost for an artificial intelligence

Check out what they said about the problem of integration:

from Runway's Solaris announcement

Isn't it so funny that they have to somehow make this creature function and fit with a bunch of old technology?

And then there is quite a bit I was confused about ...

from Runway's Solaris announcement

So everyone is just going to live in their own interface? And like their own versions of every app? How would that work? Or does it only work with the apps you download - like, you open your interface, your virtual spouse greets you - you want to post something on instagram, and Solaris just does it for you? like they're able to do it because you have the app?

I feel like they showed their true motivation with this point:

from Runway's Solaris announcement

It's so funny to see them signal to the businesses like that. Maybe I'm jaded, but I don't think they created this technology to be altruistic and benefit the good of mankind lol this is a take-over-the-world type of technology.

The rest of my questions in no particular order:

How do you constrain it? Should a line be drawn about what you're allowed to act out in these virtual worlds?

Is it able to create optical illusions? Like does it understand what its world looks like from our point of view? Could you load up a messy warehouse and play hide-and-go-seek with a character? Like could it know how to block itself from your vision, and if you rotated the camera, would it know how to stay out of sight?

I think they're underestimating how much people prefer dumb technology - like do you want your home screen to be alive? I don't think I do.

I heard the Midjourney CEO talk recently about the idea of Artificial Intelligence, and to paraphrase him, he said something like 'these things (Ai Models) obviously want to be world models and focusing completely on a slice like image generation is probably not the optimal thing moving forward' --- I want to emphasize that I am paraphrasing his words - don't quote the CEO on my conclusions - but I found his point to be interesting to think about nonetheless.

This is how I pictured 'Acting' and Filmmaking working in the future. You have this birds-eye Omnipresent view of a scene, and you have actors who are just hanging out - and then you'll be able to draw a path and explain that Actor #1 is supposed to walk over to the fridge and open it. And you'll be able to switch to a 'camera mode' to watch the scene play out and frame it how you want - you'll be able to say things like 'oh that was good, do that again' and the Actor will repeat its performance.

^ Do you think I'm off about any of these? What's on your mind