Realtime AI/ML game technologies *spawn

[Shifty Geezer]

Yeah tbh I really don’t think that video looks good at all. Most of these ‘AI re-imagined’ games look like stylistic messes.

The funniest parts are the where the AI fancies itself as a Surrealist. You have motorbikes turning into benches, fragments of car on the sidewalk as features, and especially unexpected was the realtime gender transformation at the end! It can't even maintain colour consistency across time. As the model is fed more data, it'll become more robust, but it'll also presumably increase in exponential complexity. Will it get to a point where it's just unwieldy, taking too much time to train to produce inconsistent results? I guess when we get perfect, artefact free upscaling from AI, we'll have a first step towards AI generated realtime visuals.

 

[DavidGraham]

It can't even maintain colour consistency across time.

This is an artificial limitation, the service only accepts 10 seconds clips, that's why things change every 10 seconds. This wouldn't be a problem at all if the 10 seconds rule doesn't exist.

Yeah tbh I really don’t think that video looks good at all. Most of these ‘AI re-imagined’ games look like stylistic messes.

It's a very basic and generic AI video generator not intended for game use at all, and it's not even the state of the art, yet we are getting these perfectly serviceable results, many call it amazing even, especially when you see results for GTA IV, Metro Exodus and others. Imagine a more gaming specific model.

 

[Shifty Geezer]

This is an artificial limitation, the service only accepts 10 seconds clips, that's why things change every 10 seconds. This wouldn't be a problem at all if the 10 seconds rule doesn't exist.

No, it breaks across fractions of a second. At 25 seconds, the protagonist's left leg changes from navy to white. The driver is pulled out as a white blob, maybe a jacket, but after rolling on the floor they have turned into a navy top. At 48-49 seconds, a white car turns into a black car.

It's a very basic and generic AI video generator not intended for game use at all, and it's not even the state of the art, yet we are getting these perfectly serviceable results,

They aren't serviceable at all! When you first watch it, it looks incredible, but subsequent viewings reveal just how incredibly broken it is.

many call it amazing even, especially when you see results for GTA IV, Metro Exodus and others. Imagine a more gaming specific model.

It is amazing! But it's also very, very far from being something people can use in games in realtime to make games that look like movies. This is the first 10% of the journey and it's an incredible. The next 10% requires vastly more effort, and the next 10% after that, even moreso. No-one knows what it'll take to train and operate an ML to produce solid results, and expectations for the results in the near future are based solely on hope. As I say, ML can't yet produce missing details with complete accuracy for upscaling, which is a far simpler job. When we can feed an upscaler with game data and image data and produce faultless images, we'll have evidence as to what rich-data models can achieve.

 

[Cappuccino]

This is an artificial limitation, the service only accepts 10 seconds clips, that's why things change every 10 seconds. This wouldn't be a problem at all if the 10 seconds rule doesn't exist.


It's a very basic and generic AI video generator not intended for game use at all, and it's not even the state of the art, yet we are getting these perfectly serviceable results, many call it amazing even, especially when you see results for GTA IV, Metro Exodus and others. Imagine a more gaming specific model.

I don’t want to get testy but if you think this is at all serviceable I seriously question your taste here. This looks horrible and soulless. Why are we trying to inject AI slop into an art form we are all supposedly into?

Notice how painters and photographers dislike the AI garbage? How people reject AI written scripts as cold and inhuman (because they are)? Why are we trying to bring this into games? Art is a fundamentally human endeavor, giving up genuine creativity just to make games look better (these don’t but let’s pretend AI could make it look better) feels horrible.

 

[Boss]

I don’t want to get testy but if you think this is at all serviceable I seriously question your taste here. This looks horrible and soulless. Why are we trying to inject AI slop into an art form we are all supposedly into?

Notice how painters and photographers dislike the AI garbage? How people reject AI written scripts as cold and inhuman (because they are)? Why are we trying to bring this into games? Art is a fundamentally human endeavor, giving up genuine creativity just to make games look better (these don’t but let’s pretend AI could make it look better) feels horrible.

Not to mention, it'll erase a lot of jobs. AI should be used to tackle problems that humans struggle to solve while aiming to minimize job losses. The tech industry as a whole is brutal right now. Huge layoffs, global competition due to offshoring, and a supply side market in terms of workers are some of the many problems we're facing right now. The idiocy of many corporations chasing short term profits by means of layoffs is the failure to realize that people actually need income to continue to purchase the products.

Anyway, all of that is to say that AI for the most part needs to stay out of gaming at least until the industry reaches a new equilibrium.

 

[DavidGraham]

Why are we trying to inject AI slop into an art form we are all supposedly into?

Because a simple non special generic AI successfully injected human like characters into game scenes, with realistic clothing, hair and physics, with practically zero programming effort, imagine the potential when it's a more specialized directed model in the hands of professionals.

Why are we trying to bring this into games? Art is a fundamentally human endeavor, giving up genuine creativity just to make games look better feels horrible

This is just an overly sentimental way of thinking about it, no different than conventional artists objecting to use computers to do 2D or 3D drawings instead of doing it manually like humans used to do for hundreds of years, no different than sculptures objecting to people using 3D printing instead of traditional sculpting, it is a very superficial way to think about this. AI will be just another tool in the hands of game programmers and game artists to do their thing more productively, more cheaply and in a much much more visually stunning way.

That's certainly way better than the clusterfuck of an industry that is the gaming industry right now with it's overly blown budgets, dirty money milking schemes, repetitive boring designs and slow visual progress. I mean if you like the status quo then be my guest, but I would much prefer a small team of creative people successfully making unique and polished experiences through the assistance of AI, rather than endure another decade of this bullshit we have as a gaming industry.

and expectations for the results in the near future are based solely on hope

Not on hope at all, you've seen the Adope video showcasing turning hand drawn 2D figures into full 3D models.

We are seeing developers gearing up game engines for AI integration.

This LWM is allegedly capable of generating all the components of a video game, from environments, 3D models, and gameplay to NPC (non-player character) behavior - along with detailed metadata

Startup Tales Wants to Let You Build Your Own Games with Generative AI Beginning Next Month

AI startup Tales wants to let you build your own games with gen AI starting from a simple text prompt. Early access launches next month.

wccftech.com

We've seen textures generated on the fly with a simple prompt.

We are seeing IHVs experimenting with augmenting rendering with AI, and creating a preliminary vision on how things will go.

The purpose of the image generator is to synthesize an image from a novel, unobserved view of the scene. The generator receives: parameters of the camera, a G-buffer rendered using a traditional renderer from the novel view, and a view-independent scene representation extracted by the encoder.

We envision future renderers that support graphics and neural primitives. Some objects will still be handled using classical models (e.g. triangles, microfacet BRDFs), but whenever these struggle with realism (e.g. parts of human face), fail to appropriately filter details (mesoscale structures), or become inefficient (fuzzy appearance), they will be replaced by neural counterparts that demonstrated great potential. To enable such hybrid workflows, compositional and controllable neural representations need to be developed first.

Neural Scene Graph Rendering | NVIDIA Real-Time Graphics Research

We present a neural scene graph---a modular and controllable representation of scenes with elements that are learned from data. We focus on the forward rendering problem, where the scene graph is provided by the user and references learned elements. The elements correspond to geometry and...

research.nvidia.com

We are seeing bits and pieces, but they are enough to form a picture of where things are heading, they are solid pieces based on actual progress made, certainty not based on hope. Only time will tell of course.

ML can't yet produce missing details with complete accuracy for upscaling

Complete accuracy is never a prerequisite for building an interactive gaming experience, the whole rendering process is just an approximation, and many many inaccuracies are allowed for the purpose of other greater gains.

 

[Kaotik]

Because a simple non special generic AI successfully injected human like characters into game scenes, with realistic clothing, hair and physics, with practically zero programming effort, imagine the potential when it's a more specialized directed model in the hands of professionals.

Did what now? Those game clips are literally reimagined videos, you could insert real movie, cartoon, whatever and it would do the same

 

[Cappuccino]

Because a simple non special generic AI successfully injected human like characters into game scenes, with realistic clothing, hair and physics, with practically zero programming effort, imagine the potential when it's a more specialized directed model in the hands of professionals.



This is just an overly sentimental way of thinking about it, no different than conventional artists objecting to use computers to do 2D or 3D drawings instead of doing it manually like humans used to do for hundreds of years, no different than sculptures objecting to people using 3D printing instead of traditional sculpting, it is a very superficial way to think about this. AI will be just another tool in the hands of game programmers and game artists to do their thing more productively, more cheaply and in a much much more visually stunning way.

That's certainly way better than the clusterfuck of an industry that is the gaming industry right now with it's overly blown budgets, dirty money milking schemes, repetitive boring designs and slow visual progress. I mean if you like the status quo then be my guest, but I would much prefer a small team of creative people successfully making unique and polished experiences through the assistance of AI, rather than endure another decade of this bullshit we have as a gaming industry.


Not on hope at all, you've seen the Adope video showcasing turning hand drawn 2D figures into full 3D models.

We are seeing developers gearing up game engines for AI integration.

Startup Tales Wants to Let You Build Your Own Games with Generative AI Beginning Next Month

AI startup Tales wants to let you build your own games with gen AI starting from a simple text prompt. Early access launches next month.

wccftech.com

We've seen textures generated on the fly with a simple prompt.

We are seeing IHVs experimenting with augmenting rendering with AI, and creating a preliminary vision on how things will go.

Neural Scene Graph Rendering | NVIDIA Real-Time Graphics Research

We present a neural scene graph---a modular and controllable representation of scenes with elements that are learned from data. We focus on the forward rendering problem, where the scene graph is provided by the user and references learned elements. The elements correspond to geometry and...

research.nvidia.com

We are seeing bits and pieces, but they are enough to form a picture of where things are heading, they are solid pieces based on actual progress made, certainty not based on hope. Only time will tell of course.


Complete accuracy is never a prerequisite for building an interactive gaming experience, the whole rendering process is just an approximation, and many many inaccuracies are allowed for the purpose of other greater gains.

This is all incredibly bleak and nothing like using computers to accelerate 2D and 3D rendering. If one cannot tell the difference between using a computer as a tool and having an AI literally do all of the work then you probably never really appreciated video games as art. Which isn’t uncommon, I’ve seen that type of sentiment pretty often online, where the endgoal isn’t making good art and experiencing it but instead striving for lifelike graphics with design taking a backseat.

 

[pcchen]

This is all incredibly bleak and nothing like using computers to accelerate 2D and 3D rendering. If one cannot tell the difference between using a computer as a tool and having an AI literally do all of the work then you probably never really appreciated video games as art. Which isn’t uncommon, I’ve seen that type of sentiment pretty often online, where the endgoal isn’t making good art and experiencing it but instead striving for lifelike graphics with design taking a backseat.

To be honest for an artist how one creates the art is generally not the biggest concern, as long as the artist has the final call on how the result looks (or sounds). For example, people used to value the artistry of painting something very realistic. However, after the invention of photography, the art, while still exists, is now of much less importance. On the other hand, photography itself becomes a form of art. How you choose to frame a picture, the subject of a picture, etc. are of artistic values.

Today many people looking at AI generated images and think they can't be art. However, if you ever tried to generate something using the popular AI image generators you'll know that it's never that simple. AI rarely generates the exact result you want. So in a way it's a kind of art where the artist's role is in choosing and adjusting what the AI generated, in a way similar to what a photographer picking the best picture from maybe more than a thousand pictures.

It's of course much harder for real time generated contents, but that's not very far from 3D rendering, as artists can't really anticipate all possible rendering results, from all different looking angles, lighting conditions, etc. Even the 3D models and textures are not necessarily strictly under control. There are LOD algorithms which may reduce 3D models in unexpected ways, etc. So artists can only check as much as possible and hope for the best. With real time AI generated images and videos it's the same story albeit probably more difficult, but I believe we'll have more stable AI image generators in the future, where artists can control more aspects of how an object in an AI generated video looks and even how it'll be animated.

 

[DavidGraham]

Did what now? Those game clips are literally reimagined videos, you could insert real movie, cartoon, whatever and it would do the same

Which is what I mean by simple, generic non special AI, it's not even made for games, I mean it's right there in the bit you quoted.

but instead striving for lifelike graphics with design taking a backseat.

Yep, many people want that above anything else, it's a big subset of the market, and when there is demand for such a thing, a product will be made to address it. It's why you have things such as 3D googles, Virtual Reality headsets and Augmented Reality devices.

Also, let's not kid ourselves here, it's not like the current "art" is being exploited to it's maximum potential, the industry is on a massive downwards spiral right now with games copying each others looks and boring repetitive designs becoming the norm. The freedom of making art did lead to innovations in the beginning, now it's not, innovation and experimentation have practically died except in very limited circles.

having an AI literally do all of the work

Who said anything about that? Again, AI will be just another tool in the toolbox, artists and programmers will tweak it to achieve their desired look, or desired looks (by offering a multitude of curated looks for the player to choose from).

This attack on AI generated content is really strange, the process of game production is full of automated things, there is no difference between automation through a handcrafted code or through AI, it's all automation in the end.

Ray/Path Tracing is automation of lighting, shadows and reflections, procedural animations is just automation of animation, procedural level designs is just automation of asset placement, procedural textures is just automation of textures variations, tessellation is just automation of polygon creation, procedural weather is just automated weather, dynamic time of day is just automation of lighting variations, physics simulation is just automation of physics, AutoHDR/RTX HDR are just automated HDR, these are things that were used and are widely used to this day, they didn't affect the art of making the game, not everything has to be hand made or else it looks bad! AI is going to be just another automation tool that enhances some of these these already automated things. It will also be able to automate other things previously not automated, this is the natural progression of automation.

 

[Kaotik]

Which is what I mean by simple, generic non special AI, it's not even made for games, I mean it's right there in the bit you quoted.

But your whole claim is wrong, you said it inserts characters to game world. It doesn't insert anything anywhere. It's AI model trained on countless hours of video material from YouTube and pirated films, which reimagines video it's being fed, nothing more, nothing less.

 

[DavidGraham]

But your whole claim is wrong, you said it inserts characters to game world. It doesn't insert anything anywhere. It's AI model trained on countless hours of video material from YouTube and pirated films, which reimagines video it's being fed, nothing more, nothing less.

I didn't claim it inserted anything in any game, who even claimed that? If you misread my premise then you should start reading the whole conversation again.

 

[Kaotik]

I didn't claim it inserted anything in any game, who even claimed that? If you misread my premise then you should start reading the whole conversation again.

You did. Inject, insert, same difference. And it's not even "generic AI" but specifically AI model trained on videos.

Because a simple non special generic AI successfully injected human like characters into game scenes, with realistic clothing, hair and physics, with practically zero programming effort, imagine the potential when it's a more specialized directed model in the hands of professionals.

 

[DavidGraham]

You did. Inject, insert, same difference

Injected into game scenes, game scenes as in scenes of gameplay from actual games. Again you are misreading or misunderstanding or being pedantic, I don't know which.

it's not even "generic AI"

Now you are being pedantic about it, it's a generic video AI not made specifically for games. This is very obvious from the flow of conversation.

 

[Kaotik]

Injected into game scenes, game scenes as in scenes of gameplay from actual games. Again you are misreading or misunderstanding or being pedantic, I don't know which.


Now you are being pedantic about it, it's a generic video AI not made specifically for games. This is very obvious from the flow of conversation.

I might be slightly on the pedantic side, but if someone without deeper knowledge on the matter would read your post, it would literally read like you're suggesting AI inserted something into game scenes rather than just reimagined a video which could just aswell be cartoon or whatever.
You need to remember that even on forums like these there are people who don't have the knowledge you and most active users here have, let alone people stumbling around to read from google. That's how fake news get started.

 

[Shifty Geezer]

Because a simple non special generic AI successfully injected human like characters into game scenes, with realistic clothing, hair and physics, with practically zero programming effort,

And terrible results.

imagine the potential when it's a more specialized directed model in the hands of professionals.

Is that model even possible in reality?

Not on hope at all, you've seen the Adope video showcasing turning hand drawn 2D figures into full 3D models.

Yes, I posted it. But one can't extrapolate from these early results to say 3D games will be photorealistic and amazing soon. It's a showcase, so the presentation would have been rehearsed and content selected that they knew worked. How robust is the full system? What would it take to make that realtime instead of pre-calculating each pose?

We are seeing bits and pieces, but they are enough to form a picture of where things are heading, they are solid pieces based on actual progress made, certainty not based on hope. Only time will tell of course.

As a logical person, I point you to all the other tech advances we've had over the years across so many fields that have pointed to a future that didn't happen as they appeared it would. Where are my methanol fuel-cell batteries?! These bits and pieces show the aspirations, but not if they are possible and at what point they then become possible.

Complete accuracy is never a prerequisite for building an interactive gaming experience, the whole rendering process is just an approximation, and many many inaccuracies are allowed for the purpose of other greater gains.

The original sentiment in this recent discussion was what would it take to achieve photorealism. Glitches and issues that are tolerable in computer games miss that target. So you could have a future where games are approaching photorealism in 5 years, say, but aren't confused with TV because of all the artefacts and issues.

I'll refocus the discussion on that past couple of pages to what started it. Tim Sweeney said 40 TFs would be enough for photorealism. We've hit 40 TFs and conventional rendering clearly can't achieve photorealism. Even using ML assists with many other TFs and TOPs to upscale and denoise etc., we aren't hitting photorealism. So my question was what will it take and in what timeframe. You said you think AI will get there soon.

So that means looking at the current state of the art and thinking where are we headed? My ultimate disagreement with you is you are looking at pieces are seeing 1) results better than I'm seeing and 2) expecting all the pieces will come together quickly and effectively.

Key things that are missing to hit those TV-like results in a reasonable timeframe

1) Your examples aren't realtime. Creating blobby pseudovideo with odd issues takes a lot of time and processing power. You need a massive improvement just to get these results in realtime. What level of GPU advance, how many TOPs/GOPs/PLOPs is that going to take, and when will that be ready, before even thinking about making the results better?
2) They aren't particularly good. You say they have things like hair physics and cloth physics, but these don't move right. What will it take to perfect them? Just more and more training on more and more data? Or will there need to be different solvers in play, and then how will these come together? And how much power will be needed for each element?
3) How do you train an ML on 3D game space? A video solution looks at 2D video and you can show it lots of 2D video. If you want to train a game, you need to be able to position different elements. You could spend countless hours just evaluating different states for a small fraction of time in a game. and how do you develop a ground truth to train the ML against? nVidia trains DLSS by rendering super-high quality 2D images and saying, "hey ML, this is what you're trying to achieve." How do you get a game that looks like a game and train it to look like reality? Do you render the game fully path-traced and feed the results to the source state that produced those visuals? Do you 3D scan the world and match that to a virtual state? How much effort will it be to create a ground truth for one game, let alone a model that can handle any and every game?

This wonderful photorealistic future is by no means a certainty. I think there are some areas ML would be very effective. My instinct is you could ML lighting on top of conventional geometry. You might also be able to ML enhance a game so lower rendering quality can be enhanced to Ultra quality, and you could train an ML on a better-than-realtime model to enhance the base model. But the current shortcomings between the best we have now and what it takes to create reality are huge. Even offline CGI for movies can struggle to look real. Even if the tech becomes possible, will it be affordable?

 

[Cappuccino]

Which is what I mean by simple, generic non special AI, it's not even made for games, I mean it's right there in the bit you quoted.
Yep, many people want that above anything else, it's a big subset of the market, and when there is demand for such a thing, a product will be made to address it. It's why you have things such as 3D googles, Virtual Reality headsets and Augmented Reality devices.

Also, let's not kid ourselves here, it's not like the current "art" is being exploited to it's maximum potential, the industry is on a massive downwards spiral right now with games copying each others looks and boring repetitive designs becoming the norm. The freedom of making art did lead to innovations in the beginning, now it's not, innovation and experimentation have practically died except in very limited circles.


Who said anything about that? Again, AI will be just another tool in the toolbox, artists and programmers will tweak it to achieve their desired look, or desired looks (by offering a multitude of curated looks for the player to choose from).

This attack on AI generated content is really strange, the process of game production is full of automated things, there is no difference between automation through a handcrafted code or through AI, it's all automation in the end.

Ray/Path Tracing is automation of lighting, shadows and reflections, procedural animations is just automation of animation, procedural level designs is just automation of asset placement, procedural textures is just automation of textures variations, tessellation is just automation of polygon creation, procedural weather is just automated weather, dynamic time of day is just automation of lighting variations, physics simulation is just automation of physics, AutoHDR/RTX HDR are just automated HDR, these are things that were used and are widely used to this day, they didn't affect the art of making the game, not everything has to be hand made or else it looks bad! AI is going to be just another automation tool that enhances some of these these already automated things. It will also be able to automate other things previously not automated, this is the natural progression of automation.

Again, comparing this to HDR post processing enhancements accelerated by AI is comparing apples to bowling balls. It is creatively bankrupt to do things like this and pretty much everyone that appreciates the art form sees it for what it is.

 

[Cappuccino]

To be honest for an artist how one creates the art is generally not the biggest concern, as long as the artist has the final call on how the result looks (or sounds). For example, people used to value the artistry of painting something very realistic. However, after the invention of photography, the art, while still exists, is now of much less importance. On the other hand, photography itself becomes a form of art. How you choose to frame a picture, the subject of a picture, etc. are of artistic values.

Today many people looking at AI generated images and think they can't be art. However, if you ever tried to generate something using the popular AI image generators you'll know that it's never that simple. AI rarely generates the exact result you want. So in a way it's a kind of art where the artist's role is in choosing and adjusting what the AI generated, in a way similar to what a photographer picking the best picture from maybe more than a thousand pictures.

It's of course much harder for real time generated contents, but that's not very far from 3D rendering, as artists can't really anticipate all possible rendering results, from all different looking angles, lighting conditions, etc. Even the 3D models and textures are not necessarily strictly under control. There are LOD algorithms which may reduce 3D models in unexpected ways, etc. So artists can only check as much as possible and hope for the best. With real time AI generated images and videos it's the same story albeit probably more difficult, but I believe we'll have more stable AI image generators in the future, where artists can control more aspects of how an object in an AI generated video looks and even how it'll be animated.

If you think AI generated garbage is art then you don’t like art. It’s removing the human aspect completely.

The only people I see that think AI slop is art are tech industry bugmen.

 

[Shifty Geezer]

Please keep emotions out of this thread and focus on tech, whether you agree with it or not. This is a Gaming Technology thread is for discussing what is and what might be possible, and not whether it should or should not be done which is a Gaming Industry or RPSC discussion.

 

[Cappuccino]

Please keep emotions out of this thread and focus on tech, whether you agree with it or not. This is a Gaming Technology thread is for discussing what is and what might be possible, and not whether it should or should not be done which is a Gaming Industry or RPSC discussion.

Why was this conversation split off into a Gaming Technology thread when the discussion was precisely about whether it should or should not happen, regardless of technical ability?