Predict: Next gen console tech (10th generation edition) [2028+]

[Shifty Geezer]

I'm not expecting any "magic fix" but there's obvious HW implementation limits that are running up the wall with a unified solution so I suggest we split up the giant mess of a problem that is light transport into it's smaller/simpler subsets ...

Hmmm. Perfect raytracing is actually the simplest, most unified, most elegant solution possible. The problem is its too demanding which requires breaking it down into different aspects. Dividing lighting into specular, shadows, reflections, refractions, AO, GI, etc. is the gigantic mess of ad hoc solutions desperately trying to approximate what path-tracing solves with one basic premise and algorithm, whether using RTRT hacks or not.

 

[Lurkmass]

Hmmm. Perfect raytracing is actually the simplest, most unified, most elegant solution possible. The problem is its too demanding which requires breaking it down into different aspects. Dividing lighting into specular, shadows, reflections, refractions, AO, GI, etc. is the gigantic mess of ad hoc solutions desperately trying to approximate what path-tracing solves with one basic premise and algorithm, whether using RTRT hacks or not.

Well a lot of the quality issues before with their prior art were because of the fact that they either only did pre-baking, use local data (screen space), or were poor world space representations so I advocate that we enhance solutions to these subsets of problem with better/higher quality dynamic world space data structures but we ultimately don't change the scopes of their problem spaces ...

It's highly tempting to reuse a BVH to solve all sorts of problems (AO/GI/reflections/shadows/area lights/translucency/etc) but we need to entertain the possibility though that we don't actually desire to solve literally EVERYTHING especially for real-time rendering ...

 

[raytracingfan]

The ultimate ideal of a single path tracing algorithm solving all aspects of lighting is admittedly a long way off. Even after path tracing becomes standard, certain aspects like subsurface scattering, ray tracing particle effects and volumetric fog, and caustics will still need to be broken off and handled separately until hardware is powerful enough to get them to "just work" via brute force.

Until that time, researching solutions to individual aspects of lighting that are performant on existing HW - whether they use HWRT or not - is useful. Exotic hardware features that exist solely to improve rasterized lighting is a non-starter, although certain features like "hardware support for multi-view rasterization" might catch on if they can be used not just for planar reflections, but also stereoscopic rendering or the picture-in-picture methods some games use to simulate rifle scopes or night-vision goggles.

 

[Lurkmass]

Until that time, researching solutions to individual aspects of lighting that are performant on existing HW - whether they use HWRT or not - is useful. Exotic hardware features that exist solely to improve rasterized lighting is a non-starter, although certain features like "hardware support for multi-view rasterization" might catch on if they can be used not just for planar reflections, but also stereoscopic rendering or the picture-in-picture methods some games use to simulate rifle scopes or night-vision goggles.

Single pass multi-view rasterization could also be helpful for things like dynamic cubemaps or foveated rendering but I specifically envisioned it's main application for multiple planar reflections. I feel like with some reasonable HW improvements (modest implementation complexity) we could make ~4-6 planar reflections at an update rate of 15Hz (30hz for the main view/camera) common place for AAA games with next generation consoles ...

For soft shadows, I don't think there's much choice aside from HW accelerating data structures that can store information for multiple occluders ...

I couldn't care any less for diffuse GI, AO, or translucency currently for some extended period of time ...

 

[Globalisateur]

The ultimate ideal of a single path tracing algorithm solving all aspects of lighting is admittedly a long way off. Even after path tracing becomes standard, certain aspects like subsurface scattering, ray tracing particle effects and volumetric fog, and caustics will still need to be broken off and handled separately until hardware is powerful enough to get them to "just work" via brute force.

Until that time, researching solutions to individual aspects of lighting that are performant on existing HW - whether they use HWRT or not - is useful. Exotic hardware features that exist solely to improve rasterized lighting is a non-starter, although certain features like "hardware support for multi-view rasterization" might catch on if they can be used not just for planar reflections, but also stereoscopic rendering or the picture-in-picture methods some games use to simulate rifle scopes or night-vision goggles.

Yeah about that. Could they start by getting rid of the ROPs die space, replace it with something else (compute / RT / AI) and use compute to do the same job? ROPs are soon becoming some exotic hardware. I am not saying compute should do the same job faster, but it would give more die space to something more useful, and overall making a more efficient hardware. I am probably wrong though.

 

[Seanspeed]

Currently it's not really known what any amount of TOPS or POPS can bring to console gaming. If it can be leveraged for things like physics solvers, it might have a lot of tangible benefits. Presently I think ML methods are limited to upscaling.

What kind of instructions does DLSS use, anyways? Does anybody know, or is this black box information?

 

[trinibwoy]

What kind of instructions does DLSS use, anyways? Does anybody know, or is this black box information?

Good question. I assume it’s relatively high precision FP16 or FP8 at worst. The color space itself is wide ranging so I imagine the DL factors (spatial and temporal deltas, color differences etc) will require some precision for it to work.

 

[arandomguy]

I've seen it referenced as INT8, including as an aside comment in an article from DF by @Dictator (so maybe he knows more from Nvidia directly), but I'm not sure of any otherwise public direct source from Nvidia on this.

XeSS has more sourcing that it uses INT8 (DP4A is INT8 operations).

 

[trinibwoy]

I'm not expecting any "magic fix" but there's obvious HW implementation limits that are running up the wall with a unified solution so I suggest we split up the giant mess of a problem that is light transport into it's smaller/simpler subsets ...

Those subsets are what we have today and they aren’t simpler at all. They’re complicated and severely flawed.

I wouldn't obsess over a temporary experiment and your idea of "real investment" probably involves implementing a circus clown show of more special HW states, large on-die caches, an AI denoiser and all of those are NOT hypotheticals. My recommendation of taking a divide and conquer method can't possibly be any worse than what we're already seeing right now ...

You consider the current state of RT hardware a clown show but what you’re proposing is a much bigger mess of bespoke and limited hardware that ultimately doesn’t solve the underlying problem. I don’t get it. If your goal is to avoid a clown show then RT seems to be your best option.

 

[Lurkmass]

Those subsets are what we have today and they aren’t simpler at all. They’re complicated and severely flawed.

Just because prior art on those solutions had major limitations before doesn't mean we can't make improvements upon them without using RT ...

You consider the current state of RT hardware a clown show but what you’re proposing is a much bigger mess of bespoke and limited hardware that ultimately doesn’t solve the underlying problem. I don’t get it. If your goal is to avoid a clown show then RT seems to be your best option.

@Bold That's my point all along!

There's a very good reason as to why we divided up the the entire problem space of rendering itself into their own separate specialized domains in the first place is so that we can independently optimize their individual subsets for real-time purposes especially for hardware ...

 

[DavidGraham]

but that doesn't mean AI voice variations in lines and pitch and etc. couldn't ship today

The latest demo of ChatGPT 4o is an awesome example of that with genuine human emotions and variations.

You consider the current state of RT hardware a clown show but what you’re proposing is a much bigger mess of bespoke and limited hardware that ultimately doesn’t solve the underlying problem

It's the same old software vs RT argument that has been circling around ever since Turing and DXR showed up, software can do this and that better, without ever showing up and proving any of it, we've been here before dozens of times, all while hardware ray tracing and path tracing marches on with new techniques every year that get integrated into actual games vs almost nothing on the software front.

 

[trinibwoy]

Just because prior art on those solutions had major limitations before doesn't mean we can't make improvements upon them without using RT ...

Ok but let’s wait until those improvements actually materialize before claiming RT is a clown show.

@Bold That's my point all along!

There's a very good reason as to why we divided up the the entire problem space of rendering itself into their own separate specialized domains in the first place is so that we can independently optimize their individual subsets for real-time purposes especially for hardware ...

You’re theorizing that independent solutions will be a net win but you haven’t really said why. Let’s take the example of planar reflections. RT is clearly the most robust solution for that. What’s the alternative that has a better value proposition? If the answer is to aim lower because the alternatives can’t cut it then that’s not really a solution is it? I suspect this is where you’re leading given your earlier comment about GI and transparency not being a priority. Seems most of the industry disagrees.

 

[Lurkmass]

Ok but let’s wait until those improvements actually materialize before claiming RT is a clown show.

Well RT is certainly not any less complex in a hardware implementation sense ...

You’re theorizing that independent solutions will be a net win but you haven’t really said why. Let’s take the example of planar reflections. RT is clearly the most robust solution for that. What’s the alternative that has a better value proposition? If the answer is to aim lower because the alternatives can’t cut it then that’s not really a solution is it? I suspect this is where you’re leading given your earlier comment about GI and transparency not being a priority. Seems most of the industry disagrees.

You can introduce specific data structures that are potentially more optimized for a given special case at hand while you're at the mercy of maintaining a very expensive data structure that has poor memory locality with ray tracing which introduces tons of noise by itself ...

Do you somehow think that optimized solutions for reflections, shadows, area lights, global illumination, etc. look ANYTHING like each other for real-time use ? A good example to look at are Epic Games implementing virtual shadows for the case of hard shadows because even they're uncertain of the future of ray tracing being able to outperform their own highly customized solution in that specific context ...

We could potentially do so much more if we were to lead by that example alone more often and if we weren't so invested in trying to make the bubble that is ray tracing work ...

 

[trinibwoy]

Do you somehow think that optimized solutions for reflections, shadows, area lights, global illumination, etc. look ANYTHING like each other for real-time use ?

Yes I think the optimized hardware solution is the same. RT.

A good example to look at are Epic Games implementing virtual shadows for the case of hard shadows because even they're uncertain of the future of ray tracing being able to outperform their own highly customized solution in that specific context ...

VSMs aren’t a good example because they don’t fundamentally solve the problem. They’re limited to a handful of shadow casting lights just like every other shadow map implementation.

 

[Lurkmass]

Yes I think the optimized hardware solution is the same. RT.

I have a strong suspicion that's not really going to be the case when next set of AAA game consoles rolls around ...

VSMs aren’t a good example because they don’t fundamentally solve the problem. They’re limited to a handful of shadow casting lights just like every other shadow map implementation.

Regardless of your opinion, Epic seems to believe that they still outperform RT-based shadows ...

And besides if no one truly saw any value in ad hoc solutions, we wouldn't have the biggest ISV like Epic Games looking at SDFs and many new projects shipping with their said solution based upon it ...

 

[OlegSH]

you're at the mercy of maintaining a very expensive data structure that has poor memory locality with ray tracing which introduces tons of noise by itself ...

Sorry, but that's hilarious. BVH has nothing to do with locality - it's quite linear. The locality problem is up to the sampling, not BVH. With random sampling, you will get poor locality, of course. With linear raster alike sampling, you'd get the same locality as in rasterization. With RT, it's up to you to decide which method to use. The same applies to noise - if you don't use the stochastic sampling, you're good to go - no noise whatsoever. This whole discussion is inherently flawed (including the hw complexity statements) since, for some reason, you have made some fundamentally wrong assumptions to begin with.

We had been here with the dedicated raster hardware in Pascal for multi-view geometric projections, it got nearly zero adoption. Just let it go, this is a thing of the past. Relax and watch how RT will take over

 

[trinibwoy]

Regardless of your opinion, Epic seems to believe that they still outperform RT-based shadows ...

No Epic has VSMs because it’s the most practical solution for their virtual geometry system. They have never claimed VSMs are an overall better shadowing system.

And besides if no one truly saw any value in ad hoc solutions, we wouldn't have the biggest ISV like Epic Games looking at SDFs and many new projects shipping with their said solution based upon it ...

SDF tracing is still ray tracing.

 

[troyan]

Well RT is certainly not any less complex in a hardware implementation sense ...

You can introduce specific data structures that are potentially more optimized for a given special case at hand while you're at the mercy of maintaining a very expensive data structure that has poor memory locality with ray tracing which introduces tons of noise by itself ...

Do you somehow think that optimized solutions for reflections, shadows, area lights, global illumination, etc. look ANYTHING like each other for real-time use ? A good example to look at are Epic Games implementing virtual shadows for the case of hard shadows because even they're uncertain of the future of ray tracing being able to outperform their own highly customized solution in that specific context ...

We could potentially do so much more if we were to lead by that example alone more often and if we weren't so invested in trying to make the bubble that is ray tracing work ...

UE5 is worse than games with specific RT implementations. Metro Exodus EE has a much better GI systems, Guardians of the Galaxy has a much better reflection systems. Custom Engines have a faster pace than UE5 and are a lot more flexible. UE5 lacks any kind of progression since four years.

 

[DavidGraham]

Epic seems to believe that they still outperform RT-based shadows ...

That's incorrect and you know it, because we've talked about it before.

Epic is enabling using ray traced shadows with Nanite, because it's higher quality. Hardware RT shadows for Nanite is a crucial feature for them, as RT shadows work with all content, while VSMs don't work well with non Nanite content (which covers lots and lots of dynamic objects). RT shadows also allow for lots of shadow casting lights, and will benefit from the improvements to BVH optimizations that Epic made when they optimized HW Lumen for 60fps on consoles.

Epic specifically talked about reducing the number of shadow techniques they have in the engine (more than 20!), so they will focus on using RT shadows and VSM shadows, and that's it. They could combine them when suitable.

Unreal Fest Day 2 | Livestream 3

Tune into the Unreal Fest 2023 Livestream coming direct to you from New Orleans!Schedule (In Central Daylight Time)Wednesday, October 400:00 Day 2 Schedule00...

www.youtube.com

 

[Lurkmass]

Sorry, but that's hilarious. BVH has nothing to do with locality - it's quite linear. The locality problem is up to the sampling, not BVH. With random sampling, you will get poor locality, of course. With linear raster alike sampling, you'd get the same locality as in rasterization. With RT, it's up to you to decide which method to use. The same applies to noise - if you don't use the stochastic sampling, you're good to go - no noise whatsoever. This whole discussion is inherently flawed (including the hw complexity statements) since, for some reason, you have made some fundamentally wrong assumptions to begin with.

We had been here with the dedicated raster hardware in Pascal for multi-view geometric projections, it got nearly zero adoption. Just let it go, this is a thing of the past. Relax and watch how RT will take over

The locality problem is down to the pointer indirection of traversing the different nodes in the hierarchal data structure and any 'recursive' methods you hint to ray tracing can't be used to support PBR materials since makes the assumption that materials with the diffuse property doesn't exist ...

Pascal's single pass stereo functionality (specifically only designed to be used with VR) is not what I had in mind for this as it does not support the use case of arbitrary camera viewpoints (differing x/y/z coords) which is needed for multiple planar reflections so it's a flawed design from the outset ...

No Epic has VSMs because it’s the most practical solution for their virtual geometry system. They have never claimed VSMs are an overall better shadowing system.

They never claimed "better quality" but it's undeniably "better compatibility" for the content that they're seeing ...

SDF tracing is still ray tracing.

Sure but it's not used to solve every problem ...

That's incorrect and you know it, because we've talked about it before.

Epic is enabling using ray traced shadows with Nanite, because it's higher quality. Hardware RT shadows for Nanite is a crucial feature for them, as RT shadows work with all content, while VSMs don't work well with non Nanite content (which covers lots and lots of dynamic objects). RT shadows also allow for lots of shadow casting lights, and will benefit from the improvements to BVH optimizations that Epic made when they optimzied HW Lumen for 60fps on consoles.

Epic specifically talked about reducing the number of shadow techniques they have in the engine (more than 20!), so they will focus on using RT shadows and VSM Shadows, and that's it. They could combine them when suitable.

Unreal Fest Day 2 | Livestream 3

Tune into the Unreal Fest 2023 Livestream coming direct to you from New Orleans!Schedule (In Central Daylight Time)Wednesday, October 400:00 Day 2 Schedule00...

www.youtube.com

They aren't enabling RT shadows with virtualized geometry and this 'feature' you keep talking about isn't documented at all yet as it is still in development/experimental ...

The conclusions drawn from their performance guide basically tells us that they disable RT for nanite meshes with world position offset by default, discourage the use of nanite skeletal meshes, and to basically never use RT for big levels made with nanite ... (high instance counts/geometry overlap)