CES 2025 Thread (AMD, Intel, Nvidia, and others!)

[iroboto]

Well, then I hope we'll get SSIM graphs vs ground truth rendering soon

Eeehhhhhh. With the amount of settings that we can tweak, without developer assistance, I don’t think any 3rd party tool would work. I tried to make something like this for DF but in the end artifacts ended up sort of came in as detail.

Any thoughts on how to do it would be appreciated though.

 

[Ethatron]

Eeehhhhhh. With the amount of settings that we can tweak, without developer assistance, I don’t think any 3rd party tool would work. I tried to make something like this for DF but in the end artifacts ended up sort of came in as detail.

Any thoughts on how to do it would be appreciated though.

It's not up to you to provide this. The vendor selling halucinatory rendering should provide the objective metrics for the "quality" level they are offering. Maybe we'll even get SSIM per Watt as distinguishing metric between vendors some day.

 

[iroboto]

It's not up to you to provide this. The vendor selling halucinatory rendering should provide the objective metrics for the "quality" level they are offering. Maybe we'll even get SSIM per Watt as distinguishing metric between vendors some day.

But it still seems to be on a per title basis? SSIM works well when you have source truth, like a digital movie, and testing various upscalers. But What is ground truth in this case? TAA? SSAA? Various AA algorithms can preserve more data than TAA, so it gets a bit weird to know.

We would need developers to build a demo which they designate as ground truth where settings are locked like 3D mark, then run the SSIM algorithm against the profile.

Maybe 3D mark can do it, but that would limit it to a single test.

I mean, I suppose if you link the SSIM test into the rendering path, but I can’t help but think this is going to affect frame rate lol. I just don’t know how this could be done.

 

[Ethatron]

But it still seems to be on a per title basis? SSIM works well when you have source truth, like a digital movie, and testing various upscalers. But What is ground truth in this case? TAA? SSAA? Various AA algorithms can preserve more data than TAA, so it gets a bit weird to know.

We would need developers to build a demo which they designate as ground truth where settings are locked like 3D mark, then run the SSIM algorithm against the profile.

Maybe 3D mark can do it, but that would limit it to a single test.

Why would it be on a per-title basis? When a vendor throws around 33db PNSR for S3TC (DXCT, BCT) and 35db for ETC2 then this is not per title, it's on average on a test corpus. For the approximate rendering case you also would have a corpus (in the form of a program) and averages. The program is deterministic and parameterizable, so obviously you have a ground truth output available. The ground truth a vendor compares to is what the vendor claims to approximate. A considerable amount of the techniques deal with sampling frequencies in space and time, and this is really easy to match and check.
Obviously the corpus would be more than just Cornell box, but many diverse examples. It'd be very interesting to see how Hades II would fare under DLSS4+DG for example. Because, obviously, you put the big problem cases in there. Then you would be able to understand if the solution stepped over the threshold of perceivability in general, or in which situation or for which elements.

Many game developers in the past have been very diligent, soft shadow techniques are compared against ground truth, motion blur, GTAO ofc (like other AO techniques) etc. pp. Same for raytracing techniques (in the academic field), and so on and so forth. I think Nvidia can easily afford spending a billion dollars in "proofing"/showcasing how good their proposal is outside of feel good vibes, and maybe inform the part of the industry that may not need ground truth but can't tolerate hallucinations (similar to the medical industry and MRI).

 

[neckthrough]

But it still seems to be on a per title basis? SSIM works well when you have source truth, like a digital movie, and testing various upscalers. But What is ground truth in this case? TAA? SSAA? Various AA algorithms can preserve more data than TAA, so it gets a bit weird to know.

Re ground truth -- yeah a 64x SSAA for spatial antialiasing + maybe 4x to 8x frame accumulation for temporal stability?

I'm not sure how good SSIM is for discerning temporal instability. It was designed to evaluate video compression, and highly compressed videos are actually quite temporally stable aren't they? Because video codecs are based on motion + corrective deltas.

 

[iroboto]

Why would it be on a per-title basis? When a vendor throws around 33db PNSR for S3TC (DXCT, BCT) and 35db for ETC2 then this is not per title, it's on average on a test corpus. For the approximate rendering case you also would have a corpus (in the form of a program) and averages. The program is deterministic and parameterizable, so obviously you have a ground truth output available. The ground truth a vendor compares to is what the vendor claims to approximate. A considerable amount of the techniques deal with sampling frequencies in space and time, and this is really easy to match and check.
Obviously the corpus would be more than just Cornell box, but many diverse examples. It'd be very interesting to see how Hades II would fare under DLSS4+DG for example. Because, obviously, you put the big problem cases in there. Then you would be able to understand if the solution stepped over the threshold of perceivability in general, or in which situation or for which elements.

Many game developers in the past have been very diligent, soft shadow techniques are compared against ground truth, motion blur, GTAO ofc (like other AO techniques) etc. pp. Same for raytracing techniques (in the academic field), and so on and so forth. I think Nvidia can easily afford spending a billion dollars in "proofing"/showcasing how good their proposal is outside of feel good vibes, and maybe inform the part of the industry that may not need ground truth but can't tolerate hallucinations (similar to the medical industry and MRI).

I’m not sure how to do what you’re suggesting unfortunately. My understanding of signal processing isn’t quite there. I can’t do SNR without some sort of ground truth. That seems easier? For a compression algorithms. Because we know the ground truth in this case is uncompressed. Defining what should be ground truth would be interesting to say the least now. Especially with temporal variables now added into the rendering pipeline and motion blur being an option. There are a lot of scenarios where these algorithms could be really strong at, and others very weak at.

It does sound like we need a 3rd party test suite of some sort that specifically is looking to check the value of the hallucination.

I was often under the assumption that ground truth for developers was what could be pre-rendered in engine versus what could be achieved in realtime.

This is as far as I was able to get. I think you can sort of see why we didn’t pursue this any further.