Nvidia DLSS 3 antialiasing discussion

[dorf]

Slight clarification to my previous post, cant edit (sigh)

The generated frame polls input from neighboring frames though, so it should have input from the latest actual rendered frame

The generated frame should draw half of it's input from the latest rendered frame.

So in my example scenario, in frame 1.5 the camera has panned 0.5x amount to the right. Half frame added input latency.

 

[BitByte]

Honestly one of the worst and least charitable set of posts I've seen on this forum in a while. I wrote up a bit of an explanation based on the potential benefits and drawbacks. I'd actually amend this into four potential positives and three negatives. Honestly some of the positives are all but guaranteed. The three negatives we don't really know yet. It seems obvious there will be compromises to image quality and input latency, but until we see it in action the subjective difference is not known.

On top of that there's frame pacing. It could actually really help with 1% lows and make games seem incredibly smooth. The higher you push fps the more you tend to get deviation in frame times, especially if you have slow RAM. If this generation works well, you could actually have very good frametimes for half you frames, and then you'll have much higher 1% lows. But it could also potentially magnify dips, or have other issues that lead to some odd frame pacing that causes more stutter. We don't know yet.

Potential positives:
judder reduction
animation smoothness
motion blur reduction
cpu, memory, gpu scaling
frame pacing

Potential negatives:
input latency
image quality (artifacts etc)
frame pacing





I'd recommend going to blur busters to understand motion picture response time (MPRT), the benefits of 1000Hz displays and the potential benefits of frame amplification. Also the test ufo pattern is worth a look.

Blur Busters Law: The Amazing Journey To Future 1000Hz Displays

Strobeless ULMB. Blurless sample-and-hold. Blur-free with no flicker or strobing. Lagless. Coming within your lifetime.

blurbusters.com

Frame Rate Amplification Tech (FRAT) — More Frame Rate With Better Graphics

Tomorrow, 1000fps for free. Not your grandfather's Classic Interpolation. Oculus has Asynchronous Time Warp. Cambrige has Temporal Resolution Multiplexing. NVIDIA has Deep Learning Super Sampling. Have cake and eat it too with high-detail high-framerates!

blurbusters.com

GtG versus MPRT: Frequently Asked Questions About Pixel Response On Displays

GtG and MPRT are two different pixel response benchmarks for displays. We explain the difference.

blurbusters.com

Blur Busters TestUFO Motion Tests. Benchmark for monitors & displays.

Blur Busters UFO Motion Tests with ghosting test, 30fps vs 60fps vs 120hz vs 144hz vs 240hz, PWM test, motion blur test, judder test, benchmarks, and more.

www.testufo.com

All the positives you wrote are completely irrelevant as long as those frames have nothing to do with the game logic. I’ve said it and I’ll say it again, it’s fake frames. You can interpolate all you want to make it “look” better and have more frames but it’s not real data. I already avoid using DLSS because of its noticeable image degradation, ghosting and artifacts. This is taking a bad “feature” and making it worse.

 

[BitByte]

The fraction of samples generated with a traditional rendering pipeline will continue to decrease.

If the current rate of progress of neural rendering continues unabated for a few more years, I guarantee you that fraction will eventually go to zero.

What does this even mean? A portion of the frame will always be required otherwise, you’d be generating the frames from no base data. That doesn’t even make any sense.

 

[Scott_Arm]

All the positives you wrote are completely irrelevant as long as those frames have nothing to do with the game logic. I’ve said it and I’ll say it again, it’s fake frames. You can interpolate all you want to make it “look” better and have more frames but it’s not real data. I already avoid using DLSS because of its noticeable image degradation. This is taking a bad “feature” and making it worse.

What are you talking about? When you enable motion smoothing on a tv do you not reduce judder, smooth motion and decrease persistence? That’s what it’s designed to do and it works.

What Nvidia is doing is exactly the same intent but for games. With the exception of games that have hand crafted animations, like most 2D games, this type of motion clarity is exactly what you’d want.

Also, what’s fake data? If an animation system blends or interpolates between key frames is that “fake” too?

 

[nAo]

What does this even mean? A portion of the frame will always be required otherwise, you’d be generating the frames from no base data. That doesn’t even make any sense.

There is much more to neural rendering than reconstruction of images starting from data rendered with traditional methods.
New neural scene representations and neural rendering methods are released every week. We're in the middle of a revolution.

How many "traditional" triangles and textures have been used to create and render these images? (hint: zero)

 

[Scott_Arm]

What does this even mean? A portion of the frame will always be required otherwise, you’d be generating the frames from no base data. That doesn’t even make any sense.

He’s not going to drop credentials but I’m pretty sure @nAo was/is a rendering programmer that worked on games. Just search through rendering people on Twitter and you’ll see them posting about all kinds of research regarding neural networks in rendering.

 

[PSman1700]

I don’t care about either company

Consoles also use forms of ’cheating’ like checkerboarding, taa etc.

 

[Putas]

If I had to guess tvs can also spend a lot more time processing video because the watcher is not providing any inputs. They can just delay the audio to match the frames. Pretty sure lots of receivers have the option to add audio delay for this specific reason.

Since basic Bluetooth audio became popular adding delay to video is a must. Syncing has to be done for one reason or another and there is approximately half of an eternity of time to interpolate.

 

[DavidGraham]

I already avoid using DLSS because of its noticeable image degradation, ghosting and artifacts. This is taking a bad “feature” and making it worse.

This is where I know you are in fact LYING to yourself, TAA image contain the same amount of artifacts that you describe, often more, yet I don't hear you complain about them.
Rendering is full of shortcuts, there are so many half resolution, quarter resolution, and 1/8 effects in any given scene, you can't even count them anymore, you consider those fake too?

 

[Kaotik]

Yah, I don't get this. As far as I know a lot of games decouple input from rendering. Games like Overwatch have decoupled mouse input from rendering so you can aim between frames.

New Feature – High Precision Mouse Input (Gameplay Option)

Hi everyone! I’m Derek, a gameplay engineer on Overwatch and wanted to run through this new feature in version 1.42, and exactly what it does and how it works. First, a quick summary on how aiming and shooting happens in Overwatch. Overwatch simulates (or ticks) every 16 milliseconds, or at...

us.forums.blizzard.com

I really don't see how input would be affected by frame generation besides some additional display lag.

Because GPU does it all alone, no matter if the inputs are decoupled or not they don't affect generated frames

 

[Lurkmass]

Slight clarification to my previous post, cant edit (sigh) The generated frame should draw half of it's input from the latest rendered frame. So in my example scenario, in frame 1.5 the camera has panned 0.5x amount to the right. Half frame added input latency.

No user input polling happens during frame interpolation because the application won't read the player's controls ...

What would be the point in reading controls for your past frames ? Hence, why user input polling is irrelevant in that instance. User input won't change the way frame interpolation happens if it truly had any effect ...

 

[dorf]

No user input polling happens during frame interpolation because the application won't read the player's controls ...

Real input polling doesn't happen for that frame, but input polling for neighboring frames affects camera movements etc, no? So it's a cruder roundabout way of input polling for the generated frames I think.

Let's say user input made the camera panning happen in my example. What do you think would happen in frame 1.5?

 

[DieH@rd]

Potential negatives:
input latency
image quality (artifacts etc)
frame pacing

For eSports users who might want to achieve the lowest latency, will it be possible to only enable NVIDIA DLSS 2 (Super Resolution) + Reflex without the DLSS Frame Generation that improves FPS but also increases system latency?

Our guidelines for DLSS 3 encourage developers to expose both a master DLSS On/Off switch, as well as individual feature toggles for Frame Generation, Super Resolution, and Reflex, so users have control and can configure the exact settings they prefer. Note that while DLSS Frame Generation can add roughly a half frame of latency, this latency is often mitigated by NVIDIA Reflex and DLSS Super Resolution - all part of the DLSS 3 package. NVIDIA Reflex reduces latency by synchronizing CPU and GPU, and DLSS Super Resolution reduces latency by decreasing the size of the render resolution.

NVIDIA Ada Lovelace Follow-Up Q&A - DLSS 3, SER, OMM, DMM and More

We followed up with NVIDIA for a brief Q&A on the newly announced DLSS 3, SER, OMM, and DMM features available with the new RTX 4000 GPUs.

wccftech.com

It looks like latency will not be an issue. And having reflex built into every DLSS3 title will "standardize" input latency reduction in PC space, it will become commonplace.
As for frame generation, I wonder how will the DLLS3 system conjur a brand new parts of gameworld during camera rotation? Will it be visible?

 

[iroboto]

All the positives you wrote are completely irrelevant as long as those frames have nothing to do with the game logic. I’ve said it and I’ll say it again, it’s fake frames. You can interpolate all you want to make it “look” better and have more frames but it’s not real data.

I mean, that's like saying DLSS upscale resolution is pointless, because it's not real data. Frame rate is an equivalence of motion resolution. The higher the frame rate, the sharper things are in motion. I mean, don't see how having a higher frame rate is worse than having a lower frame rate. It's not relevant that the frames are fake, it's only relevant that the frames assist in helping players have better motion resolution which plays a large part in visual clarity, identification and synchronization for gameplay.

 

[Lurkmass]

Real input polling doesn't happen for that frame, but input polling for neighboring frames affects camera movements etc, no? So it's a cruder roundabout way of input polling for the generated frames I think.

Let's say user input made the camera panning happen in my example. What do you think would happen in frame 1.5?

Think of it like this. you'll be able to move your camera on every frame that's not interpolated but you can't control your camera for interpolated frames ...

An important characteristic behind interactive media is that they give response or feedback to the user. Interpolated frames lack this special characteristic. I think the best description for DLSS 3 is that it's creates non-interactive single frame long cutscenes between interactive frames. This is why input latency is a full frame rather than half a frame in length since our application is only truly interactive for half of all times in this case ...

 

[iroboto]

What does this even mean? A portion of the frame will always be required otherwise, you’d be generating the frames from no base data. That doesn’t even make any sense.

NN can literally render whole images from foveated rendering. The challenge right now is just doing it fast enough and good enough. Once it hits this point, combined with frame generation, nAo will be correct. It will be nearly 97% all generated. We will eventually get to this point, at least with VR.

 

[trinibwoy]

For eSports users who might want to achieve the lowest latency, will it be possible to only enable NVIDIA DLSS 2 (Super Resolution) + Reflex without the DLSS Frame Generation that improves FPS but also increases system latency?

Our guidelines for DLSS 3 encourage developers to expose both a master DLSS On/Off switch, as well as individual feature toggles for Frame Generation, Super Resolution, and Reflex, so users have control and can configure the exact settings they prefer. Note that while DLSS Frame Generation can add roughly a half frame of latency, this latency is often mitigated by NVIDIA Reflex and DLSS Super Resolution - all part of the DLSS 3 package. NVIDIA Reflex reduces latency by synchronizing CPU and GPU, and DLSS Super Resolution reduces latency by decreasing the size of the render resolution.

NVIDIA Ada Lovelace Follow-Up Q&A - DLSS 3, SER, OMM, DMM and More

We followed up with NVIDIA for a brief Q&A on the newly announced DLSS 3, SER, OMM, and DMM features available with the new RTX 4000 GPUs.

wccftech.com

It looks like latency will not be an issue. And having reflex built into every DLSS3 title will "standardize" input latency reduction in PC space, it will become commonplace.
As for frame generation, I wonder how will the DLLS3 system conjur a brand new parts of gameworld during camera rotation? Will it be visible?

It doesn’t need to conjure anything. The game reads your input and the CPU updates the camera, animations, physics etc. The GPU renders the frame. This new frame is input to the OFA, frame generation process. There’s no guessing about what happens “next”. The neural network guessing is all about what happened between the last two rendered frames.

Btw we don’t need 1000fps displays and any crazy AI to interpolate those extra fps. We just need display tech that doesn’t rely on sample and hold. Hopefully micro led is bright enough to do just that. Our eyes are the ultimate interpolators given appropriately clean input.

 

[pharma]

All the positives you wrote are completely irrelevant as long as those frames have nothing to do with the game logic. I’ve said it and I’ll say it again, it’s fake frames. You can interpolate all you want to make it “look” better and have more frames but it’s not real data. I already avoid using DLSS because of its noticeable image degradation, ghosting and artifacts. This is taking a bad “feature” and making it worse.

I think people have you pegged correctly. Why on earth would you think FSR 2 is "good enough" and DLSS 3 not?

I think it's good enough for the masses which is all it needs to be. If most people deem it as good enough to use and they use it, then Nvidia's got a problem on their hand. It'll just be like G-Sync vs Freesync. G-sync was better but most people purchased free sync monitors because it was good enough and much cheaper.

 

[dorf]

This is why input latency is a full frame rather than half a frame in length since our application is only truly interactive for half of all times in this case ...

I don't it's accurate to think of it as full frame latency.

If I move my mouse to the right between frames 1 and 2, I will already see the camera panning to the right in generated frame 1.5 since the movement from 1 -> 2 is affecting that generated frame.

If input latency was truly one whole frame long as you suggest, the first frame where I'd be seeing the camera pan to the right (as per my input), would be when frame 2 hits my screen.

 

[troyan]

All the positives you wrote are completely irrelevant as long as those frames have nothing to do with the game logic. I’ve said it and I’ll say it again, it’s fake frames. You can interpolate all you want to make it “look” better and have more frames but it’s not real data. I already avoid using DLSS because of its noticeable image degradation, ghosting and artifacts. This is taking a bad “feature” and making it worse.

Do you know that input latency is decoupled from render time? Which makes most frames "fake".