Nvidia DLSS 3 antialiasing discussion

[nAo]

Kaotik: there is no frame extrapolation, which would be equivalent to predict the future.
I've heard that's a pretty hard problem to solve

Just listen to Jensen's words (the video starts playing right before the quote below):

DLSS 3 processes the new frame and the prior frame to discover how the scene is changing.
The optical flow accelerator provides the neural network with the direction and velocity of pixels from frame to frame.
Pairs of frames from the game, along with the geometry and pixel motion vectors, are then fed into a neural network, that generates the intermediate frame.

.. and the DLSS 3 blog post also confirms the same notion:

For each pixel, the DLSS Frame Generation AI network decides how to use information from the game motion vectors, the optical flow field, and the sequential game frames to create intermediate frames.

Introducing NVIDIA DLSS 3

Multiply frame rates by up to 4X!

www.nvidia.com

 

[Scott_Arm]

I think I’m going to wait for digital foundry video before I start making any sweeping claims about how good or bad the technology is.

 

[hughJ]

So is the optical flow system NN-based and trained only for full screen, screen-based synthesis? I'm curious if you could use it for temporal upscaling render-to-texture assets like planar reflections, cube maps (env maps, light probes, etc), shadow maps, radiosity maps, and so on. Many of those assets tend to get shortchanged and exhibit artifacts in order to be performant -- low spatial sampling (resolution, scene LOD), or reduced temporal sampling (updating at lower frame rates), so the bar would be pretty low for their optical flow system, assuming it would have negligible cost.

 

[Malo]

Pretty sure it’s after. Don’t need ML for prior frames.

An intermediate frame inserted after would make no sense. The changes between frames it's generating in the NN exist between the prior and current frame so adding it after the current would result in a 2 steps forward 1 step back Judder.

 

[Scott_Arm]

Another thing I'll mention. Typically gamers that want "best performance" would disable vsync and uncap their framerate to get max fps. Battlenonsense was the first person I saw that suggested this isn't always the best strategy and could increase latency if you let your gpu hit close to 100% utilization. Reflex was created for this particular scenario, to prevent the cpu from queuing frames that the gpu wasn't ready to render.

The scenario we have with DLSS3 is that it allows you to increase fps, but without the benefit of latency reduction. So going from 30 to 60 fps no longer reduces input lag by (16.6 ms), and 60 to 120 fps no longer reduces input lag by 8.3ms. You stay at the input lag of the lower fps, with maybe some additional latency but get improvements to visual smoothness and blur reduction. You also basically "halve" your cpu, RAM requirements because they have to be involved in half as many frames.

If you have a 60Hz monitor, the benefits, to be honest, are going to be very minimal. The scenario of uncapping your frames to get the best input lag won't work, because you're only getting the input lag of half the drawn frames. On top of that you don't get the benefits of the motion smoothness and blur reduction because most of what you're drawing is outside the refresh rate of your display. The only 60Hz scenario that makes sense is if you can't hit 60 native with your max settings, or whatever it is you want, and this is likely the worst case scenario for DLSS3 with such a low starting framerate, especially if the frame to frame differences are large.

Best case scenario is for 120Hz - 500 Hz displays with gsync/freesync, where every generated intermediate frame can be displayed, because you can get the benefit of all of the generated frames and hit those high refresh rates with reduced cpu, RAM requirements.

If you're the type of person to buy a 4080/4090 to use with a 60Hz display, don't be that type of person. Move on and get an actually good gaming display with at least 120Hz refresh rate. The memes about not being able to see past 60Hz are not true. Join the future.

 

[BRiT]

An intermediate frame inserted after would make no sense. The changes between frames it's generating in the NN exist between the prior and current frame so adding it after the current would result in a 2 steps forward 1 step back Judder.

It also seems nonsensical to me the other option:

Given: Frame 1 is displayed
Given: Frame 2 is completed but not displayed

1. Do not display Frame 2
2. Interpolate to create Frame 1.5
3. Display Frame 1.5
4. Delay
5. Display Frame 2

 

[Malo]

Given: Frame 1 is displayed
Given: Frame 2 is completed but not displayed

I thought neither frame is displayed yet?

 

[iroboto]

An intermediate frame inserted after would make no sense. The changes between frames it's generating in the NN exist between the prior and current frame so adding it after the current would result in a 2 steps forward 1 step back Judder.

so effectively you have rendered frames and then you have hallucinated frames.
If your render frames which are generated by the CPU are: 1, 2, 3, 4, 5, 6, 7, respectively
Your hallucinated frames which are entirely GPU driven are A, B, C, D, E, F respectively
1 + 2 = A
2 + 3 = B
3 + 4 = C
4 + 5 = D
5 + 6 = E
and so forth
The actual ordering of frames then becomes
1, 2, A, 3, B, 4, C, 5, D, 6, E ,7... and the pattern continues
This effectively inserts a frame inbetween the CPU frames (by going ahead one), doubling the frame time. The accuracy of that frame (thus artifacts) will really depend on the quality of the model, but at high enough speeds you're going to be seeing an interplay between a real rendered frame and a hallucinated one. If it's fast enough, you may not notice. But on a freeze frame, it should be obvious which ones are hallucinated.

 

[Scott_Arm]

An intermediate frame inserted after would make no sense. The changes between frames it's generating in the NN exist between the prior and current frame so adding it after the current would result in a 2 steps forward 1 step back Judder.

Yah, it doesn't seem confusing to me. You render a frame (current), and then it finds the diff between that frame and the last rendered frame (previous) and generates (intermediate) it. It then displays the generated frame (intermediate) and then the current frame. You are then adding half a frame of latency.

I don't think I could draw it out, but my understanding would be say at 60fps you are presenting a new frame every 16.6ms, but if you use generation to double you're presenting every 8.3ms. To render the current frame you have to delay drawing the current frame by 8.3ms to show the generated one, which is half of the 16.6ms frame time you are rendering at.

 

[Scott_Arm]

@iroboto

I don't think that's how it works. Every second frame is generated. It uses that optical accelerator to generate the intermediate frames between the current and previous frames, because engine motion vectors track movements of geometry/particles, not reflections or shadows. The frame generation then creates the intermediate frame, and displays the intermediate frame before the current one.

Introducing NVIDIA DLSS 3

Multiply frame rates by up to 4X!

www.nvidia.com

 

[BRiT]

The actual ordering of frames then becomes
1, 2, A, 3, B, 4, C, 5, D, 6, E ,7... and the pattern continues

That seems the absolute worst order to display the frames. As Malo said, 2 steps forward and 1 step back.

Given what Scott_Arm posted, it really does seem like they are delaying frames from when they would originally be displayed to insert the intermediate frames. If that's the situation it seems to me the pattern would be:

1, A (generated), 2 (delayed), B (generated), 3 (delayed), C (generated), 4 (delayed), D (generated), 5 (delayed), E (generated), etc...

 

[trinibwoy]

so effectively you have rendered frames and then you have hallucinated frames.
If your render frames which are generated by the CPU are: 1, 2, 3, 4, 5, 6, 7, respectively
Your hallucinated frames which are entirely GPU driven are A, B, C, D, E, F respectively
1 + 2 = A
2 + 3 = B
3 + 4 = C
4 + 5 = D
5 + 6 = E
and so forth
The actual ordering of frames then becomes
1, 2, A, 3, B, 4, C, 5, D, 6, E ,7... and the pattern continues
This effectively inserts a frame inbetween the CPU frames (by going ahead one), doubling the frame time. The accuracy of that frame (thus artifacts) will really depend on the quality of the model, but at high enough speeds you're going to be seeing an interplay between a real rendered frame and a hallucinated one. If it's fast enough, you may not notice. But on a freeze frame, it should be obvious which ones are hallucinated.

That would be prediction and actually be somewhat useful. However this does not match what Nvidia described. They are generating frames that represent the “transition between” the input frames. What you described would be forward projection of animation after the second frame which isn’t what’s happening.

It makes perfect sense why they’re not doing this. Frame A and frame 3 would have all sorts of crazy differences.

 

[Scott_Arm]

This is how I understand it to work. It uses the current and previous frame to generate an intermediate frame. Your increase in input lag is the time added to the render queue to generate frames plus the delay in presenting the current frame.

 

[BRiT]

This is how I understand it to work. It uses the current and previous frame to generate an intermediate frame.

Currently, I'm thinking it works that way too.

 

[trinibwoy]

That seems the absolute worst order to display the frames. As Malo said, 2 steps forward and 1 step back.

Given what Scott_Arm posted, it really does seem like they are delaying frames from when they would originally be displayed to insert the intermediate frames. If that's the situation it seems to me the pattern would be:

1, A (generated), 2 (delayed), B (generated), 3 (delayed), C (generated), 4 (delayed), D (generated), 5 (delayed), E (generated), etc...

Yep DLSS3 adds latency and smooths motion and the combination seems completely pointless to me. The benefit is smoother motion and this benefit is most needed at lower frame rates. However lower frame rates mean higher latency and this is worst time to add even more latency. Really looking forward to the DF analysis on this. Not sure what problem this is solving.

 

[iroboto]

@iroboto

I don't think that's how it works. Every second frame is generated. It uses that optical accelerator to generate the intermediate frames between the current and previous frames, because engine motion vectors track movements of geometry/particles, not reflections or shadows. The frame generation then creates the intermediate frame, and displays the intermediate frame before the current one.

View attachment 7044

Introducing NVIDIA DLSS 3

Multiply frame rates by up to 4X!

www.nvidia.com

I'm not sure I understand here.
In Frame 1 it's using super resolution to complete frame one, then it immediately builds frame 2 using purely DL. Frame 3 is then rendered and upscaled, and Frame 4 is made purely by DL.
I'm not seeing anything in this picture that would describe that Frame 1 and 3 need to be rendered before DL is able to generate frame 2.
In order to have a frame rate increase, you technically need to push that frame to the screen, otherwise you're just interpolating.

ah i see. the issue. They say they generate an intermediate frame using frame 1 and frame 2 as inputs. So the assumption here is that it's generating a frame in-between 1 and 2. And I'm looking at it as saying it's using Frame 1 and Frame 2 to generate an intermediate frame after 2 to 3, before 3 technically arrives.

 

[DegustatoR]

Yep DLSS3 adds latency and smooths motion and the combination seems completely pointless to me. The benefit is smoother motion and this benefit is most needed at lower frame rates. However lower frame rates mean higher latency and this is worst time to add even more latency. Really looking forward to the DF analysis on this. Not sure what problem this is solving.

Fps in CPU limited games for one.

 

[BRiT]

Not sure what problem this is solving.

Bar charts with bigger numbers.

 

[Clukos]

This is how I understand it to work. It uses the current and previous frame to generate an intermediate frame. Your increase in input lag is the time added to the render queue to generate frames plus the delay in presenting the current frame.

View attachment 7045

That’s how I understand it as well, no matter what you do you still have to pay the cost of presenting and display processing the generated frame. Nvidia forcing reflex with DLSS 3.0 might be an admission that the tech is negatively affecting latency. Very interested to see input lag coverage with it

 

[hughJ]

However lower frame rates mean higher latency and this is worst time to add even more latency. Really looking forward to the DF analysis on this. Not sure what problem this is solving.

Lower frames also mean larger deltas in information, so more artifacts and longer durations to notice those artifacts. It'll work best in super high frame rate esports titles, but those aren't the sorts of titles that people want to be sacrificing latency for smoke and mirror visual fluidity.