Nvidia DLSS 1 and 2 antialiasing discussion *spawn*

[Shifty Geezer]

"In addition to the DLSS capability described above, which is the standard DLSS mode, we provide a second mode, called DLSS 2X. In this case, DLSS input is rendered at the final target resolution and then combined by a larger DLSS network to produce an output image that approaches the level of the 64x super sample rendering – a result that would be impossible to achieve in real time by any traditional means. Figure 21 shows DLSS 2X mode in operation, providing image quality very close to the reference 64x super-sampled image."

So it's nVidia's dumb naming. Apologies to Alex!

 

[jlippo]

"In addition to the DLSS capability described above, which is the standard DLSS mode, we provide a second mode, called DLSS 2X. In this case, DLSS input is rendered at the final target resolution and then combined by a larger DLSS network to produce an output image that approaches the level of the 64x super sample rendering – a result that would be impossible to achieve in real time by any traditional means. Figure 21 shows DLSS 2X mode in operation, providing image quality very close to the reference 64x super-sampled image."

So it's nVidia's dumb naming. Apologies to Alex!

Dually dumb in case of 2X as it implies super/oversampling twice in its name, even there is none.

 

[Dictator]

"In addition to the DLSS capability described above, which is the standard DLSS mode, we provide a second mode, called DLSS 2X. In this case, DLSS input is rendered at the final target resolution and then combined by a larger DLSS network to produce an output image that approaches the level of the 64x super sample rendering – a result that would be impossible to achieve in real time by any traditional means. Figure 21 shows DLSS 2X mode in operation, providing image quality very close to the reference 64x super-sampled image."

So it's nVidia's dumb naming. Apologies to Alex!

Hah, no problem of course. Believe me, I also think the name is incredibly silly, just as bad as the SS part of DLSS.

DSR is also painful when you think about it, what is DYNAMIC there? huh?

I recommend anyone here that is interested check out the "
Truly Next-Gen: Adding Deep Learning to Games & Graphics (Presented by NVIDIA)"
presentation from GDC. It is available in video form on the website for free wiht email submittance. They actually go over a lof the integration problems, content problems, quality problems etc. that we all here have mentioned and talked about. So they seem hyper aware and interestingly very self critical. Also some neat details about those 8X8 sampled images they feed into the network as comparisons. They are not just spatially higher resolution, but they prefer if EVERY screen element is of higher fidelity: shadows, AO, motion blur, etc. all with crazy sample counts as well. There is even the implication that they would prefer games that use ray tracing to send in path traced results of the screen in one of the slides. Also the images are trained as a sequence of a few frames for the comparison, not just single shots (so motion is accounted for). I hope the slides become available for them all...

 

[jlippo]

Hah, no problem of course. Believe me, I also think the name is incredibly silly, just as bad as the SS part of DLSS.

DSR is also painful when you think about it, what is DYNAMIC there? huh?

I recommend anyone here that is interested check out the "
Truly Next-Gen: Adding Deep Learning to Games & Graphics (Presented by NVIDIA)"
presentation from GDC. It is available in video form on the website for free wiht email submittance. They actually go over a lof the integration problems, content problems, quality problems etc. that we all here have mentioned and talked about. So they seem hyper aware and interestingly very self critical. Also some neat details about those 8X8 sampled images they feed into the network as comparisons. They are not just spatially higher resolution, but they prefer if EVERY screen element is of higher fidelity: shadows, AO, motion blur, etc. all with crazy sample counts as well. There is even the implication that they would prefer games that use ray tracing to send in path traced results of the screen in one of the slides. Also the images are trained as a sequence of a few frames for the comparison, not just single shots (so motion is accounted for). I hope the slides become available for them all...

It certainly would be interesting if DLSS would learn to fix details like failing edges of screen space reflections.

 

[Ethatron]

I found a paper mentioning the correct term:
Super-resolution reconstruction filter

 

[MfA]

It also does AA, so NN-SRAA seems the most honest name. Or NN-TSRAA in case motion is taken into account.

PS. not a fan of renaming multi-layer networks deep, marketing wank.

 

[Deleted member 2197]

Anthem RTX/DLSS PC Performance Review

When playing Anthem, we quickly found that DLSS acted differently at different resolutions, a result that we didn't expect. At 1440p, we found that images appeared as if they were artificially sharpened, whereas 4K images were cleaner and less grainy as a whole. We already knew that DLSS was restricted to specific resolutions on specific graphics hardware, but now it looks like DLSS has some underlying settings that are impacting image quality.

If you look at pages two and three of this analysis, you will see a stark difference in the quality of DLSS at 1440p and 4K. Yes, 4K images are higher resolution, but what we are getting at here is that when DLSS is enabled that games will react differently depending on its resolution. In Anthem, it looks like a form of post-process image sharpening is happening at 1440p, resulting in some oddities in our screenshots.
...
At both 1440p and 4K, using DLSS resulted on a reduction in overall image quality, but at 4K the feature also gave us a whopping 35%+ boost in performance, both on the RTX 2080 Ti and the RTX 2060. At 4K, this performance boost is well worth the loss in quality, but at 1440p the graphical differences will make the decision to enable DLSS a lot more difficult.

Even on our 43-inch 4K gaming screen, we found it extremely difficult to spot the differences between our 4K Native and 4K DLSS screenshots, though careful inspection of these images does reveal that DLSS looks worse that native 4K renderings. That said, the performance increase offered by DLSS is well worth the slight increase in blur.

https://www.overclock3d.net/reviews/software/anthem_rtx_dlss_pc_performance_review/1

 

[jlippo]

The presentation on DLLS training form nvidia.
https://news.developer.nvidia.com/capturing-deep-learning-data-for-neural-network-training/

 

[KimB]

I was thinking about this some more recently, and trying to see how they could use a neural network to directly estimate the final pixel color and still match the previous observations.

The thing that convinced me that the neural net was just deciding how much sampling to perform was that sometimes it looked like there was no anti-aliasing at all (just rescaling). But this doesn't mesh with the statement that their training set uses 32x super-sampling: if they wanted to use DLSS to decide how much sampling to perform, they'd need to have a training set which contained multiple outputs at different sample densities.

The idea that they'd use a neural network to directly estimate the final color still boggles my mind. It doesn't seem like it should be possible, but apparently it works. And it also makes sense if you realize that it probably doesn't work very well a lot of the time. That probably explains the no-AA outputs.

My new bet is that what they're doing here is that DLSS is a two-stage process. First, it collects various inputs for each pixel (color, depth, normal, and motion) and feeds those inputs into the neural network. Then the neural network result is compared against the raw upscaled output using some heuristic for what constitutes a good anti-aliased image. If the result seems off, then it rejects the neural network result altogether and simply outputs the upscaled result.

A simple way to accomplish this would be the following:
1) Don't apply the neural network to single pixels. Apply it to groups of pixels, e.g. 8x8 groupings. Ideally, the neural network output will also be the scaled output.
2) In parallel with the neural network, compute the rescaled raw output.
3) Once you have the neural network result and the rescaled result, compare the two outputs to ensure that the neural network output "looks like" anti-aliasing. For example, the pixel color values should be between the color values for neighboring pixels in the raw output. If some pixels are outside some pre-determined bounds, throw the result out and just use the rescaled result.

My bet is that without that thresholding, DLSS would be incredibly ugly: because of how neural networks operate, you'd probably have a number of pixels in every single frame that were way off. Some edges would look great, while others might flash bright colors when the scene doesn't call for them. It probably still messes up sometimes, producing crap output that doesn't get rejected by this heuristic. This might explain some of the "muddy" appearance seen in some test images: if the output has much less detail than it should have, it'll be hard for any heuristic to detect that. There might be some information-based heuristics that could potentially detect loss of detail, but they won't be simple at all.

The reason why the ideal case is using the neural network to do the rescaling is that when the neural network produces a good result, that result will be very close to the full-resolution, 32x super-sampled image, and will potentially be much better even than performing 32x supersampling then upscaling. I don't know if they do this, but if they're using a neural net to determine final pixel colors, this would definitely provide the best image quality. This would, however, limit the possible choices of resolution upscaling. They couldn't do arbitrary ratios, as the number of pixels processed at a time is limited. They could probably only support small-integer fraction ratios (e.g. 3/2 as opposed to 71/64). And the upscaling ratio would further be mostly fixed for each game because it would require a full retraining of the learning model for every upscaling ratio. Most games would probably only support a single upscaling ratio, and they might only support a single ratio for every game. If DLSS is implemented in this way, it would also conform with nVidia's claims that the final result is close to the 32x full-resolution supersampled image.

Note that choice of resolution doesn't necessarily require full retraining of the neural network. A group of 16x16 pixels at one resolution will be identical to a group of 16x16 pixels zoomed out but at a higher resolution. It makes sense to train the model at different resolutions, but it should be able to cope okay even if the play resolution hasn't ever been used for training.

Finally, my earlier statement that the reason for the upscaling is to hide situations where the neural network fails is probably still accurate: upscaling limits the worst-case aliasing when the neural net outputs garbled nonsense.

 

[Davros]

Just noticed, Battlefield bad company 2 allows you to set a resolution higher than the native res of your monitor
eg: 3000x2000 on a 1680x1050 monitor and the image fits completely into the screen so there appears to be downscaling going on.
(1 problem is the game gets confused as to where the mouse pointer is compared to the options buttons)

 

[iroboto]

I was thinking about this some more recently, and trying to see how they could use a neural network to directly estimate the final pixel color and still match the previous observations.

The thing that convinced me that the neural net was just deciding how much sampling to perform was that sometimes it looked like there was no anti-aliasing at all (just rescaling). But this doesn't mesh with the statement that their training set uses 32x super-sampling: if they wanted to use DLSS to decide how much sampling to perform, they'd need to have a training set which contained multiple outputs at different sample densities.

The idea that they'd use a neural network to directly estimate the final color still boggles my mind. It doesn't seem like it should be possible, but apparently it works. And it also makes sense if you realize that it probably doesn't work very well a lot of the time. That probably explains the no-AA outputs.

White paper on approximately how it should work.
https://arxiv.org/pdf/1603.06078.pdf

 

[BRiT]

Obligatory XKCD: https://xkcd.com/2173/

 

[KimB]

White paper on approximately how it should work.
https://arxiv.org/pdf/1603.06078.pdf

Probably. But with some substantial differences to make it work at speed. In particular they note:

Applying our network to an aliased image (Fig. 5) replaces jagged edges by smooth ones. While it cannot be expected to reach the same performance as the 8× multi-sample anti-aliasing (MSAA) we use for our reference, which can draw from orders of magnitude of additional information, the post-processed image shows fewer disturbing artifacts.

It's not clear whether the performance drawbacks are theoretical in nature or could be reasonably-overcome via specialized hardware.

If my quick reading of the paper is accurate, it also sounds like their neural networks used the image alone as input. This is both better and worse than what DLSS would be capable of. Better because the entire framebuffer is the input to the learning algorithm (whereas DLSS likely must operate on a limited number of pixels at a time). Worse because DLSS is capable of using more than just color data to inform the learning model (information which they do indeed use according to their public documentation).

 

[snarfbot]

Yea i kinda prefer dlss for vegetation, techspot did a nice comparison.

Dlss makes those branches look smooth if a bit oil painty as they said.

https://www.techspot.com/article/1873-radeon-image-sharpening-vs-nvidia-dlss/

 

[Panino Manino]

Yea i kinda prefer dlss for vegetation.

It's like over-processed photos from Samsung smartphones.

 

[Deleted member 2197]

The rock on lower left side of cart seems slightly blurred in .7x sharpened. Not evident in the native or dlss shots.

Edit: Will be interesting to see how much CAS and DLSS algorithms can be tweaked going forward.

 

[Shifty Geezer]

There's some motion blur/distortion thing happening in that bottom corner. Not a like-for-like comparison there, though the rest is good.

 

[Ike Turner]

The rock on lower left side of cart seems slightly blurred in .7x sharpened. Not evident in the native or dlss shots.

Edit: Will be interesting to see how much CAS and DLSS algorithms can be tweaked going forward.

In-game motion blur. The 3 stitched video captures are not perfectly synced (the train bumps up/down which causes this):
Time stamped link:

No blur:

 

[snarfbot]

It's like over-processed photos from Samsung smartphones.

Lol yea kinda i think in motion though it would be less noticeable. It looks almost exactly like brz upscaling from 2d games, emu, gzdoom and such.

 

[Deleted member 2197]

Monster Hunter DLSS implementation on July 17.
July 13, 2019

Nvidia alleges that with the addition of DLSS, the previously unobtainable 60 FPS at 4K, with max settings and the high resolution texture pack enabled, is now possible. Nvidia adds that 30 FPS at 4K is more feasible for owners of the RTX 2070 and RTX 2060, thanks to DLSS.
...
Monster Hunter: World is also getting another DLSS feature with a new sharpness slider. As the name implies, the slider will allow users to adjust the sharpness or softness of images at their discretion.
...
While Nvidia claims DLSS implementation will net gamers a free performance boost, we've noted the upscaling technique can be hit or miss depending on the game.

https://www.techspot.com/news/80937-nvidia-claims-50-percent-framerate-uplift-monster-hunter.html