Variable Rate Shading vs. Variable Rate Rasterization

[rikrak]

Could any of you knowledgeable people offer a more detailed explanation of the difference between Variable Rate Shading (as used in DX12 and Vulkan) and Variable Rate Rasterization (as used by Apple in Metal)? If I understand it correctly, VRS simply allows the fragment/pixel shader to run at a lower resolution than the rasterized image (it is not 100% clear to me how the output of the shader is mapped to the final image — is there some filtering going on, or are values just copied?). In contrast, Apple's VRR actually rasterizes at a lower rate, producing a distorted image, and then requires you to manually rescale it to the target resolution by sampling data from these distorted low-res tiles. At the same time, with VRS each cell can be set to a different shading rate, while Apple's VRR needs to preserve a certain kind of symmetry (you can only set rasterization rates for rows and columns, not for the cells themselves).

Is there anything important to add? What are the benefits and disadvantages of the respective approach?

 

[3dcgi]

VRS replicates the color values to the pixels.

 

[BRiT]

Devs have indicated you can increase resolution using VRS. I don't recall exactly where that was, but it was mentioned on these forums a couple times now.

 

[JoeJ]

At the same time, with VRS each cell can be set to a different shading rate, while Apple's VRR needs to preserve a certain kind of symmetry (you can only set rasterization rates for rows and columns, not for the cells themselves).

I can only guess, but this sounds Apple has foveated rendering in mind? One could decrease rendering resolution towards the edges of the image. (I assume the tile either has mixed horizontal / vertical resolution given from row and column, or a min/max of both.)
If true, they either simply missed the advantage of VRS of being adaptive to the image content as well, because they have not thought about that option. But that's unlikely, so it's probably about a HW limitation (which again sounds more like a missed opportunity).

Devs have indicated you can increase resolution using VRS. I don't recall exactly where that was, but it was mentioned on these forums a couple times now

Maybe what they meant was the idea to emulate MSAA using VRS, but with adaptive AA quality per tile. Tiles could have 4x4, 2x2, or 1 samples per pixel. Even 4x2, e.g. if the aliasing edges in given tile were mostly horizontal in the previous frame. This would work when rendering to higher res and then down scaling to screen resolution.

 

[Lurkmass]

As we can see from the slide provided by Nvidia with variable rate shading in D3D12/Vulkan we rasterize our image at full resolution and run our pixel shader invocations at varying rates. Further details laid out in the D3D functional specs ... (the shading rates can be specified per draw call, texture, or per-primitive and we have combiners merge all these different sources)

On Metal we have rasterization rate maps. In this model, the zones in our image is then rasterized at different rates for each zones. We specify these rates via 2 1-D arrays with floating point values between 1.0 and 0.0 each 1-D array corresponds to the different axis in the image. Comparatively speaking, Metal's variable rate rasterization feature has lot of similarities to Nvidia multi-res shading technology but we can have more than 9 zones ...

 

[JoeJ]

We specify these rates via 2 1-D arrays with floating point values between 1.0 and 0.0 each 1-D array corresponds to the different axis in the image.

Now i got confused because that's the same limitation i expected from Metal.
It turned out setting rate per tile individually is available only in Tier 2: https://microsoft.github.io/DirectX-Specs/d3d/VariableRateShading.html ('Screenspace image-based rate selection')
(My AA example was bad too, because edges are always full res)

 

[Lurkmass]

Now i got confused because that's the same limitation i expected from Metal.
It turned out setting rate per tile individually is available only in Tier 2: https://microsoft.github.io/DirectX-Specs/d3d/VariableRateShading.html ('Screenspace image-based rate selection')
(My AA example was bad too, because edges are always full res)

In the above picture which demonstrates Metal's rasterization rate maps, our 1-D arrays will contain information about each axis' rasterization quality and in this specific example both the horizontal an vertical axis are split into 5 bins each. The value 1.0 will represent full rate rasterization while 0.0 will represent minimum rasterization quality and the values in between those will be of varying quality.

I think D3D12/Vulkan's variable rate shading extension and Metal's rasterization rate map functionality are incompatible with each other. With variable rate shading, we always rasterize our image at full rate while shading rates can vary. In variable rate rasterization, native resolution rasterization doesn't happen so the results aren't all that comparable against variable rate shading.

With Tier 1 VRS and rasterization rate maps, there is little in commonality between both. In D3D12/Vulkan, we specify the shading rates via D3D12_SHADING_RATE enum and then issue our draw calls which are then shaded at the rate we defined. If we wanted to change to a different shading rate for other draws we'll need to call the RSSetShadingRate method again and we can then start issuing draw calls again for this new shading rate. There are no comparable mechanism on Metal to do this.

With Tier 2 VRS, we now have the ability to specify shading rates with a texture or we can do it per-primitive (SV_ShadingRate). When we create our VRS texture, it will contain VRS tiles with different shading rates. In this scenario we can map Metal's rasterization rate maps on top of our VRS texture but the same isn't true for the other way around.

 

[rikrak]

Thank's, this really clears things up. So in retrospect, VRS's main purpose is to improve the utilization of shader resources as it allows you to adaptively and independently set shading rate for individual portions of the render target. It does not affect rasterization or the layout of the render target in any way. Metal's VRR in contrast applies at the rasterization stage — not at shading stage — and it "shrinks/streches" the render target space (I don't know how to call the respective mathematical transformation). VRR cannot be used to apply the same level of fine-grained LOD control as VRS can, but it can achieve a smoother change in shading rate and a higher "dynamic range" of shading rates.

I would actually speculate that Apple has developed VRR with VR in mind — this kind of render target transformation seems exactly what one needs when thinking about VR googles. Imagine combining eye-tracking with VRR, one could dedicate a large portion of the render target area to where the attention is pointed at, only crudely rasterizing the rest. One could probably achieve massive bandwidth savings here with very little loss of quality (or maybe even a net increase in quality as one can "oversample" the attention-focused area).

 

[Lurkmass]

Thank's, this really clears things up. So in retrospect, VRS's main purpose is to improve the utilization of shader resources as it allows you to adaptively and independently set shading rate for individual portions of the render target. It does not affect rasterization or the layout of the render target in any way. Metal's VRR in contrast applies at the rasterization stage — not at shading stage — and it "shrinks/streches" the render target space (I don't know how to call the respective mathematical transformation). VRR cannot be used to apply the same level of fine-grained LOD control as VRS can, but it can achieve a smoother change in shading rate and a higher "dynamic range" of shading rates.

I would actually speculate that Apple has developed VRR with VR in mind — this kind of render target transformation seems exactly what one needs when thinking about VR googles. Imagine combining eye-tracking with VRR, one could dedicate a large portion of the render target area to where the attention is pointed at, only crudely rasterizing the rest. One could probably achieve massive bandwidth savings here with very little loss of quality (or maybe even a net increase in quality as one can "oversample" the attention-focused area).

When I looked at Metal's rasterization rate maps, I indeed immediately thought that foveated rendering was an obvious application for it.

VR headsets often project 'warped' images so our naively rendered imaged can often be oversampled at the edges and undersampled at the center. There's a lot of research published by Nvidia about this if we take a look at slide page 33 in their results. Their first solution to this problem was introducing multi-resolution shading with 2nd gen Maxwell HW which was both an improvement in quality and performance. Nvidia's Pascal architecture improved on performance even further when using lens matched shading for foveated rendering.

Variable rate shading is useful for rasterization in general including stereoscopic or foveated rendering.

 

[rikrak]

When I looked at Metal's rasterization rate maps, I indeed immediately thought that foveated rendering was an obvious application for it.

VR headsets often project 'warped' images so our naively rendered imaged can often be oversampled at the edges and undersampled at the center. There's a lot of research published by Nvidia about this if we take a look at slide page 33 in their results. Their first solution to this problem was introducing multi-resolution shading with 2nd gen Maxwell HW which was both an improvement in quality and performance. Nvidia's Pascal architecture improved on performance even further when using lens matched shading for foveated rendering.

Variable rate shading is useful for rasterization in general including stereoscopic or foveated rendering.

Thanks, I have learned something today!