Neural networks for upscaling. Artomatix criticises the use of so "outdated" checkerboard rendering.

A demo video of their tech:

The tech is really interesting, but if it takes several seconds to upscale a single texture, how would they ever make it work on less than 30ms? Even if it's running on CPU cores and they optimized it for GPU cores, it would need to run a thousand times faster, and that could take way too many GPU resources.

I see how this is an interesting tech for saving on texture size (total game size), and maybe saving CPU-GPU bandwidth if it gets done inside the GPU's VRAM. It looks really cool.
But for a real-time application I wonder if having the GPU rendering at full 4K wouldn't actually be less demanding.
 
Looks more like artists tools to work with textures during game development, not in game realtime.

Edit: I see why you're wondering about working in real-time after I read the article (previously just watched video on phone). There's likely some real-time functions they have that might prove useful but obviously anything that requires cloud computing isn't useful for under 30ms rendering.

To me the whole article stinks of getting a headline to draw attention to their company.
 
Last edited:
A demo video of their tech:
This video has no bearing on upscaling. Either this video is not related to what Eric Risser was talking about, or Mr. Risser is talking bollocks. What we're seeing is a bunch of tiles being procedurally placed, packaged into a nice tool. It can't do a single thing to take a 1080p game image and reconstruct a 4k image. There might be a cloud based upscaling algorithm involved in creating the tiles from low-res sources which might work on, but that is unlikely to map well to games.

Having just perused the article, I conclude option 2: Risser doesn't know what he's talking about and is just looking for PR points to promote his company:
“The Checkerboard technique shouldn’t be too much trouble to program as it’s just a simple post-process,” he said. “They essentially just render half the pixels in a checkerboard pattern and then fill in the blank pixels by blurring together the rendered pixels. People have been doing tricks like this for decades. As for how it will stack up against true 4k rendering. I honestly can’t say without looking at a few games being played side by side. Obviously the quality won’t be as good, the question is if it will be noticeably bad.
Checkboard rendering is not blurring pixels and hasn't been in use for decades.

Artomatix reputation -1
 
he is taking the original question out of context

btw in a real video game, maybe it can be used when loading a level? so they can "bake" it? but then it will eats memory...
 
This video has no bearing on upscaling. Either this video is not related to what Eric Risser was talking about, or Mr. Risser is talking bollocks. What we're seeing is a bunch of tiles being procedurally placed, packaged into a nice tool. It can't do a single thing to take a 1080p game image and reconstruct a 4k image.

The same NN being used to create a larger texture that seemingly identifies objects/patterns and replicates them with smooth transitions could be used or at least adapted to fill in the blank pixels in a 1440p -> 2160p upscale.
I have no doubt they could achieve better visual results with this than they do with checkerboard, but doing so in real-time without taking away too many processing resources is the biggest concern here. At least by looking at the time it takes to "over-hallucinate" a single texture in that video.
 
If you use neural networks, the AI can see a game in 1920*1080, and then imagine how it would look in 4K, actually reconstructing the image to display natively at 4K. However, the amount of processing power needed could be immense, not to mention the danger of neural networks become self aware when 'upscaling' Horizon: Zero Dawn, noticing humans destroying machines in that game and as a result, rising up against humanity by hacking into the US Army servers and launching some nukes into Russia, effectively starting WW3.

Taking all that into account, I'd rather have Sony and MS use checkerboard rendering, sorry Eric Risser
 
This comes from an article at Gamingbolt.

I’m a little disappointed that Sony would go with something simple and outdated like the Checkerboard approach,” says Eric Risser, the CTO of Artomatix.

Read more at http://gamingbolt.com/sony-going-wi...inting-says-artomatix-cto#17w1BmqZREqORTWG.99
They clearly are confusing modern checkerboard rendering techniques with a simple old technique that practically nobody used.

There are two kinds of modern checkerboard techniques. Temporal and spatial. Temporal checkerboard needs to interpolate between pixels only in areas where reprojection failed. In these areas you can "see" half pixel density for a single frame before the next frame brings it back to full resolution. Spatial checkerboard uses extra data that is generated at pixel frequency. For example depth and stencil (object id). Modern GPUs (Nvidia and AMD) are able to run shader at different frequency compared to depth and stencil (even default MSAA does this). This additional pixel precise data is then used to perform more intelligent upscale at end of the frame. Artomatix clearly didn't know about these improvements.

If you think checkerboard upscale is simple, check Page 62 of this presentation:
http://twvideo01.ubm-us.net/o1/vaul...s/El_Mansouri_Jalal_Rendering_Rainbow_Six.pdf

I am sure many studios are experimenting with neural network based temporal AA and temporal upsampling. Current academic research has been mostly single frame. Meaning that results are not temporally stable in motion and result is not as good as jittered sampling over multiple frames is able to produce. But I am confident that in a few years we will see the first neural network based temporal AA+upsample solutions in games (high end PC first). Vega seems to have 2xFP16 and 4xINT8 in consumer models, so it will be really well suited for running code like this. Nvidia currently limits these features to 5000$+ professional products.
 
Note, motion compensation and other temporal trickery isn't part of NNEDI. It's just a scaler.
And that's a big problem. 4x4 upscaling greatly increases temporal aliasing. Pixel wide fluctuation/shimmering now becomes 4x4 wide fluctuation/shimmering. Need more than than 1 sample per pixel to reconstruct temporally stable 4x4 upscaled video. Modern jittered temporal sampling (similar to TAA) combined with neural network resolve could be used to generate antialiased AND spatially upscaled AND temporally upscaled video. Could be interesting to see what kind of quality would we get from a state of the art neural network. I have implemented state of the art TAA filter and other reprojection/reuse algorithms, so I know this is definitely possible. Unfortunately I no longer work at Ubisoft (= big team). I am now the sole graphics programmer in our own company, so I don't personally have time to experiment with neural networks.

This should be perfectly doable:
Input = 1080p, 30 fps, non-AA with 8x sub-sample temporal jitter
Output = 4K, 60 fps, antialiased

2x2 spatial upsample, 2x temporal upsample. This is less demanding than 4x4 upsampling (shown commonly in single image cases). Also we have 8 frames worth of jittered history (8x more input data than single frame algorithms). The neural network could be made it to look more natural by training it with real world images. This would hopefully reduce the need to add all kinds of silly filters to hide "computer-generated" look (such as chromatic aberration).
 
Can you post results here? just curious to see them

Mind you, I ran the original screenshot through an SMAA tool a while ago. This is also cropped, and compressed a tiny bit. I upscaled the original with lanczos.

I didn't grab the "boring" version of the enhanced image though.

Lanczos
RBlanczos.jpg

Enhance
RBenchant.jpg

(1800x2400)

I had another one where a character was a lot smaller in screen-space, and the eyes got upscaled rather weirdly (to be fair, the eyes were like 10x5 pixels in area :p). Unfortunately, I erased it, but that might have been more interesting for analyzing.
 
Mind you, I ran the original screenshot through an SMAA tool a while ago. This is also cropped, and compressed a tiny bit. I upscaled the original with lanczos.

I didn't grab the "boring" version of the enhanced image though.

Lanczos
View attachment 2282

Enhance
View attachment 2283

(1800x2400)

I had another one where a character was a lot smaller in screen-space, and the eyes got upscaled rather weirdly. Unfortunately, I erased it, but that might have been more interesting for analyzing.
impressive. If you look at the hair it really does wonders there.
 
LOL, that's terrible. Look at the straps to see a complete loss of actual detail, as well as on the cloth.

Hmm, the more I look the worse it gets. It actually looks like a dumb upsample followed by non-local means denoising (which as a side effect cleans up high-contrast edges).
 
Back
Top