Carmack on Ray Tracing & Rasterization

anexanhume

Veteran
For anyone holding out hope for dedicated ray tracing hardware, Carmack blows out the flame:

http://arstechnica.com/gadgets/2013...rd/?comments=1&post=23723213#comment-23723213

I wrote the following (slightly redacted) up a little while ago for another company looking at consumer level ray tracing hardware as it relates to games. I do think workstation applications are the correct entry point for ray tracing acceleration, rather than games, so the same level of pessimism might not be apropriate. I have no details on Imagination’s particular technology (feel free to send me some, guys!).

------------

The primary advantages of ray tracing over rasterization are:

Accurate shadows, without explicit sizing of shadow buffer resolutions or massive stencil volume overdraw. With reasonable area light source bundles for softening, this is the most useful and attainable near-term goal.

Accurate reflections without environment maps or subview rendering. This benefit is tempered by the fact that it is only practical at real time speeds for mirror-like surfaces. Slightly glossy surfaces require a bare minimum of 16 secondary rays to look decent, and even mirror surfaces alias badly in larger scenes with bump mapping. Rasterization approximations are inaccurate, but mip map based filtering greatly reduces aliasing, which is usually more important. I was very disappointed when this sunk in for me during my research – I had thought that there might be a place for a high end “ray traced reflections” option in upcoming games, but it requires a huge number of rays for it to actually be a positive feature.

Some other “advantages” that are often touted for ray tracing are not really benefits:

Accurate refraction. This won’t make a difference to anyone building an application.

Global illumination. This requires BILLIONS of rays per second to approach usability. Trying to do it with a handful of tests per pixel just results in a noisy mess.

Because ray tracing involves a log2 scale of the number of primitives, while rasterization is linear, it appears that highly complex scenes will render faster with ray tracing, but it turns out that the constant factors are so different that no dataset that fits in memory actually crosses the time order threshold.

Classic Whitted ray tracing is significantly inferior to modern rasterization engines for the vast majority of scenes that people care about. Only when two orders of magnitude more rays are cast to provide soft shadows, glossy reflections, and global illumination does the quality commonly associated with “ray tracing” become apparent. For example, all surfaces that are shaded with interpolated normal will have an unnatural shadow discontinuity at the silhouette edges with single shadow ray traces. This is most noticeable on animating characters, but also visible on things like pipes. A typical solution if the shadows can’t be filtered better is to make the characters “no self shadow” with additional flags in the datasets. There are lots of things like this that require little tweaks in places that won’t be very accessible with the proposed architecture.

The huge disadvantage is the requirement to maintain acceleration structures, which are costly to create and more than double the memory footprint. The tradeoffs that get made for faster build time can have significant costs in the delivered ray tracing time versus fully optimized acceleration structures. For any game that is not grossly GPU bound, a ray tracing chip will be a decelerator, due to the additional cost of maintaining dynamic accelerator structures.

Rasterization is a tiny part of the work that a GPU does. The texture sampling, shader program invocation, blending, etc, would all have to be duplicated on a ray tracing part as well. Primary ray tracing can give an overdraw factor of 1.0, but hierarchical depth buffers in rasterization based systems already deliver very good overdraw rejection in modern game engines. Contrary to some popular beliefs, most of the rendering work is not done to be “realistic”, but to be artistic or stylish.

I am 90% sure that the eventual path to integration of ray tracing hardware into consumer devices will be as minor tweaks to the existing GPU microarchitectures.

John Carmack
 
For anyone holding out hope for dedicated ray tracing hardware, Carmack blows out the flame:

Some really good points there!

Accurate shadows, without explicit sizing of shadow buffer resolutions or massive stencil volume overdraw.

Also, shadow buffers require you to manually adjust bias for the shadows, if you want to avoid shadows disappearing or bleeding through - raytracing doesn't require that either.
Games could be using some automatic solution but it creates some different kinds of artifacts anyway.

He also didn't mention that normal mapped surfaces are immediately betrayed by raytraced shadows. By their nature they're so accurate and sharp that they will be based on the actual triangles making up the low polygon model, making everything look ugly.
Tessellation might help here to an extent, but probably not on the upcoming console generation.

Global illumination. This requires BILLIONS of rays per second to approach usability.

Not sure how many rays we trace and how many bounces we have, but our render times using a full raytracer renderer are quite manageable nowadays. It is highly optimized though, dropping to quantization from full floating point data whenever it can and so on.
There are also ways to deal with noise, or to create pleasant looking noise, and a lot of it may be covered up by texture detail and by motion blur, film grain and other image processing based effects.

Raytracing also helps with subsurface scattering, he didn't mention that.

I can check some of our average render times tomorrow at the office if anyone's interested. Not sure about the FLOPS ratings of the render nodes but I can ask the sysadmins about that.
 
Would be cool info - generally, if the CGI biz starts using a technology, I always feel that the real-time industry (like games) can't be much more than 10 years behind.

So maybe just next-next-gen? ;) Definitely not the upcoming gen though, that's for sure. That one will still mostly see a lot of ray-casting for approximations I reckon.
 
The issue with raytracing is still the sudden changes in rendering times depending on the content. Move the camera just a bit and you end up tracing several times as many rays. We can work with that, because what we have to manage is making a deadline for ALL the movie - but games need to keep every frame under a certain limit.

We can also cheat a lot more. Most people would be utterly surprised at the amount of 2D/2.5D/3D matte paintings we use, and our practices are way behind what Weta or ILM can do with the approach.
 
I can't understand the buzz around raytracing
It can do almost the same things that normal rasterization does but faster, only that it does way slower.
It needs a dedicated hardware coprocessor, or a dedicated second gpu, that works like a coprocessor...
Why not stick with traditional rendering and live happy?
 
Using raytracing has significantly reduced the artist time required to complete a shot at most CGI / VFX studios.

You set a few parameters and you get:
- realistic shadows
- realistic reflections
- realistic indirect lighting
- realistic shaders

No need to manually set up shadow maps with all their parameters, or pre-render them.
No need to project pre-rendered images to cheat reflections.
No need to place a lot of lights to compensate for the lack of indirect lighting.
Also, everything looks far more realistic from the start without spending ages tweaking the shaders.


I can understand if game artists are envious of all that. Just the amount of time spent setting up, optimizing, and rendering lightmaps has to be insane.
 
Raytracing is easy to use, sure, but render times go up the better you want it to look. That's the other thing that CG doesn't have to deal with.. nothing's real-time.

By the time hardware gets to the point where it can really do raytracing in real-time, then the alternative methods will have caught up to the point where there's no difference visually, and it'll still be faster.

Besides, these other methods are getting better all the time.
 
Not sure how many rays we trace and how many bounces we have, but our render times using a full raytracer renderer are quite manageable nowadays.

Try using GI with fur in a RT. Or try motion blur with glossy reflections. ;)

There are also ways to deal with noise, or to create pleasant looking noise,

The big thing these days is importance sampling. But this requires that you supply the BRDF, PDF and sampling function.

Raytracing also helps with subsurface scattering, he didn't mention that.

Yea. But I think that's even more expensive because you'd still have to store an octree of radiance values just to resolve them at each pixel. If the light penetrates the surface even more, then you'll have to query even more data points.

There are pre-pass ways to do it, but a lot of film studios are pulling away from the point cloud baking.

-M
 
I can't understand the buzz around raytracing
It can do almost the same things that normal rasterization does but faster, only that it does way slower.
It needs a dedicated hardware coprocessor, or a dedicated second gpu, that works like a coprocessor...
Why not stick with traditional rendering and live happy?

There is a LOT of advantages to RT. And a lot of disadvantages to scanline. You'd really have to be working in a production to see all the things that completely don't work in scanline.

Just put it this way - RT mimics more of the real world behavior of light bouncing than scanline.

-M
 
Using raytracing has significantly reduced the artist time required to complete a shot at most CGI / VFX studios.

...

I can understand if game artists are envious of all that. Just the amount of time spent setting up, optimizing, and rendering lightmaps has to be insane.

Instead of being envious, they should think: if artist workload can be greatly reduced, the artist workforce can also be substantially reduced, and I may need to find another job. :LOL:
 
Oookay, some info on render times as promised...

First of all everything is fully raytraced, but not too many reflective surfaces and almost no refractions are visible in the usual type of content.
We're using area lights and global illumination, not sure about how many diffuse bounces are included though.
We have very few things to adjust with regards to quality, which is actually a welcome thing as it makes optimizing renders an easier task. It's also safe to say that even our preview settings look significantly better than the average game; however both motion blur and GI and shadows are quite noisy, so we have to take a significant hit for acceptable image quality.

Now the problem is that each scene is separated into various layers for a bunch of reasons. Sometimes it's impossible to render everything together because it'd cause memory problems and so on. It's also faster to re-render just an individual element if something is wrong with the asset or the animation.
The other reason is that we optimize a lot - for example if we render a character with metal pieces on his clothing, those would have to reflect the entire environment. But for this layer we switch all the other objects to a lower detail version with simplified shaders if the reflection is a bit glossy and distorted, because noone would see the difference.

Averages for a randomly selected shot's elements are:
- main character 40 minutes per frame
- various hair eyebrow fur layers etc. 3-7 minutes
- various set elements/dressings 5-7 minutes
- volumetric smoke elements 4-20 minutes
Some elements haven't been completed and rendered at preview settings:
- secondary characters in background 5-8 minutes
- main set 20 minutes
Altogether 32 layers for this shot, all at 1280x544 resolution.

As suggested by the above, moving from preview to production settings increases render times about 500%. So we can say that the combined rendering time for all elements will add up to a few hours.

The 2D compositing job that combines these elements together and adds various post processing effects renders in about 3 minutes per frame. These include stuff like color correction, tone mapping, relighting on both individual elements and the final image, and even crazy little tweaks like adjusting color and transparency for hair and eyelashes in some cases.

We have a few dozen render machines purchased during the past years so they're not all the same; the fastest ones have two 6-core Xeon CPUs. Such a machine may render two frames at once AFAIK.

I guess this can help puts the requirements of raytracing into perspective.
 
Instead of being envious, they should think: if artist workload can be greatly reduced, the artist workforce can also be substantially reduced, and I may need to find another job. :LOL:

Not really, what this would mean is that the level can go through more iterations and refinements.
 
GPUs wouldn't help us with raytracing. Arnold, the renderer we use, does not support it and we'd prefer it to remain that way.

Also, where the hell would we put the GPUs? These are rack based machines, 2U high usually. Heat is already a problem in our office building, it's more than 150 years old (and very nice, originally built for the Budapest shipyards, they made steamboats here).


BTW as far as I know only a few VFX studios use GPUs for significant rendering tasks, for example Weta has them calculate their spherical harmonics based lighting. The resulting intermediate datasets are then fed to PRMan running on regular CPUs.
The current trend is to get rid of such intermediate steps, pointcloud data and shadow maps and such, because it takes a lot of additional artist time to manage these. It's easier and cheaper to just raytrace and buy more CPUs.
 
GPUs wouldn't help us with raytracing. Arnold, the renderer we use, does not support it and we'd prefer it to remain that way.

Also, where the hell would we put the GPUs? These are rack based machines, 2U high usually. Heat is already a problem in our office building, it's more than 150 years old (and very nice, originally built for the Budapest shipyards, they made steamboats here).


BTW as far as I know only a few VFX studios use GPUs for significant rendering tasks, for example Weta has them calculate their spherical harmonics based lighting. The resulting intermediate datasets are then fed to PRMan running on regular CPUs.
The current trend is to get rid of such intermediate steps, pointcloud data and shadow maps and such, because it takes a lot of additional artist time to manage these. It's easier and cheaper to just raytrace and buy more CPUs.

I understand that, but from the context of Carmack's comments, it's about knowing where raytracing on highly parallel compute hardware like GPUs is. Even the best CPU still has a tenth of peak single GPU FLOP performance.
 
I've only promised to give render times, that can be used as a guideline for how computationally intensive this stuff actually is.

Then you drop in and start asking about completely unrelated stuff... ;)
 
Back
Top