Gamma and linear color space

[Xmas]

Partly inspired by the "Future of MSAA" thread, I'd like to start a discussion on how to best solve the problem of proper color calculation and display considering the possible non-linearity of content and display devices.

Linear color space means that the intensity values are proportional to the amount of photons they represent. It seems quite obvious that any lighting calculations should take place in this space. If you add a second identical light source, you get double the amount of photons, a transparent surface lets a certain percentage of photons pass, etc. It's all simple, linear math (although it's not quite that simple if you consider spectral distribution).

But there are two other things to consider.
- First, a typical display device doesn't have a linear response curve, i.e. if you double the intensity value (voltage for analog transmission), you get more than double the photons. Usually, that relation can be approximated as
Luminance ~ signal^gamma
with signal being in the [0,1] range and gamma typically being about 2.

- And second, our perception isn't linear, but approximately logarithmic. The ratio between just noticeable difference and luminance doesn't change much. This has a big impact on required precision, in that we need much more values representing the darker colors.

(One nice property of those two issues combined is that the 8bit precision DVI uses is acceptable in most cases. 16bit per channel is specified, but I know of no devices supporting this.)

Basically, it means that an 8bit linear representation per channel isn't enough for storing linear color space content. Floating point values, OTOH, are perfectly suited for this, due to their logarithmic precision. But FP16 values (the lowest supported FP precision) need more bandwidth, so it might be more practical to map those 8bit values in a non-linear way, during every read and write operation. Current hardware already supports such a mapping (sRGB reads and writes), computing ^2.2 when reading and ^.45 when writing. It is very important that this conversion is applied during any read and write operation to a sRGB buffer. Also, any operation in linear color space must take place using higher precision.

If we look at the 3d pipeline, we have textures, vertex colors and PS/VS constants as possible color input values. The latter two are FP precision anyways, so linear color space is no problem here.

Overall, what we have to consider is

texture read -> PS -> [framebuffer read -> blend ->] framebuffer write -> framebuffer read -> AA downsampling -> [framebuffer write -> framebuffer read ->] gamma LUT -> output to screen

Optional parts in []

Until very recently, games basically didn't really care about gamma conversion, and treated everything as being in linear color space, though it wasn't. I'm not even sure there are games that take advantage of the sRGB capabilities of DX9 hardware yet.

But for an overall pleasing rendered and displayed image, I think it is important that hardware and software treat colors correctly.

Every (color) texture read, framebuffer read and framebuffer write should be either sRGB mapped or FP16, and PS, blending, AA downsampling and gamma LUT need to provide enough precision.

AFAIK, ATI currently has neither FP16 nor sRGB framebuffer reads for blending, and no high precision blending. NVidia is missing the sRGB framebuffer reads for blending, as well as sRGB AA downsampling.
I hope WGF will require sRGB and FP16 in all those places.

 

[3dcgi]

I'm surprised if neither company supports sRGB reads for blending. Supporting sRGB for writes, but not reads seems very limiting. I assumed if you are doing gamma correct AA you're really just storing the frame buffer in sRGB so you'd need to support sRGB for reads.

 

[vember]

yes, it's a shame.

linear color space makes lighting look much better. The dark areas of the image no longer dissappear and the falloffs are smoother. And as you said, its suitable for vertex lighting or pixel-lighting done in one pass. It's just the blending that's the problem.

I guess you could force important transparent surfaces (like glass) to be rendered in seperate passes and use that as an excuse to add subtle screen-space distortions to them (like the excellent windows in doom3). For stuff like foilage it doesn't matter that much so their blending could still be done in gamma space. (the edges will be slightly worse though).

I have no idea how to do particle systems though. I reckon they really need linear math to look right, and they're way to blending intense to be able to use PS blending.

 

[Hyp-X]

You don't need sRGB blending.
The rendertarget (back buffer) should be in linear color space.

You can use sRGB -> Linear presentation (ATI) or emulate that with PS in a separate pass. (If you wan't some kind of tone mapping you'll have that PS pass anyway - so gamma correction is "free").

The things that are really needed are FP16 blending (8 bit is insufficient for linear color space), and FP16 AA downsampling.
Unfortunately I'm not sure when we'll see the latter one.

 

[Xmas]

You don't need sRGB blending.
The rendertarget (back buffer) should be in linear color space.

You can use sRGB -> Linear presentation (ATI) or emulate that with PS in a separate pass. (If you wan't some kind of tone mapping you'll have that PS pass anyway - so gamma correction is "free").

The things that are really needed are FP16 blending (8 bit is insufficient for linear color space), and FP16 AA downsampling.
Unfortunately I'm not sure when we'll see the latter one.

sRGB can be considered an encoding of linear color values, and the bandwidth advantage over FP16 makes it very attractive. As long as you don't do HDR rendering or excessive blending, it can be sufficient.

What I'd like to see is FB compression always decoding to FP16, blending at FP16, and eventually writing compressed FP16 or sRGB (or linear 8bit for non-color data) again.

 

[Rolf N]

The easiest solution would be to just set a ramp according to gamma=2.2 (and let the user make adjustments if this doesn't fit the display). Everything would then be immediately correct: shader ops, texture filters, framebuffer blending, just everything would happen in a perfectly linear color space from source data to display.

- And second, our perception isn't linear, but approximately logarithmic. The ratio between just noticeable difference and luminance doesn't change much. This has a big impact on required precision, in that we need much more values representing the darker colors.

I'm still unsure of that. Any links or other references on the subject? Of course I'm somewhat willing to believe you right away, but there are two interdependent issues here that may confuse the analysis.

Even with a strictly linear detection system, you'd still determine 10 times higher resolution if you let 10 times more light in, hence your conclusion could be that the detector is logarithmic if you're not careful.

Let's assume for a moment that the retina and brain perceive intensity linearly. If that's true, you could still come to the above conclusion that perception is logarithmic. Present a test subject with a dark environment and a light environment, give them some time to adjust and then have them pick visibly different luminance levels. The wildcard in this kind of experiment, which would affect the results, is iris contraction.

Such experiments should IMO be made with constant average luminance to defeat iris adjustments. Preferably just a well lit room. And I simply don't know if this is how the analysis you refer to was done.
So please, if you have any references, I'd like to see them.

____________

If you're right, let me just say that there's still some elegance in this approach vs "sRGB". If we do need significantly more precision for this, it may be feasible to just slap on 16 bit per channel fixed point ( [0;1] ) framebuffers and RAMDACs instead of sprinkling all of this compensation logic throughout the processing pipeline (or maybe the 10-10-10-2 format, as an intermediate step, could gain some more traction). Yes, this comes at a bandwidth cost. But "sRGB" isn't free either. It's a massive complexity and cost overhead, and then you need to consider rounding issues.

 

[Vadi]

- And second, our perception isn't linear, but approximately logarithmic. The ratio between just noticeable difference and luminance doesn't change much. This has a big impact on required precision, in that we need much more values representing the darker colors.

I'm still unsure of that. Any links or other references on the subject? Of course I'm somewhat willing to believe you right away, but there are two interdependent issues here that may confuse the analysis.

Isn't this the reason for gamma?

 

[Rolf N]

Isn't this the reason for gamma?

No. The reason for gamma is nonlinearity of display devices.
A "normal" monitor will display black if you give it zero voltage input and white at (commonly) 0.75 volts. The problem is that it won't display 50% grey at half that voltage, but rather 20% grey.

This in itself has nothing to do with human perception.

Cranking up the gamma is an attempt to map midtones in the framebuffer to higher output voltages, so that a value of "one half" in the framebuffer shows up as 50% grey on the monitor. That's what I'm suggesting: crank up the gamma ramp in a way that makes values in the framebuffer correspond directly to physical intensity. And XMas says that current 8 bit per channel framebuffers don't have enough precision to do so.

edited bits:
Or do you mean that displays are deliberately made nonlinear to better fit human perception?
I don't think that's the case. CRT technology is very old. I think some nonlinearity is inevitable. It might just be "tradition" to not fix it, as so much depends on it.

 

[3dcgi]

You don't need sRGB blending.
The rendertarget (back buffer) should be in linear color space.

Maybe I'm mistaken as I've never written a DX9 app, but I thought DX9 allowed you to specify a sRGB frame buffer, which is essentially 10 bit or higher precision values "compressed" with a 2.2 gamma and stored in an 8 bit frame buffer.

So a typical rendering flow might involve 8 bit sRGB textures and an 8 bit sRGB framebuffer, but all calculations are performed at an internal precision of 10 bits or higher.

 

[Xmas]

The easiest solution would be to just set a ramp according to gamma=2.2 (and let the user make adjustments if this doesn't fit the display). Everything would then be immediately correct: shader ops, texture filters, framebuffer blending, just everything would happen in a perfectly linear color space from source data to display.

That would work. Unfortunately, 8 bits per channel aren't enough for that kind of linear representation...
Also, textures usually aren't in linear color space.

- And second, our perception isn't linear, but approximately logarithmic. The ratio between just noticeable difference and luminance doesn't change much. This has a big impact on required precision, in that we need much more values representing the darker colors.

I'm still unsure of that. Any links or other references on the subject? Of course I'm somewhat willing to believe you right away, but there are two interdependent issues here that may confuse the analysis.

Even with a strictly linear detection system, you'd still determine 10 times higher resolution if you let 10 times more light in, hence your conclusion could be that the detector is logarithmic if you're not careful.

A very interesting paper:
http://www.eg.org/EG/DL/dissonline/doc/matkovic.pdf

 

[Vadi]

Isn't this the reason for gamma?

No. The reason for gamma is nonlinearity of display devices.
A "normal" monitor will display black if you give it zero voltage input and white at (commonly) 0.75 volts. The problem is that it won't display 50% grey at half that voltage, but rather 20% grey.

This in itself has nothing to do with human perception.

Cranking up the gamma is an attempt to map midtones in the framebuffer to higher output voltages, so that a value of "one half" in the framebuffer shows up as 50% grey on the monitor. That's what I'm suggesting: crank up the gamma ramp in a way that makes values in the framebuffer correspond directly to physical intensity. And XMas says that current 8 bit per channel framebuffers don't have enough precision to do so.

edited bits:
Or do you mean that displays are deliberately made nonlinear to better fit human perception?
I don't think that's the case. CRT technology is very old. I think some nonlinearity is inevitable. It might just be "tradition" to not fix it, as so much depends on it.

I think that's the case. According to the paper Xmas posted Weber's law of the human's ability to distinguish differences in luminance was introduced in the beginning of the last century.

I think the problem with nonlinear displays isn't the nonlinear response but the difference in the characteristics (printing).

 

[MfA]

By the way, be very carefull about believing anything you read about gamma responses for CRTs ... Poyton doesnt agree with the sRGB guys and some seemingly well researched papers disagree with both of them.

The only actual measurements I could find for gamma response curves dont agree with any theoretical model I am familiar with (gamma ranging from 1.4-1.7 with an offset of 0.02-0.1). I have come to the conclusion that most of the accepted knowledge about gamma is bunk.

If you just work with 32 bit floating point perception is irrelevant ... just work in a linear colour space at every step, and let the eyes/brains handle perceptual effects. Cant we just skip the headache of 16 bit per component framebuffers entirely? :/

 

[Simon F]

- And second, our perception isn't linear, but approximately logarithmic. The ratio between just noticeable difference and luminance doesn't change much. This has a big impact on required precision, in that we need much more values representing the darker colors.

I'm still unsure of that. Any links or other references on the subject? Of course I'm somewhat willing to believe you right away, but there are two interdependent issues here that may confuse the analysis.

It's well known. Try Poynton's colour and gamma FAQs or a textbook like Glassner's "Principles of Digital Image Synthesis".

 

[Xmas]

By the way, be very carefull about believing anything you read about gamma responses for CRTs ... Poyton doesnt agree with the sRGB guys and some seemingly well researched papers disagree with both of them.

The only actual measurements I could find for gamma response curves dont agree with any theoretical model I am familiar with (gamma ranging from 1.4-1.7 with an offset of 0.02-0.1). I have come to the conclusion that most of the accepted knowledge about gamma is bunk.

As long as the adjustment for the output device is done via a LUT and you have full control over the lookup curve, the actual response of the display device isn't that important.



Using a gamma value of 2.2 for storing color information as fixed point isn't related to the output device at all, but to the approx. logarithmic perception of the eye. I'm not sure 2.2 is the best value for that, but it's certainly better than storing linear values as fixed point.

If you just work with 32 bit floating point perception is irrelevant ... just work in a linear colour space at every step, and let the eyes/brains handle perceptual effects. Cant we just skip the headache of 16 bit per component framebuffers entirely? :/

Of course you can work in a linear color space anywhere, but you have to adjust for the output device eventually.
Why would FP16 framebuffers be a headache? You need quite a bit of blended layers to get banding here, and you can work in linear color space just as well as with FP32.

If, at some point, FP16 isn't sufficient any more, you change the target format to FP32 and that's it. No big deal, but the bandwidth (and transistor) savings are enough to justify FP16.



One issue I see with ATI's gamma-adjusted AA is that it automatically assumes the framebuffer is stored as sRGB, while the rest of the chip does not. It only works well if the textures are sRGB, and you consider this 2.2 value when you adjust the output to your monitor. But then you don't get linear math if the game is not sRGB aware.

 

[Rolf N]

Thanks for the links. Interesting stuff.

 

[MfA]

Using a gamma value of 2.2 for storing color information as fixed point isn't related to the output device at all, but to the approx. logarithmic perception of the eye.

If you want to work in a linear space you have to apply the gamma function for which the gamma correction was the inverse, perceptual concerns are entirely irrelevant ...

x^(1/2.2) aint a very good approximation of a logarithm.

 

[Xmas]

Using a gamma value of 2.2 for storing color information as fixed point isn't related to the output device at all, but to the approx. logarithmic perception of the eye.

If you want to work in a linear space you have to apply the gamma function for which the gamma correction was the inverse, perceptual concerns are entirely irrelevant ...

x^(1/2.2) aint a very good approximation of a logarithm.

True, but it's way better than storing linear color values in a fixed point format.

I don't care what function is used to encode the linear color values to an 8bit format, as long as it works well enough (i.e. it distributes values so you have the precision where the eye needs it the most) and can be realized with few transistors. I think an 8bit format still has its merits. You could even use "FP8", with something like 5 bits mantissa, 3 bits exponent biased with -8 and implicitly positive. That would most likely be better than 8bit fixed point, and you could easily transform it to wider FP formats.