Pixel Invariance

[Enbar]

It seems to me that some people have over reacted a little towards the current issues with pixel precision. I remember reading about a developer saying the future version of graphic APIs should be pixel precise. I sort of dismissed that as the mad ranting of a frustrated individual. Recently I heard something that brought that comment back into my mind and made me chuckle. I was talking to a person who was using the dx9 reference rasterizer and he was getting slight differences between a pentium3 and a pentium4 system. That reminded me of some software rasterizing I worked on a while back. I was working with the SSE instruction set on the p4 and found that I was getting different pixel results using different rendering modes. It turned out to be a case where (a+b)+c was getting results that were different enough from a+(b+c) that pixels were different.

Basically the moral of the story here is that people should not ask for pixel precise results from an API. They should ask for tolerances just like the OpenGL and d3d specs currently have. If these specs aren’t good enough for developers get the tolerances tightened up and algorithms more defined, but don’t go crazy and ask for exact pixel reproduction. Also it’s possible the specs are good enough and some hardware doesn’t or can’t follow them. However knowing something about the OpenGL spec I know there are places they leave a lot of wiggle room like with aniso filtering.

 

[KimB]

It is extremely disturbing to me that CPU's are pumping out different numbers for such simple calculations. I hope you oversimplified that example?

Anyway, yes, there need to be tolerances. I think that the tolerances should be twofold:

1. Single calculations should be accurate up to a specific point.
2. The average of the errors should be zero (that is, instead of the error always being low, as it would be with truncation of integers, there would be an equal chance of the error being high or low).

The first guarantees consistency (up to a certain point).
The second vastly improves the ability of processors to compute long strings of data without the errors becoming unmanageable.

 

[pcchen]

If you are using floating point numbers, you don't want to assume (a+b)+c = a+(b+c), because it is not always true. Note, however, two different CPUs both are IEEE 754 compliant should generate the same results (and the same difference between (a+b)+c and a+(b+c) for the same input).

 

[Dio]

What he said, double. There are no floating-point operations which are guaranteed to be commutative.

As I've said before, even C-written floating point code has this problem. C does not guarantee order of evaluation for arithmetic operands, so if you write D = a+b+c then you could get either (a+b)+c or a+(b+c).

Throw in an optimiser and things get worse. Restricting the optimiser to 'no optimisations that might change the results on the single-bit level' is bad for performance. Most current C optimisers on max settings do allow these things - such as the replacement of division by multiplication by the reciprocal.

 

[akira888]

What he said, double. There are no floating-point operations which are guaranteed to be commutative.

As I've said before, even C-written floating point code has this problem. C does not guarantee order of evaluation for arithmetic operands, so if you write D = a+b+c then you could get either (a+b)+c or a+(b+c).

Throw in an optimiser and things get worse. Restricting the optimiser to 'no optimisations that might change the results on the single-bit level' is bad for performance. Most current C optimisers on max settings do allow these things - such as the replacement of division by multiplication by the reciprocal.

I always had thought that the ANSI standard at the very least enforced a left to right evaluation order for arithmetic operands. At least what every book (including K+R) has told me. Oh well.

 

[sonix666]

If you are using floating point numbers, you don't want to assume (a+b)+c = a+(b+c), because it is not always true. Note, however, two different CPUs both are IEEE 754 compliant should generate the same results (and the same difference between (a+b)+c and a+(b+c) for the same input).

I doubt it. It all depends on the internal accuracy of the FPU of the CPU. IA32 chips, like Athlon and P4, produce different results normally than say an Alpha CPU for 32 and 64 bit floating points because the FPU internally works at 80 bit.

 

[Xmas]

If you are using floating point numbers, you don't want to assume (a+b)+c = a+(b+c), because it is not always true. Note, however, two different CPUs both are IEEE 754 compliant should generate the same results (and the same difference between (a+b)+c and a+(b+c) for the same input).

I doubt it. It all depends on the internal accuracy of the FPU of the CPU. IA32 chips, like Athlon and P4, produce different results normally than say an Alpha CPU for 32 and 64 bit floating points because the FPU internally works at 80 bit.

It can work at 80 bit.
Still, any IEEE 754 compliant FPU will get the same 32(64)-bit float result when adding two 32(64)-bit float numbers, no matter whether the internal precision is higher. Though if you store the result in an 80-bit register, it might be more accurate.

 

[sonix666]

In default mode, while still being IEEE compliant an x87 capable CPU will not give you the same results as you'd expect from 32 bit or 64 bit floating point operations if you run multiple operations without spilling the intermediate results to memory.

 

[Xmas]

That's pretty much what I said. But anyway, if you need fast calculations, you're certainly not going to waste your time in extended mode.
And operator==(float, float) is a big no-no.

 

[KimB]

use of operator ==(float, float) isn't entirely a big no-no. It could be used, for example, to test if a value has been changed.

 

[MaxSt]

What if you're processing medical images, like brain scans?
Would a failure to diagnose a brain tumor on early stage bother you, just because an image got a little distorted?

MaxSt.

 

[Xmas]

What if you're processing medical images, like brain scans?
Would a failure to diagnose a brain tumor on early stage bother you, just because an image got a little distorted?

MaxSt.

Calculations with discrete numbers introduce error, period. There's no point in complaining about it, you just have to be aware of it. It's the programmer's task to find an appropriate balance between precision and speed. If the hardware is not capable of the neccessary precision, it can't be used. IMO you should never rely on numerical invariance with respect to float calculations. Just make sure the error margins fit your needs.

 

[no_way]

What if you're processing medical images, like brain scans?
Would a failure to diagnose a brain tumor on early stage bother you, just because an image got a little distorted?

I'd suggest re-reading Numerical Recipes in C, chapter 1-3 again. If you didnt do that in the first place, youre probably in the wrong business.[/i]

 

[Hyp-X]

In default mode, while still being IEEE compliant an x87 capable CPU will not give you the same results as you'd expect from 32 bit or 64 bit floating point operations if you run multiple operations without spilling the intermediate results to memory.

x87 FPUs can work in 32 / 64 / 80 bit modes.
Altough the default mode is 80 bit, MSC changes it to 64 bit.
If you use D3D is sets it to 32 bit (unless FPU_PRESERVE is used).

 

[Myrmecophagavir]

What he said, double. There are no floating-point operations which are guaranteed to be commutative.

As I've said before, even C-written floating point code has this problem. C does not guarantee order of evaluation for arithmetic operands, so if you write D = a+b+c then you could get either (a+b)+c or a+(b+c).

Throw in an optimiser and things get worse. Restricting the optimiser to 'no optimisations that might change the results on the single-bit level' is bad for performance. Most current C optimisers on max settings do allow these things - such as the replacement of division by multiplication by the reciprocal.

I always had thought that the ANSI standard at the very least enforced a left to right evaluation order for arithmetic operands. At least what every book (including K+R) has told me. Oh well.

Operators have given associativities, left-to-right or right-to-left. It should be that a+b+c always evaluates to (a+b)+c in C?

 

[pcchen]

If you use double (64 bits), you'd better set the internal precision to 64 bits, not 80 bits. Although 80 bits mode sometimes gives better results, it brings more problems. A big one is double rounding (the results rounded to 80 bits FP, and then rounded to 64 bits FP when you store them into memory). Other problems such as:

double a = 0.1;
if(a == 0.1) printf("OK!") else printf("Not OK!");

If you use 80 bits internal precision, the above code may print "Not OK!" since 0.1 in 64 bits FP is different from 0.1 in 80 bits FP.

Another problem with x87 is the range. Although you can set the internal precision to 64 bits or 32 bits, but the range will remain the same. That is, some results which should overflow in IEEE 754 64 bits FP may still in valid range with x87. Normally this is not a problem. However, it is not strictly IEEE 754 64 bits/32 bits FP compliant (unless you store every temporary results into memory). SSE/SSE2 don't have these problems, since they operate directly on 64 bits and 32 bits FP.

 

[darkblu]

If you use double (64 bits), you'd better set the internal precision to 64 bits, not 80 bits. Although 80 bits mode sometimes gives better results, it brings more problems. A big one is double rounding (the results rounded to 80 bits FP, and then rounded to 64 bits FP when you store them into memory). Other problems such as:

double a = 0.1;
if(a == 0.1) printf("OK!") else printf("Not OK!");

If you use 80 bits internal precision, the above code may print "Not OK!" since 0.1 in 64 bits FP is different from 0.1 in 80 bits FP.

Another problem with x87 is the range. Although you can set the internal precision to 64 bits or 32 bits, but the range will remain the same. That is, some results which should overflow in IEEE 754 64 bits FP may still in valid range with x87. Normally this is not a problem. However, it is not strictly IEEE 754 64 bits/32 bits FP compliant (unless you store every temporary results into memory). SSE/SSE2 don't have these problems, since they operate directly on 64 bits and 32 bits FP.

although your concern is generally correct, the particular example you've given would always result in "OK" since both '0.1' constants get onto the fpu stack as doubes (the x87 does not have 80-bit interface to the world); now, *if* one of those '0.1' came as a result from a non-spilled (fpu-internal) calculation, where fpu prec had been set to 80bit, then undoubtedly the comparison would fail miserably.

 

[Simon F]

What he said, double. There are no floating-point operations which are guaranteed to be commutative.

Sorry to be pedantic, but I think you meant to write "associative".

In summary: Floating point numbers are only an approximation of the Reals.

 

[Dio]

What he said, double. There are no floating-point operations which are guaranteed to be commutative.

Sorry to be pedantic, but I think you meant to write "associative".

Always get those two mixed up. Cheers.

 

[Dio]

even C-written floating point code has this problem. C does not guarantee order of evaluation for arithmetic operands, so if you write D = a+b+c then you could get either (a+b)+c or a+(b+c).

I always had thought that the ANSI standard at the very least enforced a left to right evaluation order for arithmetic operands. At least what every book (including K+R) has told me. Oh well.

Operators have given associativities, left-to-right or right-to-left. It should be that a+b+c always evaluates to (a+b)+c in C?

I've checked this and you're both right (although I think it's a + (b+c) from my reading of K+R, which like so much else, could be wrong...). I was getting confused about the order of evaluation of operands.