Card A has a better 99% percentile performance than Card B.
And yet, if you bought Card A you would end up with 20 fps on average lower for this game and micro-stuttering. How many gamers do you know that would consider that a good purchase?
Percentiles have two main uses. First, they are useful for finding outliers. If the 99th percentile for frame rates is 20 ms, then frames at 100ms are odd - and you should probably look at them to see why. The second use is to find standing within one's group. If your daughter is in the 99th percentile, she is doing really well compared to the rest of her class.
Notice that I did not say they are useful for comparing different groups. Just because two sets have the same 99th percentile doesn't mean their behavior is at all similar.
What follows is a rather lengthy explanation of how that principle applies to this particular case. It isn't really necessary. Understanding the last sentence of the above paragraph is more than enough to understand my objection to the original chart. Using a spoiler tag to hide it from people who don't care.
So we go back to my first post. The plot is meant to compare several different video cards. Each card is compared using a percentile - which is sampled from an unknown distribution of frame rates. The assertion that the author is trying to make is that since 99% of frames for the 7970 are above 40 fps and 99% of the frames for the 780 are above 41 fps - they are the same card. After all, it is only a difference of 2.5% right?
That may seem reasonable at first, but it actually doesn't tell us anything about the performance of the two cards over 40 fps and over 41 fps. In this case, this particular site claims that the average FPS for the 7970 is 50 fps. The average fps for the 780 from their numbers is 55. So looking at the average, you get a difference of 10% - which is far more significant. Note that I wouldn't argue with you if you thought that still wasn't enough to justify cost, but it is an entirely different picture than the 2.5% paints.
So the next argument is that it tells you about frame time. The problem is that it doesn't. Choosing a number in the 7970's favor this time, look at the time spent beyond 33ms (this would be 30 hz - or where you would see a noticable jump in lag on most 60 or 120 hz monitors) for something like Crisis. In this case, the 7970 only spends 441ms beyond that number, while the 780 spends 663ms beyond it. Assuming the tests ran for the same amount of time, the 7970 outperforms the 780 by almost 50% in displaying "noticeable" frame rate drops. That number is imperfect, because one large spike can skew the number - but still the 2.5% shown by the 9th percentile frame time is in this case a huge disservice to the 7970.
So go back to my examples. My examples show that percentiles can obscure underlying behavior that is critical to making decisions. In this case, they aren't just random examples - they mirror what you see when you look at the case by case studies.
That may seem reasonable at first, but it actually doesn't tell us anything about the performance of the two cards over 40 fps and over 41 fps. In this case, this particular site claims that the average FPS for the 7970 is 50 fps. The average fps for the 780 from their numbers is 55. So looking at the average, you get a difference of 10% - which is far more significant. Note that I wouldn't argue with you if you thought that still wasn't enough to justify cost, but it is an entirely different picture than the 2.5% paints.
So the next argument is that it tells you about frame time. The problem is that it doesn't. Choosing a number in the 7970's favor this time, look at the time spent beyond 33ms (this would be 30 hz - or where you would see a noticable jump in lag on most 60 or 120 hz monitors) for something like Crisis. In this case, the 7970 only spends 441ms beyond that number, while the 780 spends 663ms beyond it. Assuming the tests ran for the same amount of time, the 7970 outperforms the 780 by almost 50% in displaying "noticeable" frame rate drops. That number is imperfect, because one large spike can skew the number - but still the 2.5% shown by the 9th percentile frame time is in this case a huge disservice to the 7970.
So go back to my examples. My examples show that percentiles can obscure underlying behavior that is critical to making decisions. In this case, they aren't just random examples - they mirror what you see when you look at the case by case studies.
