In your initial contrived example, you'd end up with a 99% percentile of 120fps. If you had used their methodology with the same contrived data, you'd end up with 1fps. Which very well reflects what a user would be experiencing. It's fair to say that your initial takedown was just a little bit flawed and a somewhat weak foundation to call their methods, and I quote, "one of the worst cases of marketing science."
I want you to look at their graph. They have labeled one of the columns. That label is "average frames/second". That is typically abreviated FPS. If you check the top of their plot, you will note that they themselves label their graph that. In other words my "initial takedown" was based
on the graph they provided. I gave them a rebuttal in the context of the plot they provided. That is pretty typical in science.
You tried to claim the problem didn't exist because I hadn't used a hidden metric rather than the actual presented metric. I simply pointed out the obvious - the problem will continue to exist regardless of the metric you use. That is because the problem is with the aggregation they chose for summary.
However, if you need to select one and only one number that has the best chance of representing what a user will experience, I think it's a pretty good proxy.
Now be constructive and come up with a better single number performance metric that can be used to calculate !/$.
Go back and read my first post. You will discover that I suggested not one - but two summary statistics that would better make their point. Interestingly enough, one of those was provided and you will note I had no problem with the second plot.
That being said, I think they are still using figures that are insufficient to make the case they want to make. I don't mind as much because the figures aren't plain wrong (as is the case with the percentile example) - they are just insufficient. There are a lot of plots that could have been done that make their point. The following would be one of them:
Here I have plotted the price vs. average performance (taken from techpowerup's site - their aggregate performance) of various NVidia and AMD cards today. I colored the AMD cards red to differentiate them from the NVidia cards. I colored the 780 OC edition blue. I then fit all of the data to an exponential function that fits the data pretty well (R^2 of .95). The prices came from Newegg as of today (I selected the lowest price available for each card in the techpowerup list).
This line is actually interesting. If you are above the line, then you are getting more performance for your dollar. If you are below the line, you are getting less. This paints the 780 OC in pretty decent light.
Of course, the problem is that AMD just has nothing to compete. So NVidia tends to define the line up here - and the 780 regular and Titan fall right on the line. Actually, it is uncanny how close they are. If I had to bet, I would guess that NVidia did a similar analysis and set their price accordingly.
In any case, the plot demonstrates a couple of things. First, the increase in performance for price tails off dramatically in the high end segment. It is almost linear at first, but then drops off rapidly. Second, NVidia actually isn't as bad off as some people are making them out to be. For the record, I expected NVidia to perform horribly in the upper segments. I was pretty surprised when they didn't. I would attribute this more to the lack of AMD cards up here to help solidify the segment more than to any silicon based success for NVidia though.
