Gaming Journalism

[Shortbread]

SMSRender Team

We commented on the original article because there were factual errors and also because Ian (the CEO) wanted to respond after internal commentary by our community over at WMD.

Some of the errors we highlighted were:

- You stated that the game was using FXAA when it uses EQAA
- You stated that PS4 was using object based motion-blur when it does not - the motion-blur is the same between PS4/XB1. (The additional PS4 temporal AA step is not object based blur!)
- We stated that high numbers of AI could cause CPU bound scenario's on XB1 and we used the 7th core to eliminate these cases. This is somewhat acknowledged in your updated article.
- We also stated that tracks that had water elements (approximately 35% of the game e.g. Azure Coast, Circuit etc) had a significant optimisation with screen space reflections gaining >20% performance. This performance improvement is not reflected in your update.
- We've gone over here how our tracks are layered with different levels of Anisotropy. If there's more general feedback that we need to improve here, we will. *EDIT* I see there may be some confirmations of the white lines mentioned now in some cross posts.

As a graphics programmer with around 20+ years of AAA development I would not dream of making the sort 'concrete' analysis you do without using a GPU debugger - static image analysis is easy to get wrong, because it provides only limited information. It would make for vastly more accurate and interesting Journalism if you could use 1st Party tools to do a more in-depth analysis, don't you agree?

I'm starting to wonder if gaming review sites (their journalists) are doing a disservice when they don't understand a particular technique/effect/render being done... and make poor assumptions and judgment calls that can lead to false information being spread and potential harm to sales.

Should gaming review sites (their journalists) be held accountable for poor information or the lack of knowledge on a particular subject?

Or

Should gaming review sites hold-off on technical jargon if they have no understanding of it?

 

[Deleted member 11852]

I personally value the work Digital Foundry do but I think they need to be clearer that their 'analysis' is merely a possible explanation for the things they observe. The articles do come across very matter of fact and, as the SMSRender Team guy states, much of the opinion they express is difficult, perhaps impossible, to be definite about without a better idea of what the CPU and GPU are doing.

They've had a fair few retractions and corrections in their articles this generation. I don't know if this is just because it's more complicated now or because it's always been a bit hit and miss but most developers just don't read the site, or if they do, are not in a position to comment and correct.

 

[Cyan]

SMSRender Team



I'm starting to wonder if gaming review sites (their journalists) are doing a disservice when they don't understand a particular technique/effect/render being done... and make poor assumptions and judgment calls that can lead to false information being spread and potential harm to sales.

Should gaming review sites (their journalists) be held accountable for poor information or the lack of knowledge on a particular subject?

Or

Should gaming review sites hold-off on technical jargon if they have no understanding of it?

Well, I don't think it's the lack of understanding, but prudence would demand a bit more realistic, cautious commentaries. Ever since I started to read DF articles I was wondering "how did they found the kind of AA featured in this game? That's impossible".

It was obvious that they didn't have that information, because they are human and didn't develop the game.

I always wondered which kind of AMAZING AA Skyrim used when it launched. The Xbox 360 version looked as clean as the cleanest Xbox 360 game ever. And I expected that Digital Foundry explained it, because the IQ was incredibly crisp, yet there wasn't a jaggie in sight, iirc.

But I never got the answer from the articles. I think they said FXAA, but I don't think it is that rather mediocre kind of AA. The game was both crisp and jaggie-less.

I hope that the Project Cars analysis becomes a turning point, because it was a low point when they didn't offer apologies once the SMSRenderTeam gave a detailed explanation of what was wrong with the article, yet it was never edited.

That's unfair, and it's something I mentioned when the article was published.

 

[Shortbread]

Could the answer be to allow the developer(s) first read of the article, before posting? Thus, giving the developer the option of answering and giving factual details where needed.

 

[Goodtwin]

I think Digital Foundry should at the very least reach out to these developers to get as many technical questions answered before publishing the article. Im sure many developers will choose not to comment, but many of them might, so its worth "trying" to get answers instead of simply making assumptions from their own observations. There are probably some developers who would actually be thrilled to talk about their work in detail.

 

[Deleted member 11852]

Could the answer be to allow the developer(s) first read of the article, before posting? Thus, giving the developer the option of answering and giving factual details where needed.

I can't see many developers wanting to do this. DF are, after all, a commercial operation. While we as readers want accurate analysis, it's not developer's jobs to proof and correct DF's analysis. I think the current situation where they publish and undergo a live peer review from readers and, sometimes, developers works. When DF miss something they get called out.

They need to write using less definitive language.

 

[Shortbread]

I can't see many developers wanting to do this. DF are, after all, a commercial operation. While we as readers want accurate analysis, it's not developer's jobs to proof and correct DF's analysis. I think the current situation where they publish and undergo a live peer review from readers and, sometimes, developers works. When DF miss something they get called out.

They need to write using less definitive language.

I would think this day and age, especially dealing with social media, that developers would welcome the opportunity to review articles (answer) beforehand, rather than after.

But I get what what you are saying....

 

[Deleted member 11852]

I would think this day and age, especially dealing with social media, that developers would welcome the opportunity to review articles (answer) beforehand, rather than after.

I imagine there is probably a mix of:

• Devs like SMSRender Team, who are willing and not under a publisher's cosh.
• Devs who simply don't have the time, or where the expertise is on hols following 2 months of crunch.
• Devs who would like to but where the publisher or PR folks wouldn't allow it.
• Devs who don't give a toss.
• Devs who have been eaten by raptors.

It only works if everything contributes. And I think it's good for DF when devs point out errors and mistakes like this. DF may be unaware of their analysis errors so this type of feedback, if taken constructively, should lead to better analysis in the future.

 

[tuna]

I think that Digital Foundry is one of the few places in the gaming press that does any good work. But they will get things wrong and it is great if people correct them.

However, having rhetorical questions like "wouldn't it be better if you had access to PS4 dev kits?" is just an attempt to belittle DF and doesn't really do SMS any favours.

 

[dobwal]

Could the answer be to allow the developer(s) first read of the article, before posting? Thus, giving the developer the option of answering and giving factual details where needed.

Agreed. Joseph Pulitzer's three rules of journalism were "accuracy, accuracy and accuracy". And the best source of accurate information are the devs that work on the game in question. And even though its not a given that devs will be cooperative, there should at least be an attempt of fact checking with them first. Secondary sources could be other devs but DF seems to strictly depend on the worst of all sources. Themselves.

The number of gamers that question DF analysis will over grow over time and trust will eventually be totally eroded as the errors keep popping up in the articles.

Just because this is the internet doesn't mean the basic rules of journalism don't apply.

 

[Deleted member 13524]

However, having rhetorical questions like "wouldn't it be better if you had access to PS4 dev kits?" is just an attempt to belittle DF and doesn't really do SMS any favours.

They're supposed to be a professional team. Are the PS4 and XBOne devkits that much expensive?

 

[iroboto]

They're supposed to be a professional team. Are the PS4 and XBOne devkits that much expensive?

Looking back at the older fat PS3 Dev unit. Yes.

I don't know how many offered are free. But they are likely not as cheap as the retail units.

 

[dobwal]

Looking back at the older fat PS3 Dev unit. Yes.

I don't know how many offered are free. But they are likely not as cheap as the retail units.

Unless eurogamer is a barely profitable mom and pop enterprise or dev kits are insanely expensive, I don't see how they can't afford them.

My question is what exactly does a dev kit debugger offer when you talking about using it on someone else's software? I am under the impression that having a dev kit won't allow you to simply buy any game off the shelf and get intimate access to the underlying software.

 

[AlNom]

I imagine there is probably a mix of:

• Devs like SMSRender Team, who are willing and not under a publisher's cosh.
• Devs who simply don't have the time, or where the expertise is on hols following 2 months of crunch.
• Devs who would like to but where the publisher or PR folks wouldn't allow it.
• Devs who don't give a toss.
• Devs who have been eaten by raptors.

It only works if everything contributes. And I think it's good for DF when devs point out errors and mistakes like this. DF may be unaware of their analysis errors so this type of feedback, if taken constructively, should lead to better analysis in the future.

That's pretty much the gist of it. Publishers can be rather fickle from what I understand. A number of times DF will reach out, but they don't bite because "such and such flaws pointed out" means they'd have to acknowledge/own up to it.

Of course, you get exceptional cases where the developer can fix the issue and then it's good PR like the frame pacing.

 

[dobwal]

That's pretty much the gist of it. Publishers can be rather fickle from what I understand. A number of times DF will reach out, but they don't bite because "such and such flaws pointed out" means they'd have to acknowledge/own up to it.

Of course, you get exceptional cases where the developer can fix the issue and then it's good PR like the frame pacing.

Still the attempt should be made. No need to run into cases of having devs point out the errors of your analysis in public, when it could have been handled in private.

 

[AlNom]

True enough...

 

[iroboto]

Unless eurogamer is a barely profitable mom and pop enterprise or dev kits are insanely expensive, I don't see how they can't afford them.

My question is what exactly does a dev kit debugger offer when you talking about using it on someone else's software? I am under the impression that having a dev kit won't allow you to simply buy any game off the shelf and get intimate access to the underlying software.

hmm IIRC:
When I last used the PS3 one, and I believe this applies for PS4 as well, you cannot play retail games with them. They could only run debugged code, if that makes sense. For instance I was playing indie games on it that was not released code, so the debug version was on the SDK unit, but not the retail version.

With regards to price, I think back then the price ranged up to 1500 or 2500 (memory is not serving me well here) CAD. So depending on which dev kit you got the cost would go up or down. Different kits had different features, but mainly I believe it had to do with memory.

 

[orangpelupa]

review in general can refer the graphic as "looking good" etc without touching the specific technicalities. But DF is the opposite, they are the investigator for technicalities. I think they need to properly explains some of stuff that need to be explained. like when they know game X use this or that AA....

i think a few DF article also tries to explain the AA method (the image looks clean but the wire lines are jaggies, this probably means it use blah blah blah) but most of them just "its blurry FXAA, its this, its that".

 

[Arwin]

I am still finding at times that I am not a fan of AA. In Project CARS, I had MSAA turned on, but turned it off, and the framerates are better, the details can be cranked up higher, the colors and lighting pop more, etc. Hmm. The shimmering in 1080p isn't that great to begin with. Starting to think more and more that the benefit of AA is linked to the ppi (which is probably a bit obvious, but I'm not sure everyone realises it).

 

[orangpelupa]

^^ playing X360 on 19" 1440x900 monitor already looks very clean