No DX12 Software is Suitable for Benchmarking *spawn*

Thanks to another poster we also have information from DSOGaming.
What is interesting is that they are seeing performance boost with their 980ti, and quite noteably so.
This suggests to me any analysis needs to look at it in further detail to see what is the behaviour and performance; whether that be down to scene-map or settings such as lighting-shadows/AO/other post post-processing effects, or maybe CPU related *shrug*.
Some of the shadow settings had a big performance hit in the past.
http://www.dsogaming.com/news/doom-...-vulkan-versus-opengl-performance-comparison/
Anyway in one scene they are saying they see original fps 105 and post patch 160fps on a 980ti.

Curious why Guru3D and DSOGaming do not match up in behaviour, but could be many reasons.
Cheers
 
CSI PC said:
Thanks to another poster we also have information from DSOGaming.
What is interesting is that they are seeing performance boost with their 980ti, and quite noteably so.
This suggests to me any analysis needs to look at it in further detail to see what is the behaviour and performance; whether that be down to scene-map or settings such as lighting-shadows/AO/other post post-processing effects, or maybe CPU related *shrug*.
Some of the shadow settings had a big performance hit in the past.
http://www.dsogaming.com/news/doom-...-vulkan-versus-opengl-performance-comparison/
Anyway in one scene they are saying they see original fps 105 and post patch 160fps on a 980ti.

Curious why Guru3D and DSOGaming do not match up in behaviour, but could be many reasons.
Cheers
Click to expand...
Some suggest it's simply due both benching in different scenarios.
Worth noting is that DSOGaming doesn't actually show average performance on any specific scene, but just one scenario where at least in couple frames there's notable difference between OpenGL and Vulkan (there seems to be notable difference in IQ between them too, though, whether it's random weather or something else, I don't know)
 
Kaotik said:
Some suggest it's simply due both benching in different scenarios.
Worth noting is that DSOGaming doesn't actually show average performance on any specific scene, but just one scenario where at least in couple frames there's notable difference between OpenGL and Vulkan (there seems to be notable difference in IQ between them too, though, whether it's random weather or something else, I don't know)
Click to expand...
Agree it could be map-scene.
But the point is it increased by 50%, but we do not know if it is the map-scene or indeed settings, it did not come across as if it was just a couple of frames in his writing - but that does not remove it being large map-scene zones and possibly linked to AO/lighting-shadows.
Guru3D made it sound like they saw no improvements at all, however it is most likely they both only tested in a single zone, and I am pretty sure neither played around with the settings such as the shadows/alias settings/etc.
The other aspect is that DSOGaming mention the game went from primarily single threaded to fully multi-threaded with patch with their CPU, so again something possibly unusual there.
Cheers
 
Well AA settings do have implications on Doom and can be good/bad depending if AMD or Nvidia I guess (maybe performance variance for Nvidia depending if engine trying this or not).
To enable Async Compute for Doom you need to set AA to either TSSAA or No AA.
Anything else disables Async Compute.
youtube.com/watch?v=cjgFNkNW8zI … DOOM GL vs Vulkan on an awesome AMD 480. Heads up benchmarkers, use TSSAA or no AA (else Async Compute is disabled)
Click to expand...
https://twitter.com/idSoftwareTiago/status/752590016988082180

Cheers
 
ToTTenTranz said:
Why is that a DX12 specific problem and not something generally related to internal benchmarks?
Click to expand...

Oh but it is! It is just that DX12 games seem to give rise to integrated benchmarks once again. FWIW we do not use integrated benchmarks at all, if they can be avoided. Not TR, not RotTR, not Metro, not AotS, not Hitman, not Batman - only maybe in order to check and mostly disprove their usefulness outside the developers space, where they of course may be useful for tweaking their graphics pipeline. In fact isuspect those integrated benchmarks to be just that: internal graphics analysis, that somehow ended up in the final game code, giving customers (through uncaring press!) false context for their hardware buying decision. (Yeeah, just get that shiny new graphics card for that strategy game with thousands of units. What? Nono, you will be fine with that Pentium G for 60 EUR.)
 
I am fairly sure my colleague used Fraps - which works like it does in other games, only the overlay does not get displayed.
 
CarstenS said:
Oh but it is! It is just that DX12 games seem to give rise to integrated benchmarks once again. FWIW we do not use integrated benchmarks at all, if they can be avoided. Not TR, not RotTR, not Metro, not AotS, not Hitman, not Batman - only maybe in order to check and mostly disprove their usefulness outside the developers space, where they of course may be useful for tweaking their graphics pipeline. In fact isuspect those integrated benchmarks to be just that: internal graphics analysis, that somehow ended up in the final game code, giving customers (through uncaring press!) false context for their hardware buying decision. (Yeeah, just get that shiny new graphics card for that strategy game with thousands of units. What? Nono, you will be fine with that Pentium G for 60 EUR.)
Click to expand...

I still don't understand why this is a specific DX12 problem. Pre-scripted runtimes have been known to not be representative of CPU usage in games since ever..

Is it because the 3rd-party tools that were used to benchmark DX11 games don't support DX12 games?
 
ToTTenTranz said:
I still don't understand why this is a specific DX12 problem. Pre-scripted runtimes have been known to not be representative of CPU usage in games since ever..

Is it because the 3rd-party tools that were used to benchmark DX11 games don't support DX12 games?
Click to expand...

Maybe it helps if I highlight a certain part for you?
„DX12 games seem to give rise to integrated benchmarks once again“.

No? Care to compare the ratio of DX12 games with integrated benchmarks to DX11 games?
 
CarstenS said:
No? Care to compare the ratio of DX12 games with integrated benchmarks to DX11 games?
Click to expand...
Then it's a matter of more developers shipping new games with an integrated benchmark, got it.

Maybe it's because reviewers are asking for them? Or IHVs because of new cards and a new API?
Does it occur more often now than it did during the DX11 early days? Or DX9's?
 
CarstenS said:
I am fairly sure my colleague used Fraps - which works like it does in other games, only the overlay does not get displayed.
Click to expand...
Could you ask him if he could do them again with PresentMon, just to be sure the results are indeed correct?
https://github.com/GameTechDev/PresentMon

Just running results again for couple cards should show if the results are notably different, or close enough to prove Fraps numbers are correct.
With all the issues there has been with DX12 benching, I wouldn't trust a program that officially doesn't support DX12 at all
 
ToTTenTranz said:
Then it's a matter of more developers shipping new games with an integrated benchmark, got it.
Maybe it's because reviewers are asking for them? Or IHVs because of new cards and a new API?
Click to expand...
That's why I wrote: „internal graphics analysis, that somehow ended up in the final game code, giving customers (through uncaring press!) false context for their hardware buying decision.“ (my bold)
Even lazy press should care enough not only to produce benchmark charts but to select them in a way that gives them a meaning for the customers.

ToTTenTranz said:
Does it occur more often now than it did during the DX11 early days? Or DX9's?
Click to expand...
Let's see: Apart form Windows Store games (Gears, QB, Forza - for obvious reasons), every title seems to sport one: Aots has it, RotTR has it, Hitman has it, Warhammer has it (and had even a Standalone-Demo before). Heck, Fable Legend even had a benchmark without a game. I strongly suspect Deus Ex Mankind Divided. The only title I can come up with right now that has none but could have is the indie game Caffeine.
 
CarstenS said:
Specifics? I certainly can and am wiling to help there, but i cannot translate the whole article (timewise)
Click to expand...
Just a curious thing when viewing the graphs using google translate and IE at your site. The graph dropdowns don't seem to work(using google translate) and the current solution is flip/flop between two web pages when reading reviews (translated - for reading / original - for modifying graphs) . Kind of weird but only seems to happen at your website.
 
CarstenS said:
I am fairly sure my colleague used Fraps - which works like it does in other games, only the overlay does not get displayed.
Click to expand...
As Kaotik mentions,
any chance that a comparison can be done between FRAPs and PresentMon as they do measure at different data points, and I have seen some measurements in FRAPs that seem unusual in DX12.
Maybe this can be done as well for say one of the worst performing DX12 games as it makes it easier to also compare FRAP/PresentMon to visual-gaming perception and which 'feels' more representable to the real-world gaming.
I assume also part of the headache with PresentMon is deciding best way to analyse the data as it may be possible for it to be skewed depending upon how averaging/'sampling resolution' (yeah I appreciate it is not technically doing this)/etc.

In fact there needs to be a test to ensure that either program actually measures 'accurately' (yeah both are accurate but measure differently) for both DX11 and DX12, by that I mean that it is unknown if the way they work means the data can be correctly compared between the different APIs, definitely an issue if using separate solutions for each API.
Shame so far it seems no internal benchmark can be trusted to give just the numbers of what the gamer sees-perceives to match against the independent tools.
Need some kind of DX11-DX12 'calibration' utility to compare these measurement tools.
Thanks
 
Last edited:
Sorry for OT, I'll keep it brief: Our tech department says, there's no easy solution to this and potentially we'd have to chase after google whenever they do changes to their translate algorithm. But the team is looking into it once more. Sorry for the inconvenience. We hope to have more content here in the future though: http://www.pcgameshardware.de/EnglishArticles/

Kaotik said:
Could you ask him if he could do them again with PresentMon, just to be sure the results are indeed correct?
https://github.com/GameTechDev/PresentMon

Just running results again for couple cards should show if the results are notably different, or close enough to prove Fraps numbers are correct.
With all the issues there has been with DX12 benching, I wouldn't trust a program that officially doesn't support DX12 at all
Click to expand...
We did a comparison with the standalone benchmark's integrated measurements and Fraps already. According to my colleague the results are within the margin of error, but he did not explicitly repeat it for the final game.
 
ToTTenTranz said:
Then it's a matter of more developers shipping new games with an integrated benchmark, got it.

Maybe it's because reviewers are asking for them? Or IHVs because of new cards and a new API?
Does it occur more often now than it did during the DX11 early days? Or DX9's?
Click to expand...
It could be simple marketing. When a new API is released reviewers are looking for content so an internal benchmark is an easy way to keep your game being talked about.
 
Could also be a source of marketing revenue from IHV's to gaming companies to use specific internal benchmark maps that favor their product.
 
You must log in or register to reply here.

Similar threads

R
id Software releases Open-Source Wolf3D iPhone port
Replies
13
Views
6K
Ailuros
Ailuros
O
Unreal creator Tim Sweeney: "PCs are good for anything, just not games”
3 4 5
Replies
91
Views
18K
Mobius1aic
M
A
Full transcript of John Carmack's QuakeCon 2005 Keynote
3 4 5
Replies
98
Views
34K
nemisis
N
Back
Top