Digital Foundry Article Technical Discussion Archive [2014]

Recently I had a chance to play Titanfall on the XB1, though there is some awful screen tearing at times, the game for the most part is quite responsive. Honestly, the dips don't bring things to crawl or take away from the action. Some gamers have a higher tolerance for framerate-drops than others ...me being one of them. Anyhow, Titanfall is quite fun and amusing ...graphic wise, it's no show stopper, but its appealing enough to be enjoyable. Score: 8/10

 

I think you do not understand why the average was ditched and why "which scenes are chosen" will always become be a weak spot in the technical discussion if they indeed say "the average is xx fps". It's such a subjective issue that they most probably wanted to avoid it altogether. The developer is the only party that is qualified to present a series of scenes for the machine to run to present an average fps. How DF is presenting it right now is spot on. Show the viewers which scenes dip, which scenes are alright, and leave it as such. Informed viewers will have a good idea what fps the game is running at.

The DF video flies in the face of what you are saying. Simply having one or two titans on screen with the player wandering around/shooting is more than enough to drop the fps down to 40. 3:40 if you can't find one of them.
Titans are pretty much commonplace in game so I don't see why having one or two of these and shooting at them constitutes "heavy action". It's like saying having two tanks/APCs fighting each other in BF4 is "heavy action". If the console can't handle the smoke, explosions, and fire that Respawn is throwing at it, then there's something wrong, and somebody needs to fix it.

As we can see from the videos, these "effect heavy" fights are in fact commonplace. Simply having one enemy in a titan and you handling/shooting anti-titan weaponry at it is enough to bring the fps down to the 40s. If this is your idea of a "demanding moment", then titanfall is obviously flooded with these demanding moments, to the point that you shouldn't even call it out as a rarity.

So you would rather have the player taking time getting a tour of the map to get a good representation of the fps?
Benchmarking software must be made by total idiots to bring the most demanding scenes to their benchmarking scenes and not give higher marks to all tests across the board.

Are you playing a fps game or a touring game? Any fps player will say the firefight is where the frame rate is most needed. DF's scenes aren't scripted to be overly chaotic, they're in fact all relatively commonplace scenes that you will run across many times on any given map/game.

What you're saying is exactly what I described here. You want to cherry picking relatively uneventful scenes to prove that the game runs smooth/close to 60fps. Doesn't work like that. We might as well as point the gun at the sky or at the ground and present a perfect 60fps frame rate test.

 

While I can agree that most COD SP campaigns didn't run solid 60fps all the time, when it comes to COD MP (which is what Titanfall should be compared against) it has always been very solid 60fps. Especially the IW / Xbox360 versions, it will be very hard to argue otherwise.

It just so happens that it's hard to record MP footage and sites like DF often record SP cinematics since it's easier to compare the recordings between different consoles etc.

 

I clearly do understand and just articulated what you just in fact regurgitated *facepalm*. But, ok? And I disagree, I want average to come back and feel while far from perfect, it's one aspect of information that is valuable. Recently they wrote in their Shadowfall writeup that the avg was 47 FPS IIRC. And I definitely appreciate that information.

And you're still being misleading and inaccurate. Specifically with saying the framerate is "50+", implying 50-60, when there's "nothing" onscreen. That's simply not true. And in fact there can be many things onscreen, such as other troops, without dipping from 60, in fact it's the norm until fire is opened and heavy alpha effects kick in.

Again we have two videos we're discussing, in the first framerate only dips below 40 for one momentary heavy Titan explosion in the whole 8:25. Which is also edited to be essentially all worst case scenarios. In the second vid the FPS is considerably worse, I'll admit, but then again it's unfair to focus on only the worst vid, we should consider them equally rather than dismissing the "good" one just because it's better.

Funny thing is you make it sound like hardly anything is going on in the game. When even "only" two Titans is a huge amount of action and effects. They are huge after all. Even one Titan could fill your whole screen if you are close enough and be on fire (as is the case at the 3:40 mark you point out), and that would be extremely demanding, it's loaded with alpha effects.

Some benchmarks on PC games are notorious for not reflecting actual gameplay demands, actually...

I'm playing a game with various gameplay, including running across the map and other less demanding things, which probably take as much or more time than the firefights. I already agree the FPS dips in fights is more important. But should we just start taking the worst second of FPS for any game and start calling that the FPS it runs at? Are we to judge KZ Shadowfall MP only by the instances it drops to 30 FPS and call it a 30 FPS game? No we look at the whole picture.

No in that specific case I was just debunking your claim that the game runs at 50 FPS when "nothing" is happening. Which was easy enough. But I also dont think cherry picking worst cases is correct either. It's a mix of worst best, and in between, and a total picture.

I hate to go to this level of no-lifeness, but I'm considering simply writing down the FPS every time it changes in the (bad) DF vid, and getting an average from that. It'll necessarily be lower than the real average because the FPS will only stay the same for multiple seconds when it's 60, and be varying only when less than 60, but it seems obvious from a glance that the average using that method will still be well above 40! For example, counting randomly from 4:00 to 4:30 and only recording when the FPS changes values (randomly chosen timeframe btw) looks like (timestamp if you want to check my work https://www.youtube.com/watch?v=ihU3J4dZNzA&feature=youtu.be&t=4m0s ) and damn that's action packed.

Spoiler

51.5
50.9
57.2
55.9
53.6
53.3
49.0
48.8
49.7
52.2
50.4
52.7
56.1
56.2
52.4
53.7
45.4
50.0
55.6
53.4
52.3
53.9
57.4
60.0
59.9
60.0
59.0
58.8

Avgerage=53.9. Real average would be higher as whenever the vid hits 60 it typically may stick there for several seconds.

I can see accusations of bias as I dont see any dips to 30's, but I swear I randomly just chose a time. Anybody is welcome to give me a "worse" 30 sec or 1 min clip timestamp and I may do it later. Probably too time consuming to do the entire vid but I may.

 

I prefer 'worst case scenario' benchmarks for MP games as they define the floor for my experience. Not an artificial worst case like a private server where everyone agrees to stand in a circle and fire alpha heavy effects at once but a few rounds online and then run the analysis. If during those standard rounds of MP low FPS spots pop up, I can say that if this is the worst I can expect to see am I happy with that?

It's clear from the initial XB1 DF article that TF does dip below 40 when things get heavy but 'things getting heavy' is why I play MP games. I don't own TF for PC but 64 player BF4 (Conquest Large 4 life ) on PS4 is analagous, it most certainly is not a 60FPS title but reading the DF review I could see that it was mostly a 50ish experience with some nasty moments. This has been borne out in play where I mostly enjoy smooth response until a lot happens at once however the awful rubber-banding BS with BF4 masks most of the framerate stuff anyway in a red mist of EA hate.

 

A lot of the game, as a pilot, is spent indoors fighting other pilots, and avoiding confrontations with titans. It's not avoiding action. It's just playing smart and playing the maps as they were designed.

 

I swear I randomly chose a time too:.2:30 to 3:00. It even had 4 full seconds of 60 fps in it!

Code:

2:30	39
2:31	37.9
2:32	38.4
2:33	38.6
2:34	41
2:35	45.2
2:36	60
2:37	60
2:38	60
2:39	60
2:40	47.4
2:41	44.1
2:42	47.7
2:43	48.9
2:44	50.3
2:45	43.1
2:46	39.6
2:47	34.3
2:48	41.7
2:49	45.5
2:50	45.5
2:51	40.9
2:52	32.3
2:53	33.7
2:54	39.1
2:55	39.9
2:56	41.8
2:57	53.7
2:58	58.3
2:59	54.6

Average: 45.41666667

Surely a 50+ fps game.

Your first video that's 8 minutes is from beta.
So we're using beta instead of the product code?
You consider Beta code (which nobody can play now) an equal of product code (where everybody is playing) when doing comparisons?
Great argument! Do please carry on.





I think we already avoided using the last titan standing video.

Unless it's clear the issue is an anomaly and totally out of the norm (like stuffing 6 titans on screen and having them do stuff, or thousands of oil drums and detonating them altogether), there's no reason that this is unfair. The video is taken from very regular gameplay.

I get more than 3 titans on screen in tutorial. I see 4+ titans in some matches on screen at once. Suddenly 2 titans is a lot of action now.


So we have from your replies the following.

Argument: the scenes I choose don't dip below average 50 fps so it's a 50+ fps game.

You basically just said that maps are designed to avoid titans and that's the smart way to play.
And you're using that as an argument to disregard titan vs titan clips, (which should be how it's designed), pilot vs titan fps clips, and titan vs pilot clips.

Thus your argument is: My playstyle allows me to play indoors without titans most of the time so fps is not a problem for me. Thus fps isn't an issue for the game.



My argument is simple: DF is providing the fps tests as is and informed viewers know perfectly well from the way it is presented what scenes will result in what range of fps. Providing a average fps is subject to cherrypicking certain scenes more than others in game and thus not an objective numeric and should be thrown out the door altogether to prevent this "I think it is 50 fps, not 40 fps like your cherrypicked scenes" scenario. The current format shows very clearly how the game performs under various scenes/loads.

We know in general that the fps is stable 60 fps when there isn't a titan around, it starts to dip into the 50s whenever titans are around or pilots are doing a bit more, and will hit 40s and even 30s when we have 2~3 titans fighitng on screen. These are pretty much objective observations and is way better than the highly subjective "the average fps of Titanfall is xx fps".

Statistically, simply having a average fps in such a test without other stats to back it up is a bad idea.
It doesn't too many statistics courses to know that averages are meaningful only when you also take into account the distribution.

 

That sounds like COD, I thought TF was about verticallity and Titans? In any case, you have a 60fps game indoors fighting a few humans on screen. Then you have an entirely different beast if you go outside and face larger draw distances and/or titans.

 

it'd be great to show 50th, 90th, 95th, and 99th percentile.
Fact is the X1 Titanfall is no where near 60fps avg, not sure what's the point of beating a dead horse.

 

Gotta wonder how much the CPU is holding things back with these spikes appearing alongside Titans.

 

I've always wondered why we don't get median or mode FPS numbers, statistics are our friends I really enjoy Tech Reports PC h/w reviews with their frametime numbers and 99th percentile but of course getting frametime numbers from consoles is probably next to impossible w/o each studios tools and debug code.

 

The median would be much better, particularly when your ceiling is 60 and where averages can be dragged down by a few crappy moments. Medians are better as a figure you'll get the most but, ideally, for those that understand them, the full distribution of fps (1-60) would be great.

 

The problem is that when you are in a titan you looking through virtual displays and titans are basically mobile devices where performance per watt is very very important.

There is only so much reservation a Titan core can devote to the displays because you have more mission critical apps running in the background like cameras, minon detectors, shields, electronic smoke, mobility, Google Maps, Office and the occasional Angry Birds. Yes, I said Angry Birds. While Titan Falls generally take seconds, guess how long it takes to deploy them back in space. What pilot, who wants to unwind after coming within inches of their life, wants to reload apps. The displays on Titans are cheap OLEDs that run at 30hz. So running at 60 fps would actually ruin the accurate depiction of what it is like to be in a Titan.

SMH at the people that just don't know.

 

So about the same average as Shadwfall MP. 45=not bad. The overall average would be a good deal higher in real gameplay, when you factor in DF vids are all demanding scenes.

Also I dont know that counting YT "seconds" is really accurate. I just count the changes. Sometimes there might be several average changes within a second that I could tell. But anyway.

Also I honestly did choose a random segment, but i have strong doubts you did

So the Beta framerate was higher than final? Actually I suppose it could have been because wasn't the beta lower textures? But I attributed the difference more to differing levels.

But DF cut it to be all fights it seems, mostly Titan fights. Not representative cross section of gameplay.

Have you seen the game? They're two TITANS. Not two COD soldiers.

I dont believe I've said that. I said I suspect the average, across say more than 10 minutes of non-cut gameplay, is over 50, and it almost certainly is, although that's not proven I suppose.

It does show clearly, but the only cherry picking is the demanding scenes in the video. The real overall gameplay should be a higher average. But even the video the average is likely over 50.

Wait a minute, so your subjective view is now better than the epitome of objective, a mathematically inarguable number? You have this backwards.

At least in this case I can mostly agree with your observations, unlike "the framerate is 50+ when nothing is onscreen".

Are you asking for the median frametime or something? We dont have that. But we do have the video to look at. The average is only more info on top of that. Why would you want less information about the games framerate? BTW you should take this up with hundreds of websites with decades of PC benchmarking, that for the most part, rely on an average FPS. It's almost like it's really important.

 

Those were nice, when it was basically an AMD "gotcha". Then AMD put framepacing in their drivers and now basically all video cards are "good" in that dept. So honestly to me they now are back to having little value. You dont see any interesting deviations there, EG the faster average FPS card basically always does better at 99 percentile times, etc.

 

I would like to see performance distributions as well, but it would be a tricky issue. You'd still have the same issues of getting a representative sampling of gameplay, and you'd have to decide what the distribution would actually be of. FPS is generally done as a time-average, so any distribution would have to make a note of what that time-average was. A direct distribution of frame times would perhaps be more "native," though it would say astoundingly little about typical behaviours (a double-buffered game occasionally toggling between 17ms and 33ms frame times might produce the same graph as a triple-buffered game sitting at 45fps) (also, you'd have an interesting question: do you weigh 33ms frames twice as high in the histogram as 17ms frames, to represent the amount of time the player spends experiencing a particular performance level?).

Maybe there's some more interesting high-level ways to map performance.

 

You think I cherrypick the demanding scenes, I think the demanding scenes are chosen for a very valid reason, because they are the scenes that actually MATTER.
You want DF to include more scenes where relatively little is going on. For what? To show titanfall runs a flawless 60fps when looking at a wall?

Which is why DF did the video with "action" in mind, instead of making a video of "inaction" to prove a point that doesn't really help anybody.
The fact that this is the way DF chose to do it and very little people complain about it actually means something.

Because average framerate is absoultely trash if you want to understand how the game performs.

If you want to see how graphics cards perform with respect to each other, using the average is fine. If you want to know how a game is running, using the average alone is one of the worst ways you can mislead people.

Are we benchmarking a GPU? NO
Do we want to know how the game runs and how it performs under various circumstances? YES

We're talking about a GAME, and these two questions require different metrics and methods.
For the former average fps is the usually way to go, as min/max usually follow suit to a certain degree, as long as the scenes rendered are identical.
We do not care if one card averages in case A 1000 or case B 10 fps, as long as another card averages in case A 1100 or in case B 11 fps, we understand that the second card outperforms the first.

For the latter there is absolutely nothing you're comparing the average fps to.
So what if the game runs 60 fps 90% of the time?
If it drops to 5 fps 10% of the time you have an unplayable game, even if the game averages out to a very healthy 54.5 FPS in this case.
But is it really acceptable? No.
Is the average fps a good representation of how the game plays? No, of course not, you need way more metrics, and much preferably, the entire distribution to give an accurate conclusion as to how the game plays.

What matter is how the scene is presented to the player, and in this respect, the worst case scenarios will undeniably leave the most impression, and, again, it is what MATTERS.
In this case, what matters is if it maintains a smooth frame rate or not. Whether the current results of 40 to 50 fps on the more demanding scenes is acceptable to up to each person's own interpretation, You can say that 40~50 fps is acceptable to you, that's your opinion and I respect your opinion. What I am against is somehow saying DF is being unfair to Titanfall. There is no point to show a couple of minutes minutes of inaction just to please people that Titanfall runs at 60fps, and DF certainly did put in scenes where it achieved 60fps to illustrate under what circumstances are you not stressing the console and the fact that 60fps segments exist.

 

But a lot of the game is still action, just not as heavy as the scenes they selected. There are really two aspects to the game in Titanfall. Framerate from inside a Titan is not as good as when you're a pilot. You spend a lot of time, as a pilot on foot, fighting over hardpoints without any titans around. I think that's where the DF disconnect comes in for people who haven't played the game. There is nothing wrong or incorrect about DFs analysis. It's perfectly valid. It's totally true of combat inside or near multiple titans. Some games I get in my titan and I'm unstoppable. I go 20-1 and I'm in my titan the whole time. In those games my min, max and average framerates are probably much lower than the games I play where I have my titan on guard mode and I spend almost all of my time on foot fighting over hardpoints. Then there are the games somewhere inbetween. I think that's all the people who have played the game a good deal of time are trying to say. It's not that they're trying to use staring at a wall or the ground as a performance metric.

 

Your sample of 53.9 fps and my sample of 45.1fps just proved that your "a mathematically inarguable number" is totally subjective and subject to cherrypicking. They are inarguable for their respective clips, but neither of them by themselves are good representations of the game.

Which is why I keep on saying that giving a average fps is bad, VERY bad, and quite frankly, counterproductive. I'm not the one requesting putting a number down for the average fps, I'm actually against it because it is precisely what causes this "disconnect" that you have just described.

Why can't we just accept from the DF videos that Titanfall drops frames whenever Titans are around and blowing up or blowing other stuff up, as I have said all along? Whether it drops to 50, 40, or even 30 really depends on the scene. All we know is that they do happen according to the frame rate test. Once you accept this, it doesn't matter whether the average is 40 or 100.

Just so you know, DF actually updates that number every second, which is why counting it every second is actually a good idea.

 

I agree with you. But at the same time, just giving a minimum without any context is misleading, because you don't know how often or how long it stays there. Some people are looking at the worst of the DF videos and thinking the game plays like that all of the time, but it isn't true. To go the opposite way and say those videos are not representative at all is also not true.