The Callisto Protocol [PC Stutter Fest]

[Remij]

It was about shader stutter. And even when they identified it, even with the help of NV engineers it took quite a while for them to track it down and fix it. Of course, now that they know what was causing the stutters and how to fix it, you see them reach out to other Indie developers to help them identify and point them in the right direction to fix it. Not only that you see them piping in from time to time to let others know that shader stutter is a problem and not necessarily an easy one to identify and fix when you're deep in development. IE - lots of bugs, lots of performance issues, lots of optimization to do, etc. So you may not know until the final build of the game that it's not a bug and instead it's due to JIT shader compilation.

Even though there are developers that keep an eye on DF videos, not all developers do and when you're an indie developer working on an ambitious project like The Ascent, you may not have a lot of time as the engine and rendering architect to pay attention to other games during development of your title.

Now, if it slips through again on their next title, I'll definitely blast them about how they couldn't miss it and fix it before release now that they know the root cause and how to fix it, but this was their first studio title which came out before there was so much attention being paid to shader stutter. So, I give them kudos for taking it seriously after their game was released and before shader stutter got as much publicity as it has.

I fully expect we'll continue to see this for a bit and I'll generally give Indie devs (like Neon Giant) more slack, especially if it's their first major title release (The Ascent).

Now someone like Epic releasing the latest Fortnite with shader stutter? They should absolutely know better by this point. The Callisto Project? Tough call. They obviously ran out of development time for any version that wasn't the PS5 version. So, who knows if shader stutter would or would not have been fixed if they actually had enough time to put into the PC version.

Regards,
SB

I understand you, I really do.

But that's about identifying why something is happening... not that it is happening in the first place.

It's very simple. You play the final build of your game on a completely fresh PC, and you see what happens. That immediately tells you if there is a problem. Within minutes of a game launching there are forum posts speaking about this particular issue when it happens. This testing should all happen BEFORE a game is launched. Discover the issue, and take as long as you need to find the solution.

This goes for ALL types of stutter. It seems to me.. like developers simply don't test their games on clean PCs to see what the actual user experience will be like. I simply can't fathom any excuse making sense to me.

I can't give any developers any slack about this. It's either a case of them not testing it, or they fully understand the issues the game is shipping with, and ship it anyway. Whether it performs sufficiently enough or not is entirely subjective... but shader compilation stutter isn't hard to notice. I guess my standards are just too impossibly high and I'm being unrealistic.... while console gamers enjoy none of this BS.

 

[swaaye]

Just think - there's probably a team of hundreds who worked on this game and knowingly dumped it out to you like it is. Some might be lurking here.

Days Gone is pretty similar. No precompilation for some reason. I don't play very many of these recent AAA ports so I don't know how prevalent this is.

 

[mr magoo]

It's very simple. You play the final build of your game on a completely fresh PC, and you see what happens. That immediately tells you if there is a problem. Within minutes of a game launching there are forum posts speaking about this particular issue when it happens. This testing should all happen BEFORE a game is launched. Discover the issue, and take as long as you need to find the solution.

This. I cannot understand this. ITs impossible to miss, to me its screams "we know and we dont care".

 

[Remij]

Just think - there's probably a team of hundreds who worked on this game and knowingly dumped it out to you like it is. Some might be lurking here.

Days Gone is pretty similar. No precompilation for some reason. I don't play very many of these recent AAA ports so I don't know how prevalent this is.

Yeah, probably.

Apparently every developer in all the studios in all the lands miss this very hard to find stutter, and still release their games. Only the gifted few can discern these specific types of phenomenon.

I mean, Capcom also apparently had no idea RE8 stuttered every time you killed an enemy either... same with when the daughters grabbed you and the game starts hitching incessantly. It took a few months, after some modders figured it out.. but I guess some Nvidia engineers finally got over to Capcom to diagnose what the problem could be. Totally understandable.. game dev is hard.

 

[Remij]

This. I cannot understand this. ITs impossible to miss, to me its screams "we know and we dont care".

Exactly. How else are we supposed to interpret it at this point?

I welcome any developer on here to walk me through how this could legitimately pass through, if the developer does the absolute least due diligence required... by playing the final shipping version, on a fresh PC.

If developers aren't doing that last step... than ANY reasoning they give is just meaningless.. because that's the absolute least that I expect from them.


The only answer is... ship it anyway.. and we'll try to fix it if people make a big deal about it.


And this is going to keep happening unless this mentality changes.


I really pitty the next semi-big Unreal Engine game that is releasing across the next gen consoles and PC. It's going to get crucified if this issue persists. PC gamers are getting pissed off at the blatant disrespect of releasing titles in this state.

 

[Malo]

Don't most of these studios outsource QA to overseas companies? I highly doubt devs themselves are loading up fresh VMs to test anything.

 

[Remij]

Don't most of these studios outsource QA to overseas companies? I highly doubt devs themselves are loading up fresh VMs to test anything.

Maybe? But regardless, QA companies that the studios hire are developers on the games too.

At the end of the day, someone from the studio signs off on this stuff.

 

[iroboto]

Don't most of these studios outsource QA to overseas companies? I highly doubt devs themselves are loading up fresh VMs to test anything.

They have internal QA as well. Iteration time too slow to have it over seas only.

 

[dobwal]

Exactly. How else are we supposed to interpret it at this point?

I welcome any developer on here to walk me through how this could legitimately pass through, if the developer does the absolute least due diligence required... by playing the final shipping version, on a fresh PC.

If developers aren't doing that last step... than ANY reasoning they give is just meaningless.. because that's the absolute least that I expect from them.


The only answer is... ship it anyway.. and we'll try to fix it if people make a big deal about it.


And this is going to keep happening unless this mentality changes.


I really pitty the next semi-big Unreal Engine game that is releasing across the next gen consoles and PC. It's going to get crucified if this issue persists. PC gamers are getting pissed off at the blatant disrespect of releasing titles in this state.

It’s obvious now such a step of testing the ship build should be included in QA but it’s not ideal for a development environment.

So if devs finalize code, compile test builds and hand it off to QA and testing shows the build is stable and relatively error free. Ideally you are good to go to build a shipping build and release. Because a ship build is just the test build with the stats, console commands and profile tools disabled.

It’s becoming more and more apparent that the compiler may produce core code that is not identical between builds. That’s extremely problematic because if you have to QA shipping builds now. How do identify and rectify issues as they disappear the moment you compile with your tool sets enabled?

This would explain why this issue may readily slip through the cracks because a pipe’s propensity to leak shouldn’t be dependent on whether or not a plumber is wearing his tool belt.

“Lazy dev” may be a viable description when it’s just an isolated case. However when an issue pops up across the market, “Lazy devs” is just a lazy assertion that avoids any thought of why a problem is systemic.

 

[Remij]

It’s obvious now such a step of testing the ship build should be included in QA but it’s not ideal for a development environment.

So if devs finalize code, compile test builds and hand it off to QA and testing shows the build is stable and relatively error free. Ideally you are good to go to build a shipping build and release. Because a ship build is just the test build with the stats, console commands and profile tools disabled.

It’s becoming more and more apparent that the compiler may produce core code that is not identical between builds. That’s extremely problematic because if you have to QA shipping builds now. How do identify and rectify issues as they disappear the moment you compile with your tool sets enabled?

This would explain why this issue may readily slip through the cracks because a pipe’s propensity to leak shouldn’t be dependent on whether or not a plumber is wearing his tool belt.

“Lazy dev” may be a viable description when it’s just an isolated case. However when an issue pops up across the market, “Lazy devs” is just a lazy assertion that avoids any thought of why a problem is systemic.

Great post which brings up some honest points. However, it still doesn't explain why developers aren't testing to see what the final code is like for someone on a fresh PC.

I get that QA'ing the "ship build" isn't ideal for them to diagnose and profile issues that it might have because those tools aren't included in the build.... however, this particular issue isn't something that needs profiling tools to detect.

It's literally a matter of.... ok why is the shipping build hitching like this... what's wrong here... we can't release this. And NOT releasing the game, until it's fixed.

At the end of the day, if you're not doing that basic test of playing the final build that people will download to their PCs.... then it doesn't matter what the hell you test during development.


Edit: You're right though. Your last two points are on the money. There's an issue there with regards to QA.. and it's worth looking into what can be done about that.

Glen just posted this:

I hope he does, and I hope he shares his information with everyone so we can help prevent this stuff from happening in the future.

 

[pjbliverpool]

Great post which brings up some honest points. However, it still doesn't explain why developers aren't testing to see what the final code is like for someone on a fresh PC.

I get that QA'ing the "ship build" isn't ideal for them to diagnose and profile issues that it might have because those tools aren't included in the build.... however, this particular issue isn't something that needs profiling tools to detect.

It's literally a matter of.... ok why is the shipping build hitching like this... what's wrong here... we can't release this. And NOT releasing the game, until it's fixed.

At the end of the day, if you're not doing that basic test of playing the final build that people will download to their PCs.... then it doesn't matter what the hell you test during development.


Edit: You're right though. Your last two points are on the money. There's an issue there with regards to QA.. and it's worth looking into what can be done about that.

Glen just posted this:

I hope he does, and I hope he shares his information with everyone so we can help prevent this stuff from happening in the future.

He's being surprisingly open and honest there. I've nothing but respect for that. I'm pretty confident they'll get these issues fixed.

 

[MfA]

They have internal QA as well. Iteration time too slow to have it over seas only.

Silly question, but it's a silly problem so I still have to ask ... they do just clone a fresh windows install on QA machines for each new build right? Right?

 

[Remij]

He's being surprisingly open and honest there. I've nothing but respect for that. I'm pretty confident they'll get these issues fixed.

Yes absolutely. He got right out in front and has been responding to tons of messages about all versions and aspects of the game. Just that communication is key.

I really hope they do. While the game has issues in is own right, I am enjoying the game itself. It still has lots of stuttering issues and literally every jumpscare is ruined by it.. but he seems determined to get things fixed.. so good on him/them.

 

[iroboto]

Silly question, but it's a silly problem so I still have to ask ... they do just clone a fresh windows install on QA machines for each new build right? Right?

Not sure. I’ve been shown the QA tester rooms in Ubisoft Toronto, but they didn’t let me into the room. I didn’t see anything beyond the door during the tours, except to know that particular hallway filled with doors is QA.

 

[mr magoo]

https://twitter.com/i/web/status/1599508890168983552

New patch. Still No rt reflections on XSX.

 

[trinibwoy]

It’s obvious now such a step of testing the ship build should be included in QA but it’s not ideal for a development environment.

So if devs finalize code, compile test builds and hand it off to QA and testing shows the build is stable and relatively error free. Ideally you are good to go to build a shipping build and release. Because a ship build is just the test build with the stats, console commands and profile tools disabled.

It’s becoming more and more apparent that the compiler may produce core code that is not identical between builds. That’s extremely problematic because if you have to QA shipping builds now. How do identify and rectify issues as they disappear the moment you compile with your tool sets enabled?

This would explain why this issue may readily slip through the cracks because a pipe’s propensity to leak shouldn’t be dependent on whether or not a plumber is wearing his tool belt.

“Lazy dev” may be a viable description when it’s just an isolated case. However when an issue pops up across the market, “Lazy devs” is just a lazy assertion that avoids any thought of why a problem is systemic.

I would bet lots of money that the QA builds also stutter. It doesn’t make sense for shipping builds to generate different code and it’s even less likely that the shader compilation behavior is any different. My guess is this was a well known issue but other issues took priority. It’s only a priority now because of all the backlash from DF and others.

 

[Below2D]

I heard it's fixed now. Anyone who has the game care to chime in?

 

[Roger Cobb]

I always used to rush out and buy horror games whenever they came out. Now I seldomly do this because some just seem rather boring. I particularly find games in space to just be boring in particular.

You see, I never got into the more popular Dead Space games, so when I noticed this was kind of similar to those, I gave it a miss. However, I may get it one day if I find I have absolutely nothing else to play.