Is the difficulty of debugging complex games non-linear?

Cars no, rockets yes. Modern pilotless systems employ very different physics models to reach their destination and the physics models get refined with more testing (usage). Modern missiles can adapt to different altitudes, environments (temperature, humidity) and factor into their mission execution meteorological parameters too...

I work in automotive engineering, and in fact part of my work is QA (well, I do testspec and testimplementation). And I can tell you... there is more than you'd like. In a networked car (modern car, that is), you'll have 50+ ECUs communicating with each other at all times...
You're both talking very much about software. ;) When Rodéric used cars and rockets as examples, I understood him to mean as examples of physical engineering where physical testing produces bug-free products, ergo correct testing of software can produce bug-free products in the same vein. I've just Googled how many parts a car is made out of and Toyota says 30,000. 30,000 lines of code is nothing, and that 30,000 lines of code could well be dependent on thousands/millions of lines of library code you don't have control over, and countless lines of OS code you have no control over, and potentially the many interactions between those parts of code in parallel. I recall a Naughty Dog bug on PS1 where the memory card was wiped by save games thanks to a controller timing issue or somesuch. The level of bizarreness in outcome doesn't happen in other fields AFAIK, save maybe the most esoteric of quantum sciences.

I dare say 30,000 lines of bug-free code is doable on the simplest of processors if you're not trying to do anything too adventurous. If you start doing really funky things to extract performance or capacity from very limited components such as self-modifying code and poke/peeking to memory addresses, even only 30,000 lines of code could be extremely to keep 100% bug free.

Of course, Rodéric is more saying that where a car is 30,000 components, each component is a self-contained unit made and tested to spec, each functional piece of code can and should be constructed similarly, so the final software is an assemblage of working parts that can be trusted to work because each part does. But I'm not convinced game development can be economical produced in such fashion.
 
It was a q/a tester for dice (sorry my memory isnt what it was)
http://bf4central.com/2013/11/ea-pushed-battlefield-4-quality-control/
http://bf4central.com/2013/11/ea-pushed-battlefield-4-quality-control/

This person is saying that they did not know about the bugs because they did not have time to adequately test base features. That points to project management issues but what it does not imply is that EA pushed them to release with known bugs. This happens, though, especially on ambitious (over-ambitious?) projects like BF4.

No they are not, and as for my asertion there's a mindset well youve just demonstrated it

The mindset is that software has to ship at some point, and it will not ship if you address every known bug. So you address the most important ones and leave the least important ones or the risky fixes alone. This is how all software works, so I'm not sure why it'd be unethical for games to work this way.

Your not seriously offering that as a defense are you "sorry we forgot to fit the brake pads to your wife's car and she died, but Maui was looking particularly lovely that week" :D

You're being a bit ridiculous here.
 
You're both talking very much about software. ;)

With missiles, the software and hardware are developed together. You don't build the pointy doom bomb then install Microsoft Windows Death From Above Edition ;)
 
With missiles, the software and hardware are developed together. You don't build the pointy doom bomb then install Microsoft Windows Death From Above Edition ;)
Of course not. :rolleyes: *sigh*

You use a decent Linux distro with homebrew warhead drivers built from an outdated Github repository...
 
This is a very sensitive issue, so I'll have to be pretty vague... don't connect the following with any specific projects or studios, as I talk to a lot of people officially and unofficially.

Producers are definitely aware that the game code they release is far from perfect - "held together by tape" and such. Programmers do protest against it, too.
But they have to balance this with the need to be able to pay all their people, which obviously becomes problematic if the title misses its release window.


I believe what we're seeing now is a transition period, where the product planning people haven't yet adjusted to the realities of nextgen development. Compromised releases will obviously not do well; like with ACU where Ubi had to scrap a probably large amount of revenue with the free DLC and such. They can obviously see that it is in their best interest to produce stable games, so I fully expect them to take action.

On the other hand, this absolutely doesn't mean that studios will plan for additional development time; it's far more likely that in the near future they'll look at cutting features instead. As long as people keep buying yearly releases of the big franchises, they'll probably will prefer to stick to publishing them.
Of course we also have good examples, like Activision extending COD's development cycle by an entire year, or Bioware adding a completely new franchise on top of DA and ME.

The point is, I guess, that this is a pretty well known issue within the industry.
 
The mindset is that software has to ship at some point, and it will not ship if you address every known bug.
Yes it will it will just take longer

so I'm not sure why it'd be unethical for games to work this way.
I'll explain since you don't understand, Selling faulty goods is not ethical And no, other publishers do it is not a defence

You're being a bit ridiculous here.
Sorry I thought You were offering the following a defence.
"Do you know how much worse it is to be stuck fixing a title that's gotten a reputation as a bug fest when you should be on a beach somewhere celebrating its success?"
 
Humans can only grasp and process limited complexities. Game code is so complex that even deciding if something's buggy or not becomes equivalent to:
http://en.wikipedia.org/wiki/Halting_problem
So if you want to guarantee the game is bugfree, you won't finish it before the heatdeath of the universe.

I don't think "we" want to argue that shipping with bugs is okay, and that "we" are professionally indifferent to it, quite the contrary; but that shipping with some sort of bugs is unavoidable, because of the impracticability of fixing it beyond some threshold. Everything needs to be practically realizable and economic, even relationships (which would be ethically on the top spot IMHO), or they can't be real.
There are no good bugs, of course not, but there are acceptable bugs, and there are shamefully lots of stupid bugs. There are bugs because of the absolute limits of doability, and there are ... well the other bugs, which are (possibly) avoidable mistakes.
 
So @forumaccount are you really implying that Ubi devs didn't know that AC:U has all the problems before its release and are basically surprised about the bugs and thought that everything is fine??
 
So @forumaccount are you really implying that Ubi devs didn't know that AC:U has all the problems before its release and are basically surprised about the bugs and thought that everything is fine??

I haven't talked to anyone at Ubi about it (and I certainly don't speak for them), but having been in a similar situation in the past I think that the most plausible explanation is that they weren't aware of most of the major bugs. It's hard for people to understand until you look at numbers. Say 5 million people bought AC:U and played it day one. On that day let's say that 5000 people ran into serious issues. That's less than one percent of players, but those 5000 players can and will (and even should) make a ton of noise about the issues. A bug that occurs less than 1% is very easy to miss in development.
 
Again, and this is now pure speculation on my part, Ubi might have even been aware of those bugs, but missing their launch window could have easily cost them more than even the revenue they've lost in the aftermath.

This is not the kind of decision that any of us would ever want to make. Literally thousands of people's lives could have depended on it.

Not trying to say that it was a good thing to do - only that disappointed gamers are probably not able to fully appreciate the stakes involved.
 
Again, and this is now pure speculation on my part, Ubi might have even been aware of those bugs, but missing their launch window could have easily cost them more than even the revenue they've lost in the aftermath.

This is not the kind of decision that any of us would ever want to make. Literally thousands of people's lives could have depended on it.

Not trying to say that it was a good thing to do - only that disappointed gamers are probably not able to fully appreciate the stakes involved.

Aside from retailer penalties (and I don't know if those even still exist), their launch window didn't seem like an especially notable or valuable spot in the year and they probably could have bumped. Ubisoft has shown that they'll take the hit if they think the situation warrants it. Watch Dogs being pushed back was probably a much bigger deal. That was targeted for a console launch and had bundles lined up. That's a lot of stuff to throw away, but they knew they needed to.
 
But this paints a very very dark picture of the capabilities of the AC:U devs to be honest and would imply that Ubi really needs changes. Same for Evolution and Driveclub.

@forumaccount: you paint a depressing picture of high incompetence in the gaming industry...and I really support @Roderic's opinion that game devs need to learn QA.

And it is ridiculous to me how game development is compared to high tec industries...if in aerospace industry people would work like game devs, every day 10,000 people would die in airplane crashes...every day over 100,000 users debug in over 10,000 flights.

My uneducated opinion: with the last gen and the ability to patch console games after release, game devs and publishers got increasingly more easy with shifting the definition of C bugs to save money or deal with miss-management when deadlines are in danger.

Not much happend the last gen as consumers just accepted it with only a little bit of internet moaning, devs didn't felt the need to improve their QA departments in the same way the complexity of the game tec increased. It just happens recently that gamers and media outrage makes so much noise that it hurts publishers financially...thus it is clear to me that we'll see a reaction from game devs were they substantially improve and invest into QA...
 
Say 5 million people bought AC:U and played it day one. On that day let's say that 5000 people ran into serious issues.
thats very generous of you I recon the truth is more like
"Say 5 million people bought AC:U and played it day one. On that day let's say that 5 million people ran into serious issues"
Its not running well on any platform even overclocked i7's with twin 980's

This is not the kind of decision that any of us would ever want to make
True I know ive been there (the reason I gave up being a programmer) but fixing or not releasing is the only decision you can make because ripping of your customers can never be justified...
 
True I know ive been there (the reason I gave up being a programmer) but fixing or not releasing is the only decision you can make because ripping of your customers can never be justified...
That's an ethical assertion. In the world of free commerce, if your customers are willing to be ripped off and continue to pay, there's no reason to change habits. The only reason for Ubisoft et al to stop releasing seriously bugged games is if it negatively impacts their earnings. If releasing bugged games and patching later makes more money than engineering less-buggy software, that's what they should do as a business operating in the free-market economy according to the social rules.

A discussion on the morality of that belongs in the RSPCA forum.
 
But this paints a very very dark picture of the capabilities of the AC:U devs to be honest and would imply that Ubi really needs changes. Same for Evolution and Driveclub.

@forumaccount: you paint a depressing picture of high incompetence in the gaming industry...and I really support @Roderic's opinion that game devs need to learn QA.

And it is ridiculous to me how game development is compared to high tec industries...if in aerospace industry people would work like game devs, every day 10,000 people would die in airplane crashes...every day over 100,000 users debug in over 10,000 flights.

My uneducated opinion: with the last gen and the ability to patch console games after release, game devs and publishers got increasingly more easy with shifting the definition of C bugs to save money or deal with miss-management when deadlines are in danger.

Not much happend the last gen as consumers just accepted it with only a little bit of internet moaning, devs didn't felt the need to improve their QA departments in the same way the complexity of the game tec increased. It just happens recently that gamers and media outrage makes so much noise that it hurts publishers financially...thus it is clear to me that we'll see a reaction from game devs were they substantially improve and invest into QA...

It doesn't help to fetishize other software industries. Government and defense projects are massively flawed. I went to some training with a number of Raytheon and Boeing engineers and the stories I heard don't paint a picture of a software development utopia. Even the medical industry has huge issues. Recently I heard of a hospital (I think UCLA?) that had major issues rolling out a /preexisting/ medical records package.

The depressing reality is that software (and hardware!) development is highly susceptible to bugs and errors across every sub-industry.

There's a big problem with tying this to competence and that problem is that you don't actually have any idea what Ubisoft's QA and testing processes are like, or DICE's, or mine. You don't have any idea what the root cause is of any of the bugs you're complaining about. The people with that knowledge are professionals and are capable of tracing those things to the root problem and making corrections. Everyone else is just speculating. Even me... I'm just making guesses educated by my own experience.

The future looks better, though, for a few reasons. For many years tool development for C and C++ had stagnated, but things are improving in this area and a lot of the LLVM toolset and related tools are becoming available for games. Rust is not ready for production yet but it looks very promising and if adopted has the potential to eliminate thousands of these little bugs straight away. I think even the movement of code from CPU to GPU has potential to improve stability, and doing multithreading well requires a base of experience that didn't exist in 2004 but does now.
 
thats very generous of you I recon the truth is more like
"Say 5 million people bought AC:U and played it day one. On that day let's say that 5 million people ran into serious issues"
Its not running well on any platform even overclocked i7's with twin 980's

If you're talking about PC the big elephant in the room is graphics drivers, which regardless of how competent you are as a developer and how much you test can completely screw you. Moving to things like Mantle with lighter drivers can help resolve those issues but Mantle hasn't really panned out yet in any useful sense so drivers will remain a big problem for the foreseeable future.

I'm a console developer so I don't typically deal with it, but it's a really nice feeling realizing that Nvidia or AMD completely broke your app and there's nothing you can do about it other than wait for them to fix it.
 
It doesn't help to fetishize other software industries. Government and defense projects are massively flawed. I went to some training with a number of Raytheon and Boeing engineers and the stories I heard don't paint a picture of a software development utopia. Even the medical industry has huge issues. Recently I heard of a hospital (I think UCLA?) that had major issues rolling out a /preexisting/ medical records package.
I can add to that. In the UK there's a pathology package that has had frequent serious bugs resulting in desperate rushed patches, and to this day contains some potential lethal bugs regards data corruption. But it's a private industry and the management doesn't want to spend on testing or decent product cycles. And an academic recently brought in to help engineer the software is wanting techniques that'd result in insane amounts of time to produce a final, working product - it's not viable given a working product in use. So in this medical industry, with lives at stake, the software is being patched together to work as best it can, with engineers desperate to do a good job and management pissing about trying to make money.

Also, I suddenly remembered the special edition PS2 hardware that couldn't run some games. That was a bug at the electrical-engineering level using properly tested components built to spec, but the interactions being unpredictable and resulting in deviant behaviour from that expected. Unless one is omniscient, bugs are going to happen!
 
It doesn't help to fetishize other software industries. Government and defense projects are massively flawed. I went to some training with a number of Raytheon and Boeing engineers and the stories I heard don't paint a picture of a software development utopia.

I've worked both sides of this. The problem if you're a contractor like BAE Systems on a long project like building super carriers for the UK, is that the projects will span multiple Government administrations who have different ideas on spending. As a contractor you never get the money up front so your start out heading for goal X and find that suddenly there isn't money to complete the next stage to have to retarget a new goal and the changes are, that'll change too. Vast amounts is wasted on underlying foundations for things that are no longer needed. Then there's projects like Typhoon where you have a number of contractors across a number of countries all involved. Then the consortium tries to tell the final product to other countries who need capabilities modified and the project gets drawn out long than it was originally budgeted for.

From the other side, commercial contractors can be a pain in the arse. They are commercial organisations and their bottom line is their priority. Theres no incentive to excel, only to meet minimum specifications. They are liable to reorganise and engage in mergers, or sell elements of the businesses to other organisations which complicates everything where they have access to sensitive material. Just maintaining basic security (in Government terms) is hugely complicated.

However in terms of software development, which I've done in the defence private sector, as long the goals don't change, it's not different to anywhere else. We had specific disciplines and development approaches intended to eliminate software being a case for failure. But then we're were turning out new software for missile packages every few years - much of it was iterative improvements and even then. The testing was convoluted. Certainly more went off target on changes to the hardware package than the software side but with software you dealing with far more known quantities than materials, fuels, explosives, pressure, altitudes, heat, humidity, magnetic fields, dust, first, water, sun.
 
Back
Top