Is the difficulty of debugging complex games non-linear?

Discussion in 'Console Technology' started by Commenter, Nov 13, 2014.

  1. Commenter

    Newcomer

    Joined:
    Jan 9, 2010
    Messages:
    234
    Likes Received:
    17
    Considering AC: Unity had been in development for 1500 years with a team of ten thousand people (ok slight exaggeration), are we right to demand completely bug-free experiences that run at 1080p / 60fps? The testing and debigguing of a piece of software has to be be balanced against the costs as well, and if the increase in potential bugs and glitches is non-linear or even exponential to the increase in complexity then will glitchy AAA titles that need subsequent patching become more hte norm than the exception?
     
  2. TheAlSpark

    TheAlSpark Moderator
    Moderator Legend

    Joined:
    Feb 29, 2004
    Messages:
    21,917
    Likes Received:
    7,829
    Location:
    ಠ_ಠ
    The difficulty of debugging non-linear games is complex, yes. :p
     
  3. Commenter

    Newcomer

    Joined:
    Jan 9, 2010
    Messages:
    234
    Likes Received:
    17
    That's no what I meant. I mean will the difficulty of rooting out all bugs in a game go up exponentially with games written from the ground up to take advantage of the new hardware?
     
  4. Ethatron

    Regular Subscriber

    Joined:
    Jan 24, 2010
    Messages:
    874
    Likes Received:
    278
    The difficulties to stop making a game (aka deadline) raise exponentially with the amount of money (or time, or people) believed to be available.
    The answer to you question is in the consequences of the acting upon this belief.
     
  5. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    When many crafters work semi cooperatively and you don't have solid documentation, it's usually hard to get things to work as intended for sure.
    Making programs, and games in particular, is not at an industrial stage with processes, testing protocols in place and everything else, but rather at the crafting level with many crafters seemingly cooperating...

    Most games companies don't have peer reviewing, they don't have unit testing, they don't have documentation (and when they have some it's desperatly bad, outdated and generally useless), and they don't have software architects/designers, just a lot of hands of (rather less than more) skilled individuals doing what they have been told with little interactions with people who might know better (and usually little knowledge in algorithms).

    Granted I haven't worked everywhere, and there are notable exceptions.

    Games companies DO NOT even TRY to make bugfree games !

    (And why would they since people still purchase their software and consider it acceptable to have bugs anyway ?)
     
  6. pMax

    Regular Newcomer

    Joined:
    May 14, 2013
    Messages:
    327
    Likes Received:
    22
    Location:
    out of the games
    Would you mind to elaborate? My experience is limited to a sub-sector (game engine) of the field, yet I saw it different. Dont consider startups (<50-100) - they always trade quality for speed.
     
  7. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    A lot of game companies are <100 people and exist since several years, so are no startup.
    If you worked in a company selling game engines (or other middleware), you necessarily had better standards. (Or the company went bankrupt.)
    That said you weren't making games ;)
     
  8. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,031
    Location:
    Under my bridge
    Bugs do increase exponentially with software complexity, and you cannot begin to understand the complexity (and misery) of software development without a decent amount of first hand knowledge. There are so many interacting parts (libs and APIs) developed by people that you might have no control over yourself and there's the issue of never knowing if a bug is your fault or theirs.

    Two examples
    1: At university I had a simple program in two loops. It wasn't completing. I put a printf statement between loops to find which loop was failing. The printf didn't show so I knew it was the first loop that was wrong, but I couldn't find the problem. No-one could. Turns out the problem was that the system buffered the output and wouldn't actually display the statement until another statement was printed. Totally unnatural behaviour and something you cannot intuitively work with - you just have to learn the systems inside and out.

    2: My game used to connect to Google Play services leaderboards just fine. I updated the library I was using and it stopped working. But at the same time, Google updated their library. So which library was to blame? Debug log showed in big, red letters "Google Play Resources cannot be found," suggesting something wrong with the settings or compilation. Lots of struggles later, nothing. And by lots, I mean weeks of going round in circles accomplishing nothing. The more things you change to try and fix one bug, the more chance you have of breaking something else. Then I end up chasing any every bit of info, having learnt something from that Uni experience of yesteryear, and I look up the resources error message. It's a bug in Google's API, reporting they're missing when they're not. It's a lie! So the information one's using to logically try to the solve the issue us useless and you have to have to find other insights.

    Then there's the fact that tutorials only ever cover simple examples and are a completely useless basis for anything more complicated, and the non-existent or outdated documentation where you have to work everything out yourself including methods/functions that don't work how they are described to work.

    Ultimately, software development deals with unreliable information, and that in my mind makes it far more difficult that brain surgery or rocket science. These work with constants, whereas computing works with ever changing states and rules and misinformation. "Rocket Science" came up in the atomic era and something cool and futuristic and difficult, but it's an outdated analogy now.

    However, in answer to the OP's question whether games should be bug free, they should work well. It's a fundamental requirement of the concept of a sale and a fair transaction that the product is fit for purpose. If a company cannot make the game they want in the budget they have without enough bugs to sink it, they shouldn't make it and should pick an easier target. If they want to experiment with bug-ridden games, they need to finance/sell them as open betas/Early Access type products. It's not good enough to say, "the games people want cost too much. We'll have to release them broken and then pay for bug fixes with the sales revenue." If the business can't handle games of that scope without low-bug releases, the games should fit what the market will support.
     
  9. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    13,323
    Likes Received:
    8,857
    Location:
    London, UK
    A lot will hang on the engine. I know little about games development but have some experience with a software platform called VBS2 by Bohemia Interactive in Australia. Anybody who'se been involved in military or first-responder simulators will know Bohemia's products which are very powerful and extensible - sorry, this is sounding like an advert!

    My team has done a fair amount of work on this platform and it's a joy to use. The engine and the tools are the key. AC:U is running on AnvilNext and I imagine a lot of code was completely rewritten, which is a tremendous opportunity to lose track of hacky fixes from old versions and introduce new issues.
     
  10. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    Back to the OP question:
    Game companies are here for profits, not art, that's almost a byproduct to them. They'll produce the cheapest games they can for the highest price they can to maximise their profits, which means the quality will be as low as the market will allow them.
    (As they have no reason to do otherwise, but of course they will adapt to the market and change the quality to match its requirements.)
    So I expect games to continue to come with day 1 patches and further patching during their first months.

    [The problem is that since games are good and selling well, game companies are believing they are doing a good job, hell they even get compared to the movie industry ! Problem is, although the games are good, the engineering behind them is not, but game companies have no reason to believe anyone telling them they are no good at engineering, since they make games that work and are very profitable...]
    [And why would they improve anyway since they make huge profits, it would be expensive to introduce processes & ISO standards since they'd have to train all their people, and they also fear it would impair creativity. Engineering != creativity. Game designers dream, engineers make it a reality (with tradeof sometimes), but rare are the people to acknowledge it.]
     
    #10 Rodéric, Nov 14, 2014
    Last edited: Nov 14, 2014
    rekator likes this.
  11. Commenter

    Newcomer

    Joined:
    Jan 9, 2010
    Messages:
    234
    Likes Received:
    17
    @Shifty Geezer

    Reminds me of this I read on Quora.net.

     
    Laa-Yosh and Pete like this.
  12. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,031
    Location:
    Under my bridge
    Now imagine you're a developer who isn't that important. "Sorry, we won't help you, you're wrong and we're too busy." The project is dead, there's nothing you can do about it. Tough luck.

    That is, in my mind, the very real prospect of development. I had two years of work wiped out just when it was completed when an update to Adobe Flash that I plugged into didn't work with my code and I was the only person on the planet affected by the bug. The development teams involved wouldn't help me. The only option was a complete rewrite from scratch (as I couldn't afford to hire an expert to produce a fix, which may even have been impossible). The project was abandoned.

    For modern developers on middleware, you're at the mercy of the middleware vendors. If you find a bug and it's low priority, you have to find a workaround and make a change. So you can't comfortably engineer with a sense of trust that what you design will be implementable. You read the manual, make the test programs, everything's looking good - oh look, a physics bug that isn't going to be patched for a year. :rolleyes: If you're a big house or a Serious Coder, you can do a lot yourself, but you still have dependencies. There can't be many people out there writing their own physics from scratch, their own UIs, their own audio drivers, etc. And of course the man hours to make an engine with great tools like Unity or Unreal Engine is the same for your in-house project as it is for those middleware vendors who are doing it full time, so that's a mammoth undertaking.
     
  13. Karamazov

    Veteran Regular

    Joined:
    Sep 20, 2005
    Messages:
    3,210
    Likes Received:
    2,774
    Location:
    France
    one big question would be are devs less concerned by finding and resolving bugs since they can now patch their games after release, would they use the release like some free big scale beta test and just wait for feedback from players. Games like BF4 took up to almost one year after release to be "finished"
     
  14. Nesh

    Nesh Double Agent
    Legend

    Joined:
    Oct 2, 2005
    Messages:
    12,569
    Likes Received:
    2,843
    I feel that this problem may have existed with other games as well but the devs havent noticed it. I experienced my memory card losing everything on the PS1 a few times without an observable cause
     
  15. Nesh

    Nesh Double Agent
    Legend

    Joined:
    Oct 2, 2005
    Messages:
    12,569
    Likes Received:
    2,843
    I would have been at the verge of committing suicide
     
  16. rekator

    Regular

    Joined:
    Dec 21, 2006
    Messages:
    785
    Likes Received:
    28
    Location:
    France
    It's probably not dev, more commercial rules… not to many companies are ok to delay a costly marketing plan, and less are ok to have a good secure time dev zone… And this is for almost all industries, look at the auto and the number of recalls… We're in a world where immediate profits is the rule, don't care of futur, will all died… Punks rules the economy!!! :mad:
     
  17. Karamazov

    Veteran Regular

    Joined:
    Sep 20, 2005
    Messages:
    3,210
    Likes Received:
    2,774
    Location:
    France
    yes i should have said publishers instead of developpers. Games are way more complex now and less scripted, increasing the chances of bugs like we see in the latest AC.
     
  18. Goodtwin

    Veteran Newcomer Subscriber

    Joined:
    Dec 23, 2013
    Messages:
    1,163
    Likes Received:
    641
    Absolutely, there is no reason to not demand and expect a game to be properly tested and bug free with good performance. However, this doesn't mean your entitled to anything. As the consumer, you have to be willing to speak with your money. So if you don't like the fact that publishers stick to these unreasonable deadlines at the expense of quality, then you need to become an informed consumer before making the purchase. If reports and showing a half baked game, then if you truly believe that it is unacceptable, then you need to stick to your principles, and not buy it. People are not entitled to a good experience, that's where people get it wrong, they are entitled to purchase the product as it is presented. If you buy a crappy product, the manufacture/developer is not obligated to do anything if they so choose not to do it. The product that is on the shelf for $60 is what they are offering, the consumer then has to make a decision, is that product worth my money or not.

    As for the developers and the complexity and quality control, of course it has gotten much more labor intensive to stay on top of this. I remember Shin'en speaking about how much easier it is for them to stay in complete control of their code, because there is only a few people working on their projects. When you have hundreds of people writing code, and they don't even work in the same facility, its going to be even tougher to keep things clean. Ultimately publishers will have to make decisions, do they want to make these massive games that need an army of workers to develop. I guess it depends on what sells, as long as sub part gaming experiences continue to sell, publishers will assume that this what the consumer wants and continue to walk the same path.
     
  19. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,190
    Likes Received:
    3,409
    No!

    Absolutely!
     
  20. sebbbi

    Veteran

    Joined:
    Nov 14, 2007
    Messages:
    2,924
    Likes Received:
    5,293
    Location:
    Helsinki, Finland
    We do have code reviews. All code submitted needs to be reviewed by someone. Mostly we use pair reviews (one coder reviews other's code), but for some big core functionalities we do peer reviews in larger groups (in a meeting room). This kind of review session for example might happen when a big core API refactoring is ready in it's own branch. We go though the changes and plan how to integrate it to the main development branch. When you have 50+ coders working on a the same code base, you need to ensure that the build never breaks. We have also a big QA/QC team working daily and shelve testing facilities for risky submits that need bigger regression tests.

    We have automated unit tests on all our core libraries. UI code and rendering code (not including GPGPU code such as sorting and low level framework / engine code) often doesn't have full test coverage. Testing code that provides visual interpretation / interface for a human beings is hard to automate. To ensure that these things work properly, we have lots of added debug functionality (that can be enabled without recompiling the project, allowing quick debugging whenever needed).

    We do have software architects. But that's not as clearly separated role as it is in many software development companies. It would be impossible to maintain any code quality (or productivity) in big multiple-studio collaborations without someone actively looking at the code quality.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...