Is the difficulty of debugging complex games non-linear?

Discussion in 'Console Technology' started by Commenter, Nov 13, 2014.

  1. joker454

    Veteran

    Joined:
    Dec 28, 2006
    Messages:
    3,819
    Likes Received:
    139
    Location:
    So. Cal.
    I actually think this is relatively close to the truth. It happened in pc gaming many years ago as well. Originally pc games were not buggy, I remember playing games like Space Quest, etc, on my pc without issues but over time as pc's got more connected and it became possible to "patch" them then the frequency of bugs in games started to increase. Now consoles are starting to see the exact same phenomena. I don't think it's fair to hide behind "game complexity" as an excuse either. Think of some of those early pc games like Ultima, Wing Commander, etc, some of those were quite complex for the time period, and far more so when you factor in the significantly more primitive dev tools they had, the smaller dev teams, more obtuse programming languages and so on yet those games were not riddled with bugs.

    Personally I'd say the poster child for all this is the old pc game Ultima Online. That game was by far the most buggy game ever shipped on this planet or the next and it was that way purely because the online connection was required and hence they knew they could ship a partially broken game, which they did. That required online connection (a unique thing at the time) permitted them to turn A bugs into C bugs and ship the game because they knew they could patch it later. One could make a case that this all started with that very game, and as game playing boxes got more connected so did the willingness to ship with more bugs.

    Still I do think the root cause of all this though is over promising, trying to get too much packed into a game because they know that they can just put the devs on crunch and patch the game after ship anyways.
     
  2. Laa-Yosh

    Laa-Yosh I can has custom title?
    Legend Subscriber

    Joined:
    Feb 12, 2002
    Messages:
    9,568
    Likes Received:
    1,455
    Location:
    Budapest, Hungary
    Wing Commander 1 had some pretty serious bugs... And I think even Ultima 7 had some problems. Then there was Ultima 8 where they had to cut major parts of the story but left the dialogue text in the system files.
     
  3. joker454

    Veteran

    Joined:
    Dec 28, 2006
    Messages:
    3,819
    Likes Received:
    139
    Location:
    So. Cal.
    Those later Ultima games were "pathchable" since at that point Origin was large enough to offer that kind of support to where you could get patches from them, which ultimately feeds to my point. I don't remember any bugs in the first Wing Commander though and I played that game to death! The only thing I can recall is that it required an expanded memory manager to get all the games features, and if people didn't use the officially supported one (was it QEMM?) then maybe those people saw issues.
     
  4. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    15,979
    Likes Received:
    3,223
    Since I've spent a lot of time rabbiting on about one of the causes of the current state of games, I think it's about time i offered a solution
    to at least part of the problem
    Are you sitting comfortably, then i will begin :
    The EULA. The following clause should be banned
    THE SOFTWARE IS PROVIDED TO YOU “AS IS,” WITH ALL FAULTS, WITHOUT WARRANTY OF ANY KIND, WITHOUT PERFORMANCE ASSURANCES OR GUARANTEES OF ANY KIND,
    And replaced by
    THE SOFTWARE IS GUARANTEED FOR 28 DAYS
    That would go a long way to making games better
     
  5. Laa-Yosh

    Laa-Yosh I can has custom title?
    Legend Subscriber

    Joined:
    Feb 12, 2002
    Messages:
    9,568
    Likes Received:
    1,455
    Location:
    Budapest, Hungary
    The later missions in WC1 bugged me to death as I recall, game breaking slowdowns.
    As for U8, well, anything beyond the Serpent Isle was compromised by EA. However, Garriot had a pretty good vision about the future of RPGs - just look at Dragon Age III, it feels just like what he intended.
     
  6. dobwal

    Legend Veteran

    Joined:
    Oct 26, 2005
    Messages:
    5,435
    Likes Received:
    1,497
    The "Thank you for playing Wing Commander" when exiting is actually a memory manager error that was hex-edited by the dev team.
     
  7. sebbbi

    Veteran

    Joined:
    Nov 14, 2007
    Messages:
    2,924
    Likes Received:
    5,293
    Location:
    Helsinki, Finland
    All games and all computer programs have bugs. Not all bugs are critical, and not all bugs are found during the development. It's not easy to draw a line where a product can be considered "faulty goods".

    Do you think that Super Mario Brothers games were "faulty goods"? Here is a wiki about the bugs found in the Mario games: http://www.mariowiki.com/List_of_Super_Mario_Bros._glitches. That's a lot of bugs in a released game. Street Fighter 2 had a serious bug that allowed you to cancel your attacks and perform another ones quickly, making it impossible for your opponent to block them. Turns out that people loved that bug so much that similar "combos" become the trademark feature of fighting games for years. However this was pure luck. The same situation could have enabled (easy) infinite combos as well, ruining the game experience completely (= "faulty goods" for competitive play).

    Not all critical bugs get found in the testing process. Gamers tend to overestimate the human resources of game development companies. This is easier to tackle with an example. Our example game developer company has 20 full time (8 hours a day, 50 weeks a year) testers. A game project is 2 years long. During this time the total time spend on testing is 20 * 8 * 50 * 5 = 40K hours. This sounds like a lot...

    The game gets released. One million players buy it in the opening weekend, and play an average of 4 hours on both Saturday and Sunday. This is 8 million hours in total. During the first weekend the players play the game 200 times as much as the test team did during the whole 2 year project. I think we have all heard the following angry statement: "I can't believe you didn't find this bug, because we found in the first week after the launch!". I hope this example puts some perspective into this statement, and explains why it can happen.

    Why all bugs cannot be found? It is very hard to find something that you don't know to exist. And it's impossible to prove the non-existence of something (bugs in this case), unless you can go though ALL the permutations and show that the thing (bugs) never exists (proving by contradiction). This is impossible for games, because the sheer number of input permutations per frame make the search space too large. NES dpad had 8 directions (+one permutation for not pressing any). A+B buttons give 4 permutations. Total number of possible input permutations per frame is 9*4 = 36. On a 60 fps game, you have 36^60 input permutations per second. It is impossible to go though all the possible program states even for a single second of game play. Also, as Ethatron said, proving that software works properly (never hangs) is proven to be impossible (http://en.wikipedia.org/wiki/Halting_problem).

    One thing why players can find bugs more easily (especially during the first weeks) is that developers know the game from the beginning of the development process. Developers and long time testers eventually get blind to the UI and the game mechanisms. They know them so well that less often used button press combinations practically never get used. In a similar way a hardcore player would find completely different bugs from the game compared to a casual player. However since there are millions of players, all the different player categories are present (on the launch week) and the coverage becomes very good. Unfortunately it's often impossible to get as varied bunch of testers to test the game during the development. Also testers and players have different motives to play the game. Some of the mechanisms are much more used by actual players (and by different ways).

    Edit:
    I wanted to add an example of a critical crash bug that was found in a game I was developing (almost 10 years ago). The engine used too much dynamic memory allocations. When you played enough missions without booting the game, it eventually crashed because the memory fragmented too much causing allocations to fail (http://en.wikipedia.org/wiki/Fragmentation_(computing)). This bug was found in the late stages of the game project, so we had limited time to fix it. As the engine itself was deeply designed around dynamic memory allocation, it would have been a massive task to rewrite the whole memory management of the engine. That refactoring would have made so many big changes in the code base that it would have likely caused many other bugs in the process. This was not acceptable as the game was already very stable and everything else was in a good shape. So we spend lot of programmer and technical artist time to reduce the memory consumption of the game to free some memory to delay the problem. Once our test team confirmed that the game could be played from the first mission to the last without booting the device we were happy. If you'd played the single player campaign two times in a row without saving or booting the device, it would have crashed for sure. Nobody noticed this bug or complained about it after the launch. It would have delayed the game launch at least by 6 months if this bug would have been properly fixed. I don't think this would have been good for either us or the players.
     
    #107 sebbbi, Dec 7, 2014
    Last edited: Dec 7, 2014
    rekator, MJP, Shifty Geezer and 2 others like this.
  8. joker454

    Veteran

    Joined:
    Dec 28, 2006
    Messages:
    3,819
    Likes Received:
    139
    Location:
    So. Cal.
    I don't think people are complaining about bugs in general, that sort of thing is inevitable in software as Sebbbi mentions and I believe that most people can understand that. I think what people are complaining about is the ever increasing willingness to ship with known bugs and/or with critical game breaking bugs. I suspect all of us have been there in our professional career at some point where we've heard "We'll patch it later", where A bugs getting magically reclassified as B bugs and so on not because the developers had a choice but because they had to meet the deadline imposed by the publisher or face various fines, penalties or worse case complete obliteration. It's this sort of behavior that seems to be getting worse and worse as dependency on that internet connection to "Patch it later" increases. I can understand that shipping a bug free piece of software is all but impossible because internal q/a can never match real world use but the reality is that software is knowingly being put out with sometimes monstrously long lists of known bugs. You'll never get a perfect bug free game but it would at least be nice to know that you can play and complete a $60 game that you just bought as it was intended.
     
  9. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,577
    Likes Received:
    16,028
    Location:
    Under my bridge
    Davros was! He was wanting bug-free software.

    That's basically implied in EU consumer law. If a game doesn't work, you're entitled to take it back to the shop for a refund, regardless of the EULA. Statutory Right > EULA (contract law). Consumers have used this principle successfully with some games, certainly regards returning a disk. Getting a download refund my require more effort, but it's also doable.
     
  10. sebbbi

    Veteran

    Joined:
    Nov 14, 2007
    Messages:
    2,924
    Likes Received:
    5,293
    Location:
    Helsinki, Finland
    I think there are both technical and project management reasons why there are more bugs in modern shipped games compared to old times.

    First, the team sizes have grown massively. AAA game development teams are easily 5x+ larger than they used to be two generations ago (GameCube / Xbox1 / PS2). Code bases of modern games are massive. I remember games with 10k-20k lines of code (total), but nowadays a single bloated source code file can be that large (I have seen a 60k line file). Games such as GTA 5 must have way over million lines of code. It's very hard to manage a code base this big and a team this big (hundreds of programmers writing code to the same code base).

    The second big change was multithreaded programming (PS3 and Xbox 360 both required it to perform well and quad core PCs got popular at the same time). Testing multithreaded code is hard. Timing bugs (race conditions) can be absurdly difficult to track. I have seen bugs that cause a random crash once per week (but obviously never when attached to a debugger). Testers spend days of time to repro a single bug like this (and failing). Writing code in a multithreaded setting is hard for programmers as well, because you can't be 100% sure your code works by testing it a few times. The new code might only crash in some specific scenario (something else is happening in background at the same time).

    Obviously also internet connected consoles make games easier to patch, but I firmly believe that the two things mentioned above have had bigger impact on game quality. It is never good publicity for a developer to release a buggy game.
     
  11. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    Okay I'm going to play devil's advocate here, because as much as I agree with you, I think there are dead simple answers:

    If they can't handle big code bases, then they shouldn't write them to begin with. (Data centric ? Or narrowing the scope ?)
    If they can't write proper multi-threaded code, then they should rely on mutex or go single threaded.

    It sounds like you are saying they are forced to use thing/tools beyond their skills/organisation...
     
  12. Laa-Yosh

    Laa-Yosh I can has custom title?
    Legend Subscriber

    Joined:
    Feb 12, 2002
    Messages:
    9,568
    Likes Received:
    1,455
    Location:
    Budapest, Hungary
    You've gotta be kidding us here.
    Of course they are forced.

    Hardware went multicore and there's no other choice to extract any reasonable performance then to write multithreaded code.

    Customer expectations about features are sky high, open worlds and crowds and lots of traffic and whatever.

    On top of it all, this is something that a lot of people predicted and warned about many years ago.
     
  13. forumaccount

    Newcomer

    Joined:
    Jan 30, 2009
    Messages:
    140
    Likes Received:
    86
    Despite most concepts in CS being at least 30 years old at this point, not all of them are in wide usage. Symmetric multithreading is very much a new paradigm, especially game development circles. The complexity of the Cell required people to utilize it, and in even more complicated ways than they might normally have chosen. Best practices take a while to shake out... but now everyone is kind of focused on the new frontier which is compute.

    The other issues which I alluded to earlier are that in consoles the part of the software stack over which developers have no control has grown in size and complexity, and the actual physical hardware components have dropped in reliability. Both of these lead to a lot of issues. There are ways to mitigate software problems, via education, code reviews, automated testing, etc. But you can make a perfect game and it will still fail for some users because it's running on an imperfect stack of hardware and software that are entirely beyond your control.
     
  14. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    I'm gonna continue to play devil's advocate.

    So you're telling customers that they are responsible for the bugs now ?
     
  15. Laa-Yosh

    Laa-Yosh I can has custom title?
    Legend Subscriber

    Joined:
    Feb 12, 2002
    Messages:
    9,568
    Likes Received:
    1,455
    Location:
    Budapest, Hungary
    Yes. They want more and more complex games but they're not willing to pay more money for neither the hardware nor the software.
     
    DSoup likes this.
  16. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    I'm not disagreeing on the source of bugs increase in the past years. (Although we could discuss how much game "industry" marketing is responsible for emphasizing the desire for ever more and improved gfx, and therefore how much the game "industry" drove itself into a corner/wall..)

    I just disagree that the game "industry", today, is following software engineering best practices or even considers doing so, since a majority of people in this "industry" consider bugs to be natural byproducts rather than errors which shouldn't have been made...
     
  17. Billy Idol

    Legend Veteran

    Joined:
    Mar 17, 2009
    Messages:
    5,984
    Likes Received:
    822
    Location:
    Europe
    I would agree with you Laa-Yosh, when every single AAA game released would have the major problems we are discussing here and not only BF4, AC:U and Driveclub (everyone knows that bug free does not exist, but game breaking bugs which makes games unplayable for the majority of gamers). But there are also a lot of games which run very good without major issues (hence they are also not in the media as of late).

    So, as not all game devs have these major problems it seems, I would argue that it is more a problem related to specific devs and publisher and hence it is more related to their QA methods and their development philosophy. Imho, studios like Evolution, DICE and UBI just are not capable enough to pull off their projects. They need to quite improve imo to keep up with other devs which release "non problematic" games.
     
  18. Dominik D

    Regular

    Joined:
    Mar 23, 2007
    Messages:
    782
    Likes Received:
    22
    Location:
    Wroclaw, Poland
    BS. No set of "best practices" will result in a perfect software. There are things one can do better or worse, but these are always trade-offs. But even if we decided to stick with guidelines that prioritize code reliability over everything else (speed of execution, speed of delivery, whatever) you would still end up with bugs, some of them heisenbugs, that you wouldn't be able to ever hit in your environment but which would destroy someones gaming experience. One of the reasons people can't understand this, I think, is that there's this myth of consistency in how HW works, so as long as your code behaves to the specs, everything will be fine. No, it won't. In fact: HW internally is partially software (firmware for your CPU or GPU is just that: a piece of software). Voltage and temperature changes can impact timing, which can bite you in multithreaded code. I could go on and on. Arguing that bugs can be avoided is soul-crushing to those who write code because they come from the same realms Hobbits do, yet people seem to believe in it.
     
  19. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    Nobody ever said "perfect", even though being close to 0 bugs can be achieved, it's just too expensive to consider for entertainment... (and as you mentionned requires hardened hardware too).

    When quality doesn't match expectations and methods to improve it are known, it's legitimate to ask why those methods are not applied.
     
  20. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,577
    Likes Received:
    16,028
    Location:
    Under my bridge
    Davros did. We need to be clear about levels of bug and what's realistically attainable. If we all agree 100% bug free doesn't exist (and anyone who doesn't shouldn't be in this thread as they'll only contribute noise from here-on-in), then the issue becomes one of dealing with the number and severity of bugs, in particular real game stoppers that seem to be increasingly prevalent.

    The question becomes one of whether crippling, game-stopping and seemingly obvious bugs are the product of poor development, perhaps lacking strong engineering principles, or if these sorts of bugs are an intrinsic aspect for modern software development and there really is nothing that can be done about it.

    The reality, IMO, exists part way between. I don't believe all software is developed to best principles with effective management and a strong emphasis on quality, but I also believe that software is so inherently complex that there will be significant bugs that get through to release and these will only increase with increasing game complexity, especially when games need to be made to deadlines.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...