Is the difficulty of debugging complex games non-linear?

Discussion in 'Console Technology' started by Commenter, Nov 13, 2014.

  1. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    You have similar brain washing inside the game "industry" too, with people saying that you shouldn't complain about unpaid overtime and that you're lucky to work in the business anyway, that's just insane, but representative IMO.

    I agree customers are way too forgiving, and with proper engineering + testing we'd get a lot less bugs, but that'd mean hiring more experienced people that will likely refuse to get paid little and work long hours, so not a good option for maximum profits... (and as long as people pay for less why bother ?)
     
  2. Sigfried1977

    Legend Veteran

    Joined:
    Sep 14, 2008
    Messages:
    5,005
    Likes Received:
    1,817
    I think the rise of social media also contributes to modern games appearing so much shoddier than their predecessors. Back then you just ran into a game breaking glitch and that was it. Or maybe you started some heated discussion on gamefaqs. Nowadays you're screaming at the top of your lungs on twitter, and before you know it, a dozen editorials are being written by the enthusiast press.
     
  3. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,196
    Likes Received:
    3,411
    Oh yes, publishers aren't just content with screwing the public, they will happily screw developers too.

    One of the reasons I'd never survive in a q/a job is because I care and if I found out they were planning to release the game without fixing the bugs I would kick up the mother of all sh1t storms and probably get fired.
    For example battlefield 3 on pc cannot be completed if you rebind the jump key (default spacebar) all the keys were originally hardcoded then for the pc release someone went through the code (and in the case of the jump key) they changed :
    If spacebar pressed
    to
    If key_bound_to_spacebar pressed

    but they forgot to do it for the final qte. Its 2 minutes of coding and if I was q/a on bf3 I would of found out the guy responsible sat on his desk and refused to move until he fixed it.
     
    #63 Davros, Dec 2, 2014
    Last edited: Dec 2, 2014
  4. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    13,361
    Likes Received:
    8,901
    Location:
    London, UK
    The games industry is probably not representative of professional software development. Prior to working in Government I worked in aerospace and defence where there are few, if any, margins for error in software. It's an entirely different discipline but then more robust technologies are often used, including specific language implementations and special compilers and libraries. Each body of code has to have specific and measurable qualifications for approval.

    Obviously, the impact of a face not being rendered on an NPC in AC: Unity is a bit different to software that drives the tweakers trying to jam an incoming missile's radar. It's symptomatic of being part of the entertainment industry, alas.
     
  5. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,196
    Likes Received:
    3,411
    Also programs do not generally have a publisher, coincidence ?
     
  6. forumaccount

    Newcomer

    Joined:
    Jan 30, 2009
    Messages:
    140
    Likes Received:
    86
    Bugs do slip through the net... that's exactly what's happening. If you think this is a massive conspiracy to trick consumers into buying low quality products I'm not sure it's worth my time trying to convince you otherwise. But if you're interested in the actual reasons behind the problem, I can offer some insight.
     
  7. pMax

    Regular Newcomer

    Joined:
    May 14, 2013
    Messages:
    327
    Likes Received:
    22
    Location:
    out of the games
    ...just be ready to move, and change company. While it is true that nobody is necessary, it is also true that replacing people has a big hidden cost for a company: and the project will suffer much more than it seems (it all boils down to how much incompetent your manager are, normal or "talented").

    I've seen projects made rotating contractors - the only one that worked was made by EXTREMELY skilled guys, which could adapt and jump in with little hassle. But they aren't so many ;)
     
  8. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,196
    Likes Received:
    3,411
    sorry dupe no way of deleting posts
     
    #68 Davros, Dec 4, 2014
    Last edited: Dec 4, 2014
  9. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,196
    Likes Received:
    3,411
    Sorry but in regards to battlefield 4 thats exactly what happened the devs were told to ignore the bugs and q/a were told to sign off on the game so the release date could be met E.A admitted it. You think this behavior hasnt happened with other games. Conspiracy no, but I believe there is a mindset within the game publishing industry the thinks its o.k to release a game if it has bugs.
    Are you trying to say publishers would never release a game with a known bug?
    The 300 + bugs fixed in the last assassins creed do you believe q/a and the devs missed every single one of them, they were so hard to find and yet within 2 weeks they were located and fixed.
    edit: less than 2 weeks as this was patch number 3
     
  10. forumaccount

    Newcomer

    Joined:
    Jan 30, 2009
    Messages:
    140
    Likes Received:
    86
    I'd be interested if you could find a source for EA admitting that. I've talked with engineers who worked on BF4 and from those conversations I'd be quite surprised if this was true.

    No, some bugs are perfectly fine to release with. We call them C bugs, and if you can't fix them in time then the game goes out the door anyway and you either address it in a patch or don't address it at all because it was never that serious. That's why it was a C bug in the first place.

    So Ubisoft has like 8 million people working on AC:U and I'm sure they can churn through bugs quite quickly now that the bugs are known. That second part is key... you don't think they could have had the same velocity before the game came out? Do you know how much worse it is to be stuck fixing a title that's gotten a reputation as a bug fest when you should be on a beach somewhere celebrating its success? Do you know how much worse it is for the publisher and marketing? They have to apologize, offer free crap, etc, just to keep their customers. This is not a cheap endeavor that people are willing to just absorb the costs of in full knowledge.

    The patch notes say 90+ low repro crashes addressed, which probably means that they were working off dumps submitted by users either through PSN or other means. Working off dumps means you can fix or at least partially address the bug without ever actually running into it. (And by the way, "low repro" when you have millions of sales could be "500 people run into this crash a day"... but 500 people can make a lot of noise about an issue while the other few million aren't running into it at all.)

    Actually these kinds of sweeping patches are highly risky and it wouldn't be surprising if AC:U runs into another wave of bad issues. Post-launch is a vulnerable time and you have to choose what you fix and what you don't all the more carefully. If you just go haphazardly changing things trying to address crash bugs that you can't reproduce you're likely to either not fix the bug or introduce a new issue. But those guys are professionals so I'll assume they know what they're doing.
     
    BRiT and MJP like this.
  11. forumaccount

    Newcomer

    Joined:
    Jan 30, 2009
    Messages:
    140
    Likes Received:
    86
    I'll just relay an anecdote about my last title. We had support for crash dumps throughout the entire development of the title. Every crash run into by QA was logged and uploaded into a searchable database. After release I searched our db for the number one issue from crash dumps and it was something we'd never hit once during development. This is a bug affecting thousands if not tens of thousands of players. I think we'd hit the number two issue once or twice over the course of years of dev, but were never able to reproduce it.

    If we hadn't had crash dumps, it would have taken weeks to track it down. It would be extremely easy for someone to go "these lazy/corrupt devs, why did they release this game with this obvious flaw?" but if we don't even know the bug exists what can we even do. AAA games these days are some of the most heavily QA'd games in the history of the game industry and this still happens. Corruption and laziness are very hard to believe when you begin to understand that.
     
  12. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    It's all nice to have an army of testers, but why focus on an after the fact process when you could be preventing bugs from showing up in the step before ?
    (Answer is : It's perceived as cheaper to do so than to go industrial with processes and standards, plus most programmers believe they don't write bugs, or few, and I'll not even mention using C++... Oups I did :p)

    Besides being the most heavily tested games is a lie (talking about coverage here, not time spent) and even if it wasn't , you already narrowed the scope to games, meaning either you know or you expect other industries to be much more thorough...

    Don't get me wrong, I'm not saying game devs are lazy scum, they sure aren't, I've been there in various places, I know well, and it's almost impossible to be bug free without a massive budget, which can't happen unless people are willing to pay more and/or publishers narrow margins, but that doesn't mean there aren't simple/not so expensive steps that could improve things in a noticable way.

    I think it's more like you say game devs don't do so bad, but some of us are pointing out that it could be done better, and not for that much more time/effort/money.
     
    #72 Rodéric, Dec 5, 2014
    Last edited: Dec 5, 2014
  13. forumaccount

    Newcomer

    Joined:
    Jan 30, 2009
    Messages:
    140
    Likes Received:
    86
    All software has bugs, it's an unfortunate reality of software development. We spend a lot of time and money improving not just our human testing but our tracking and automated testing.

    OK, well the thread is about why games, so I'm comparing to other games. One thing I know about other industries is that their software is also buggy, sometimes also in blatant and significant ways.

    Usually when people bring this up they're comparing to PS2 era when games couldn't be patched yet were perceived as more stable. In 2014, compared to then, our methods have only improved and we have significantly more and better testing, but there are still issues. This supports the fact that something changed about the software or platform that's making finding bugs more difficult than it was before.

    I didn't say game devs don't do so bad. I'm telling you that games are significantly more complex today than in the past and this is a major contributing factor to the increase in bug-ridden releases just in the last couple of years. We can name them all... Skyrim, BF4, Simcity, AC:U. I don't believe a single one of those games was intentionally released in that state knowing that people would run into serious issues.

    Usually it only takes one of those massive stability failures before everyone gets on board with better QA processes, crash logging and tracking, static analysis, unit testing, automated functional testing, code reviews, etc etc etc. But nothing is a panacea. It wouldn't surprise me if the AC:U team does all of that already.
     
  14. Rodéric

    Rodéric a.k.a. Ingenu
    Moderator Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,031
    Likes Received:
    898
    Location:
    Planet Earth.
    Why are they more complex today ?
    Do they need to be ?
    Even if they need to be that complex, why can't they be made (more) reliable ?

    Aren't cars and rockets at least as complex ?
    Why are they more reliable then ?

    To make my point clear, it's not like they can't be made more reliable, it's that there is no reason to reduce margins to do so, unless the market demands it.
     
  15. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,031
    Location:
    Under my bridge
    Because there's far more going on, both in content depth and number of features active concurrently, and they're having to run multithreaded these days thanks to the CPU limits.

    Yes, otherwise games would be stuck in a very limited scope of early 2000s tech.

    They can, but at exponentially higher cost for the increased complexity.

    No, not IMO as described earlier. You don't have an equivalent of multithreading in physical engineering. You won't have brake pads sometimes applying friction before the tire rolls on the ground and other times applying friction after the tire moves. You won't have gravity change one day because God updated the Universe to Version 2 and metricised physical laws. The same science that worked for rocket science and automobile engineers 50 years ago still works for them now, whereas computing is a completely different beast and the skills from 50 years ago are pretty much useless now.

    And let's also not forget that increasing complexity in automobiles has resulted in more bugs, such as cars where the brakes don't work and kill people. And the bugs in the automotive industry resulted in people deciding whether the cost of fixing the bug in released cars was cheaper than the losses due to getting sued if something should go wrong.
     
  16. NRP

    NRP
    Veteran

    Joined:
    Aug 26, 2004
    Messages:
    2,712
    Likes Received:
    293
    Thanks forumaccount for sharing your insight and providing some perspective. It's too common these days for laymen and "armchair quarterbacks" who don't understand the massive technical complexity to just assume nefarious behavior when something goes wrong.

    I am curious though, some people have mentioned bad managers. What makes a good manager?
     
  17. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,196
    Likes Received:
    3,411
    It was a q/a tester for dice (sorry my memory isnt what it was)
    http://bf4central.com/2013/11/ea-pushed-battlefield-4-quality-control/

    No they are not, and as for my asertion there's a mindset well youve just demonstrated it

    Your not seriously offering that as a defense are you "sorry we forgot to fit the brake pads to your wife's car and she died, but Maui was looking particularly lovely that week" :D
     
  18. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,031
    Location:
    Under my bridge
    yes they are, simply because 'no bugs' == 'perfection' and perfection is a nigh impossible target for anything beyond rudimentary. 'Good enough' as an essential target for actually getting stuff finished. An example of a tolerable bug is a rare graphical glitch that you have trouble reproducing. It doesn't affect everyone, doesn't impact the game, and would cost far more to fix than it's worth. "Forgot your brake pads" level bugs aren't irrelevant C bugs. That's a major fault. Busted servers where you can't connect to a game aren't a tolerable C bug, but forumaccount was making the distinction between allowable bugs and real game-killing doozies.
     
  19. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    13,361
    Likes Received:
    8,901
    Location:
    London, UK
    Cars no, rockets yes. Modern pilotless systems employ very different physics models to reach their destination and the physics models get refined with more testing (usage). Modern missiles can adapt to different altitudes, environments (temperature, humidity) and factor into their mission execution meteorological parameters too. These have can affect on performance (the missile's ability to track, navigate, turn and it's range). The type of payload can affect this as well. As for gravity, well I'm afraid gravity isn't a constant except in crude astronomical body terms. Modern missiles have ballistic gravimeters to measure this. Missiles that they augmented by visual tracking systems have to compensate for light pollution, actual pollution (have you seen Beijing?) and smoke.

    Your modern missile will be compensating for these things, fighting countermeasures, using any and every RF, IR, MW and satellite signal to course correct while (often) maintaining two-way communication with C&C so it can terminate (self destruct) or alter it's mission objective on the fly.

    Sorry for going off topic.
     
  20. TheWretched

    Regular

    Joined:
    Oct 7, 2008
    Messages:
    830
    Likes Received:
    23
    I work in automotive engineering, and in fact part of my work is QA (well, I do testspec and testimplementation). And I can tell you... there is more than you'd like. In a networked car (modern car, that is), you'll have 50+ ECUs communicating with each other at all times. Engine, Brakes, ESP, Airbags... if any of those components goes mental on the road, it can mean a lot of dead people. ISO 26262 is a core component here (functional safety). Each and every major and minor software release gets tested to a high degree, first in laboratory settings, as well as "on the road" (and several other testing environments, like HILs).

    A car isn't just a mechanical beast anymore. It's as much a computer today than any other gadget people buy, including games consoles. Your console doesn't start using PPC ISA while gaming, either^^

    And, by and large, we do ship with the same "C Bugs" as well... but our definition of "C Bug" is pretty different, because of ISO26262. Customer satisfaction is of the utmost importance, but saving lives is just as important.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...