DeanoC
- Thread starter K.I.L.E.R
- Start date
Don't ask me I just work here 
IMHO more developers won't help, but more time for testing will. But that will add to the costs of development which is going to be hard to swallow.
Open testing, I like the idea lots of managers don't. I think releasing a demo before actual release can help reduce compatibility bugs a lot, but again it costs serious cash (time and effort).
IMHO more developers won't help, but more time for testing will. But that will add to the costs of development which is going to be hard to swallow.
Open testing, I like the idea lots of managers don't. I think releasing a demo before actual release can help reduce compatibility bugs a lot, but again it costs serious cash (time and effort).
What I would be interested in are the testing methods for such projects. I dont think the standard stuff (black-box, white-box, unit tests etc...) is sufficient here - and I guess that efficient testing methods are an (if not the most) important aspect to minimize bugs and compatibility issues in a project this size and complexity, especially because this should be the cheapest way.
For some components, unit tests are done, but to be honest the parts are few and far between (maths libs, input, etc).
Ad-hoc corner cases finding, the coders can predict certain areas they suspect bugs lie (loaders, tools, graphics, networks) and delibrately stress test them (i.e. build test models, pull network cables etc).
Their is also internal iterative testing, building the game means alot of data will pass through our hands, their is usually a "shout if it breaks" policy. These either get added to internal TODO lists or fixed on the spot (you can tell when this happens the usually unmoving programmer gets up, moves to an artist's machine and then comes away swearing that can't possible happen).
Then their will be external 'proper' testing, a QA team will play the game, test installer, for PC hardware compatiblity labs (computers set up with various combination of hardware/PS), etc. This bugs will be entered into a database, fixed by the devs (could be an art , design or code bug), then re-tested before being closed. At this stage bugs are classified with A being bad, generally no game can ship with a class A bug. Before any bug can be ignored, it usually has to be signed off by the publishers producer.
When the publisher is happy (or deadlines have been over ran several times, ) the game will either be shipped (if a PC title) or if a console title go into submission.
Submission brings dread to any console team, as here the console owner (Sony, Nintendo, MS, etc) hammers the game to make sure it meets their own standards. Things like saving have to meet exact standards, often the standards doc (currently known from the Sony name of TRC's) have pages of details that you have to get exactly right. Also they will test to try and make it crash etc (Things like running it constantly for 72 hours to make sure things like memory fragmentation doesn't affect it). Usually you fail at least once, which causes a resubmit which may delay release for 6 weeks.
The harsh submission is why console titles have much better quality standards then PC tltles, I've seen publishers ship games with some terrible bugs on PC, whereas on console even minor bugs usually have to be corrected as they know they can't get away with it.
For a game they only really successful test strategy is to play the game over and over again, often not to win but to test. Things like walking into every wall to test collision, saving everwhere, doing stupid things. We usually have a number of aids that allow us to jump anywhere and test that area.
Ad-hoc corner cases finding, the coders can predict certain areas they suspect bugs lie (loaders, tools, graphics, networks) and delibrately stress test them (i.e. build test models, pull network cables etc).
Their is also internal iterative testing, building the game means alot of data will pass through our hands, their is usually a "shout if it breaks" policy. These either get added to internal TODO lists or fixed on the spot (you can tell when this happens the usually unmoving programmer gets up, moves to an artist's machine and then comes away swearing that can't possible happen).
Then their will be external 'proper' testing, a QA team will play the game, test installer, for PC hardware compatiblity labs (computers set up with various combination of hardware/PS), etc. This bugs will be entered into a database, fixed by the devs (could be an art , design or code bug), then re-tested before being closed. At this stage bugs are classified with A being bad, generally no game can ship with a class A bug. Before any bug can be ignored, it usually has to be signed off by the publishers producer.
When the publisher is happy (or deadlines have been over ran several times,
) the game will either be shipped (if a PC title) or if a console title go into submission. Submission brings dread to any console team, as here the console owner (Sony, Nintendo, MS, etc) hammers the game to make sure it meets their own standards. Things like saving have to meet exact standards, often the standards doc (currently known from the Sony name of TRC's) have pages of details that you have to get exactly right. Also they will test to try and make it crash etc (Things like running it constantly for 72 hours to make sure things like memory fragmentation doesn't affect it). Usually you fail at least once, which causes a resubmit which may delay release for 6 weeks.
The harsh submission is why console titles have much better quality standards then PC tltles, I've seen publishers ship games with some terrible bugs on PC, whereas on console even minor bugs usually have to be corrected as they know they can't get away with it.
For a game they only really successful test strategy is to play the game over and over again, often not to win but to test. Things like walking into every wall to test collision, saving everwhere, doing stupid things. We usually have a number of aids that allow us to jump anywhere and test that area.
Thanks for the comprehensive answer. So there seems to be no "magic testing algorithm" or the like - it all boils down to testing, testing, testing, testing,...(repeat ad nauseam. 8) )
Two things that i keep thinking about:
a) Would it make sense or even be possible to establish submission practice in the PC world, or at least in Win32 world? For example something like the WHQL, and only those games that pass get a "Designed for Windows" or something like it. The only (but sadly a fundamental) problem I see is that MS will want some license fee for it...
b) How on earth did Morrowind (XBox) get through submission?! That's by far the buggiest console game I've ever seen...
Two things that i keep thinking about:
a) Would it make sense or even be possible to establish submission practice in the PC world, or at least in Win32 world? For example something like the WHQL, and only those games that pass get a "Designed for Windows" or something like it. The only (but sadly a fundamental) problem I see is that MS will want some license fee for it...
b) How on earth did Morrowind (XBox) get through submission?! That's by far the buggiest console game I've ever seen...
Snyder said:Thanks for the comprehensive answer. So there seems to be no "magic testing algorithm" or the like - it all boils down to testing, testing, testing, testing,...(repeat ad nauseam. 8) )
Two things that i keep thinking about:
a) Would it make sense or even be possible to establish submission practice in the PC world, or at least in Win32 world? For example something like the WHQL, and only those games that pass get a "Designed for Windows" or something like it. The only (but sadly a fundamental) problem I see is that MS will want some license fee for it...
b) How on earth did Morrowind (XBox) get through submission?! That's by far the buggiest console game I've ever seen...
Their was a sort of submission program around Win95 time, but the problems where that it was more worried with Win95 intergation rather than game quality and how do you enforce it. If 1 game ships without it, it just penalises (from a cost/time point of view) those who did. Its like WHQL drivers, how many gamers are bothered whether a driver is WHQL'ed or not. We just click through the warning dialog box and ignore the whole process.
The key to remember about console submission process, is to remember its not independent. If a console company decide they want the title out NOW, submission can be alot easier (the converse also happens, I can't (for libel reasons
) mention specific titles, but don't piss console owners off else submission can be a HARD process).Saem said:
We do use software engineering!
It just that games aren't software in the conventional sense, what software engineering practise helps against an artist using to many polygons, or a puzzle that can't be solved? Or impossible spec (It must look like Doom3 but run on a GBA
) Or a software design process that can handle not only changes in specification but not knowing where/what your doing to start with.Dev houses try software engineering practises but end up with the standard game ad-hoc method. We rely on good staff, a good game developer is a jack of all trades, the programmer changes game designs, the artists changes level designs, the designer does some coding and some art. The most important game dev policy is to get good people, good meaning an individual genius but able to work in a team.
Current favorite methods are a loose design doc (big rigid design docs didn't seem to work that well), some kind of informal review (formal reviews don't seem to work that well), informal pair programming (not pair programmer as defined by XP, but more 1 guy helping another out through the tricky bits). The methods are like standard software engineering but replacing the strict parts with an informal creative process, it seems to suit both sides (art and science) of game development..
You can't make a process out of fun, its an evolutionary, chaotic process that means making games is very very hard. Its a curious blend of art and science, which makes a lot of game devs a little 'unhinged'.
Didn't say you weren't. Just suggesting that there maybe room for more?
Specing is part of the problem of most projects. Doesn't matter which one. That's both on the hardware and software side of things. I'm not sure if you're really covering anything here which is beyond the scope of this. If an artists doesn't meet specifications you say so and they go back and fix the "bug". Just like a piece of code not doing what it's meant to do. This isn't hard it's a requirement that has to be fulfilled and everybody no matter the line of work should be familiar with this concept.
Which practices, waterfall, spiral..? It's not just the managment that has to say we're going to use these practices, it's got to be the mindset shared by the entire team.
It sounds like rigid design documents are the ones that end up going beyond design, I've encountered a few of those. One instructor I've encountered used to force insist all design documents are one page only -- when he was working in the industry and in class. Of course the size of the page is left ambigious.
Software engineering is an art and a science, well at least this is what I've been taught. I'm just of the thinking that the game industry still hasn't matured that much -- the reasons are fairly obvious -- and budgets haven't grown large enough in most cases that people are putting such software engineering practices in place. I'm sure hackers of any young genre of software hand simillar views.
Software engineers and programmers who have fun doing whatever they do could say the same thing. Making (insert program type here) is fun and chaotic... I'm sure general software engineering processes have been adapted to various things like OS development and so on. I'm sure similar adaptations can be made for game design that hopefully brings more method to the madness.
Their is always room for more, we have developed methods inside the industry that works quite well. Of course it can be improved but the myth that we use little software engineering is wrong.Saem said:
Saem said:
We have art procedure (things like sign off, bug testing, etc) but it still doesn't help with the creative issues. Games aren't like most software design, we are much closer to art then most software. Ask a conventional paint artist to work to software engineering practise, I bet you won't get a Mona Lisa. Games are very similar, everybody involved are highly creative. While we have to have software engineering, we also have to accept that the creative mind doesn't work well with logical procedures. So we try and find a middle ground. Theirs no doubt we could improve, but it won't come from outside the industry, one day sometime in the industry we write a book with a new software engineering paradigm. It will be radically different from conventional software engineering, as it has to accept the vital difference, that this is an artform! Its really is, it relys as much on intuition and random thought processes as computer science.
Saem said:
I'd say its a slow spiral. Your spin several times without getting any closer (circular motion) before you start the downwards spiral to completion. The initial spins give time for the design doc to be fleshed out and ideas to be more concrete. Their are also jumps (forwards and backwards), the euraka moments. These are quite vital, no game starts with the good bits completely known, when they happen you have to rearrange things a fair bit.
Saem said:It sounds like rigid design documents are the ones that end up going beyond design, I've encountered a few of those. One instructor I've encountered used to force insist all design documents are one page only -- when he was working in the industry and in class. Of course the size of the page is left ambigious.
It doesn't help that we have overloaded the word 'design' to mean to completely different things, which is confusing at times. Their is 'design', the process of making a game and 'design' that a game designer does. We generally have 2 main documents, the concept doc, which is a short overview and the design doc which contains the game is contained. How detailed the design doc should be, is often dependent on the game designer. Some like to have incredible detailed 800+ page doc, some prefer a 20-40 page doc.
Yes software engineering is an art and a science and we use it, but the form we use is radically different from the taught form NOT because of budget or time but because the usual meaning doesn't work. We've adapted and are adapting to meet the different requirements.Saem said:
I disagree, games are basically an artform that involves software, we are far closer to movies, music or painting then we are to OS and word processors. We only get lumped together with other software development because in the old days, the programmer did everything.Saem said:Software engineers and programmers who have fun doing whatever they do could say the same thing. Making (insert program type here) is fun and chaotic... I'm sure general software engineering processes have been adapted to various things like OS development and so on. I'm sure similar adaptations can be made for game design that hopefully brings more method to the madness.
Its something that many outside don't realise, every good game developer (whether art, code or design) is an artist. And any process we use is more related to the process an author or a movie maker uses than what an OS uses.
But different people think different things, prehaps I'm wrong. I know modern software engineering (I study part-time to keep myself upto date, I did UML a few years ago) and IMHO it doesn't apply 100% to games.
DeanoC said:
I pretty much agree with everything you've said. In the gaming related things that I have done, conventional software engineering that you are taught in colleges and universities does't really apply to games. Mostly because a game engine just doesn't work like a conventional application. Conventional applications are generally purely input driven, while games are not. In an application you provide input which then causes the application to perform an operation and then wait for more input. In a game things don't really work like that. The players input modifies the state of the game, but the player is never really in control. The game engine controls itself.
Now as to your comment about games not being like operating systems, I would 'almost' disagree with that statement. I would say games are much closer to operating systems than applications. Many games have sophisticated interperaters and virtual machines. I know of a number of game engine (including one I'm working on) that use a natively multithreaded interpreted languages. Such things do fit in the operating system domain. Not using sound software engineering practices can really hurt you here. At the other end of the spectrum you don't really want to become over engineered either.
Saem said:
I've personally never had to work with a doc that big (But I know people who have), but even 400 odd pages (which I have had to use before) is difficult. Its becomes impossible to find the info you need, so you don't, you use half-remembered facts and then get moaned at (because generally a designer who writes a big design doc, wants it followed to the letter...). I'm on the small design doc side of the fence
