A look at Halo 3 testing

[nelg]

Halo 3: How Microsoft Labs Invented a New Science of Play

All game companies test their products, but generally they just pay people to report any bugs they find — monsters that disappear or places where graphics don't render properly. But because it is owned by Microsoft, which launches dozens of Xbox and PC games every year, Bungie has access to one of the most advanced game-testing facilities ever built. Pagulayan and his team have now analyzed more than 3,000 hours of Halo 3 played by some 600 everyday gamers, tracking everything from favored weapons to how and where — down to the square foot — players most frequently get killed.

 

[archangelmorph]

Halo 3: How Microsoft Labs Invented a New Science of Play

Talk about brute force playtesting..

Still..

All the money M$ sauced into this little research venture *should*/*could* pay-off if their findings could be formalised and published, making it available to other developers (either buy selling the publication or releasing it for free)..

I'm sure alot of games designer would benefit from alot of the knowledge gained from the studies they performed and it would definitely benefit M$ to share "insider trick/knowledge" on "x ways to guarantee fun in games of type x.." to other 1st and second party devs..

But then I guess it all depends on whether there actually exists a set of clear-cut rules which provide a "formula for awesome".. I believe there is..

But then again.. I believe anything the world could be formalised and described mathematically in C++...

 

[nelg]

One has to give MS credit for making game-play as an equally important aspect as other areas.

All the money M$ sauced into this little research venture *should*/*could* pay-off if their findings could be formalised and published, making it available to other developers (either buy selling the publication or releasing it for free)..

I'm sure alot of games designer would benefit from alot of the knowledge gained from the studies they performed and it would definitely benefit M$ to share "insider trick/knowledge" on "x ways to guarantee fun in games of type x.." to other 1st and second party devs..

I agree. Perhaps MS could give developers who are producing Xbox games access to it. If the benefits are as good as they appear, it would be a shame to see its utilization go under used.

 

[Scott_Arm]

damn, I'd hate to be a level designer for Bungie. Not that I think this kind of testing is a bad thing, but constantly tweaking levels based on focus groups would be really really annoying.

 

[RobertR1]

From all the interviews with Bungie, I get the feeling that internally they were not pleased with Halo2 and really are pressuring themselves to make sure Halo3 in an undisputable (as much as can be) epic.

 

[kyleb]

damn, I'd hate to be a level designer for Bungie. Not that I think this kind of testing is a bad thing, but constantly tweaking levels based on focus groups would be really really annoying.

It easily beats having hardly anyone even bother to beat your game because they get tired to running out of ammo, wandering around in circles and getting slaughter in the same damn place every time. Pretty much any decent game gets tweaked in response to play testing throughout development, the more comprehensive the testing the better chance developers have of overcoming whatever faults might exist in their design.

 

[PARANOiA]

It easily beats having hardly anyone even bother to beat your game because they get tired to running out of ammo, wandering around in circles and getting slaughter in the same damn place every time. Pretty much any decent game gets tweaked in response to play testing throughout development, the more comprehensive the testing the better chance developers have of overcoming whatever faults might exist in their design.

Exactly. Counterpoint would be Lair, which seemed like it could have used a lot more feedback in the development process, since most reviews indicate it had a lot of potential, but was just poorly implemented.

 

[JacksBleedingEyes]

I think someone was missing the point. The game wasn't tweaket based on focus groups. The developers analyze what is wrong in general and tweak it to their resolution not what the specific focus group has in mind as they are not the experts in the subject matter. If you let the focus group be in control your going to get that Simpsons episode where Home had to design a car. A lot of good ideas poorly implemented. Users don't know what is good, but they certainly know what's bad.

 

[22psi]

From all the interviews with Bungie, I get the feeling that internally they were not pleased with Halo2 and really are pressuring themselves to make sure Halo3 in an undisputable (as much as can be) epic.

They have openly stated that they (Bungie) was not happy with Halo 2.

 

[Scott_Arm]

It easily beats having hardly anyone even bother to beat your game because they get tired to running out of ammo, wandering around in circles and getting slaughter in the same damn place every time. Pretty much any decent game gets tweaked in response to play testing throughout development, the more comprehensive the testing the better chance developers have of overcoming whatever faults might exist in their design.

I totally agree. I just meant it would suck to be the level guy that has to make the changes. I guess if they're really good there shouldn't be that many.

 

[Scott_Arm]

I think someone was missing the point. The game wasn't tweaket based on focus groups. The developers analyze what is wrong in general and tweak it to their resolution not what the specific focus group has in mind as they are not the experts in the subject matter. If you let the focus group be in control your going to get that Simpsons episode where Home had to design a car. A lot of good ideas poorly implemented. Users don't know what is good, but they certainly know what's bad.

No, I wasn't missing the point. Maybe I just got the terminology wrong. But basically, they have a "focus group" that plays the game, and someone monitors them and decides on what the appropriate change is to make the experience better for the "focus group."

 

[archangelmorph]

I totally agree. I just meant it would suck to be the level guy that has to make the changes. I guess if they're really good there shouldn't be that many.

Surely such a level designer will benefit greatly from this because they'd have the opportunity to rapidly learn from there mistakes and gain a solid understanding of what aspects really work to improve the experience far far greater thn the rest..

In a world where experience is key, such a designer would gain far more knowledge and understanding from this than in any other position where the design feedback loop wasn't so strong..

 

[Tap In]

No, I wasn't missing the point. Maybe I just got the terminology wrong. But basically, they have a "focus group" that plays the game, and someone monitors them and decides on what the appropriate change is to make the experience better for the "focus group."

how can things like having hard data showing points on a screen in real time where there are choke points on the maps you designed or that users are using too much ammo for the particular level (based on your computer data) so you can adjust it for the game to flow, be considered "catering to a focus group"?

the point of the story is the data points they are collecting in this play testing as well as from the Beta where every user's footsteps and shots fired were recorded as hard data and processed.

 

[Scott_Arm]

Surely such a level designer will benefit greatly from this because they'd have the opportunity to rapidly learn from there mistakes and gain a solid understanding of what aspects really work to improve the experience far far greater thn the rest..

In a world where experience is key, such a designer would gain far more knowledge and understanding from this than in any other position where the design feedback loop wasn't so strong..

I'm just the type of person that hates revisiting the same work over and over again.

 

[Scott_Arm]

how can things like having hard data showing points on a screen in real time where there are choke points on the maps you designed or that users are using too much ammo for the particular level (based on your computer data) so you can adjust it for the game to flow, be considered "catering to a focus group"?

the point of the story is the data points they are collecting in this play testing as well as from the Beta where every user's footsteps and shots fired were recorded as hard data and processed.

They are also monitoring the players expressions to see how satisfied or frustrated they are as they play the game. The pop up boxes that ask the players to rate the segment that played etc. This all fits into a focus group method. They have a Phd psychologist running the place. That's why I decided to use the term "focus group." It's not exactly a focus group, so maybe the terminology is wrong, but there are some similarities. But really, who cares if it's a focus group or not? We all read the article, and arguing about semantics is off topic.

I'm not even complaining about catering to focus groups. I think the way they're doing testing is awesome, if it provides a better gameplay experience.

 

[Tap In]

They are also monitoring the players expressions to see how satisfied or frustrated they are as they play the game. The pop up boxes that ask the players to rate the segment that played etc. This all fits into a focus group method. They have a Phd psychologist running the place. That's why I decided to use the term "focus group." It's not exactly a focus group, so maybe the terminology is wrong, but there are some similarities. But really, who cares if it's a focus group or not? We all read the article, and arguing about semantics is off topic.

I'm not even complaining about catering to focus groups. I think the way they're doing testing is awesome, if it provides a better gameplay experience.

fair enough :smile:

 

[archangelmorph]

I'm just the type of person that hates revisiting the same work over and over again.

(Are you a level designer..?)

So am I!! But being a programmer we don't have much of a choice (i.e. bug fixing)..

& at the end of the day if it makes the game better then the time spent reworking is going to be worth it!!