*spin-off* Activision's Call of Review Hero: Controversy: Spin-Off

[Scott_Arm]

I just explained, Joshua. I'm not dancing around anything. If a certain feature is expected[/ B] of games, then not having it is bad, yes. Maybe there's a compromise involved that will totally pay off. You seem to be under the idea that you're springing gotchas on me, when all along I've said that reviews can be subjective while being supported on objective arguments. Not having a standard feature isn't an immediate 'it sucks', but I do expect a reviewer to examine why this standard feature was not included.

So it's only bad not to have it because it is expected, and not because of anything related to image quality or graphics.

So when we make the criteria for what is expected, how do we choose? Is it based on image quality or is it just because another game had it so this one should too?

I understand that you'd like to see certain talking points mentioned in each review, but I don't understand how anyone would ever decide what those talking points are. Different things will be important to different people, and vary depending on the type of game. Even a common buzzword like depth of field is impossible to evaluate on its own. You can hope that reviewers will point out the strengths and weaknesses in each game they review, but standardizing the sub-topics of the reviews is impossible.

 

[obonicus]

So it's only bad not to have it because it is expected, and not because of anything related to image quality or graphics.

Except it directly relates to image quality and graphics. DoF will never be standard because it doesn't always fit. But I didn't bring up DoF, you did. Yes, we are talking about a smaller set of criteria -- we're talking about more important criteria. So far you haven't really rejected any of the criteria I set forth, you just brought up a slippery slope.

So when we make the criteria for what is expected, how do we choose? Is it based on image quality or is it just because another game had it so this one should too?

Based on image-quality, of course. Will you argue that, in isolation, AA doesn't improve image quality? That 720p doesn't look better than 640p? That 20fps will look bad compared to 30fps? That's the sort of thing we look at. Maybe next-gen it will be shadowing, games that have sub-standard shadowing need to make it up some other way.

I understand that you'd like to see certain talking points mentioned in each review, but I don't understand how anyone would ever decide what those talking points are. Different things will be important to different people, and vary depending on the type of game. Even a common buzzword like depth of field is impossible to evaluate on its own. You can hope that reviewers will point out the strengths and weaknesses in each game they review, but standardizing the sub-topics of the reviews is impossible.

This is a slippery slope. We can take a first step in this first direction without going down the path of insane arbitrary graphical criteria. B3D isn't a good sample for this, but the tech forum has a thread on what sort of things should be standard next-gen, which is exactly what I'm talking about. That's how hard raising this sort of criteria would be.

 

[Acert93]

So lets see your extensive checklist and sample reviews--since we all are putting up strawmen and revel in the slippery slope whereas you clearly have a defined and functional concept that we should all presently be holding reviews for.

So where is this all important check list that every game should be held to and extensive details given in the written review?

I can even start:

1. Resolution
2. Anti-aliasing
- Technique(s)
- Level
- Pattern
- Adaptive / Edge Specific
- Design Considerations (oh wait, take this one out)
3. Texture Fidelity
- Filtering Method(s) (Bilinear, Trilinear; any AF; quality of the AF pattern; Specify surfaces applied to and variances e.g. does it work with Parallax Maps)
- Texture resolution by object (characters, faces, weapons, ground, sky)
- Number of texture layers per object + percision of each (e.g. using low diffuse with a high level normal)
- LOD bias
4. Characters
- Number on typical screen
- Poly count
5. Character Animation
- Animation frames (1k? 2k?)
- Transitions
- Framerate
- Physics based
- How many animation blends
- MoCapped or Hand Tweened, or both
6. Guns
- Poly count
- Texture layers
- Texture resolution
- # games in game
- Degree of modification
- Technical details of the muzzle flash
- Muzzle flash case dynamic shadows on dynamic objects?
- Does the character show the weapon on the shoulder when not used?
- Does it rust?
7. Framerate
- Ave
- Max
- Min
- % Dropped
- % Torn + where torn (i.e. not all tears are visible--gotta explain that too)
- This needs to be done for each chapter of a game
- Oh, and we need all these figures for every possible resolution output (480p, 720p, 1080p, etc)
8. HDR
- Color Format
- What HDR techniques are used (tone mapping, bloom, iris effects, etc) How long is the iris effect, how pronounced the bloom, any washing out of the image? Need full histograms for the readers
- Noticible Banding
9. Shadowing (we want resolutions, techniques used for each object, how they are filtered and any edge smoothing, what is the LOD, not only is it a unified or dynamic system but readers need to know what role normals, AO maps, and other techniques are used with the shadow maps or other techniques, etc)
10. Indirect lighting... Indirect Shadowing...
11. Shader Effects
12. Specular Techniques
13. Transparency buffer resolution
14. Cube maps (resolution, update rate, is it anti-aliased, are there dynamic objects, any frame latency, etc)
15. Lens Flares ........

As you can see this is just the tip of the iceburg... But I was nice enough to start your list.

 

[Scott_Arm]

Except it directly relates to image quality and graphics. DoF will never be standard because it doesn't always fit. But I didn't bring up DoF, you did. Yes, we are talking about a smaller set of criteria -- we're talking about more important criteria. So far you haven't really rejected any of the criteria I set forth, you just brought up a slippery slope.

Well, depth of field isn't really different than any other feature.

Based on image-quality, of course. Will you argue that, in isolation, AA doesn't improve image quality? That 720p doesn't look better than 640p? That 20fps will look bad compared to 30fps? That's the sort of thing we look at. Maybe next-gen it will be shadowing, games that have sub-standard shadowing need to make it up some other way.

I would never evaluate any of them in isolation. If you had exactly the same game with AA than one without, then yes, the AA game would look sharper. But that would never happen because to add AA you have to trade something else. None of these things are free. That's why you have to evaluate as a whole and you can't reprimand a game for lacking a particular feature. Same goes for resolution, frame rate, shadowing, lighting, post processing, blur and all other shader effects. All of those things are used to create a complete image. The sum of the parts is important, not the parts on their own.

This is a slippery slope. We can take a first step in this first direction without going down the path of insane arbitrary graphical criteria. B3D isn't a good sample for this, but the tech forum has a thread on what sort of things should be standard next-gen, which is exactly what I'm talking about. That's how hard raising this sort of criteria would be.

I haven't read that thread, but I imagine that opinions are divided, or there is a vocal minority that disagrees. But once again, do you have one set of criteria for 2D platformers, and sub sets of criteria for sprite based vs rasterizer? Is there another set of criteria for racers, and one for games that are supposed to look like cartoons, split for subtypes that are rpgs, action games or kids games?

I have no problem with reviewers that want to delve into a little more of the technical aspect, but I don't think there is a way to standardize it, or make qualitative judgments based on a check list.

 

[obonicus]

Well, depth of field isn't really different than any other feature.

True, but Joshua has a problem with Depth of Field particularly, so I'd rather move away from it.

I would never evaluate any of them in isolation. If you had exactly the same game with AA than one without, then yes, the AA game would look sharper. But that would never happen because to add AA you have to trade something else. None of these things are free. That's why you have to evaluate as a whole and you can't reprimand a game for lacking a particular feature.

Except you are. You're talking about trade-offs, that means that for a positive you have a negative -- implicitly you're assigning a negative worth to certain choices. And what you're talking about isn't so different from what I'm talking about, but rather than internalize this discussion( and realize that as a B3Der you're far more knowledgeable about these things than your average reviewer) you externalize it. Simplifying CoD to two major points; the game sacrificed HD resolution for 60fps. These are facts. Was it worth it? That's a completely subjective. The young hotshot fighter pilot will say yes, that he'll take any compromise in favor of more responsiveness. The old fogey will say that neither his reflexes nor his eyes are good enough to pick up 60fps and that he'd rather have the eye candy. But if you bring up both points in your review you open the field up for your readers to actually come to different conclusions, if they don't find your conclusion satisfying. Without these points you have only your conclusion, you have no frame of reference.

Same goes for resolution, frame rate, shadowing, lighting, post processing, blur and all other shader effects. All of those things are used to create a complete image. The sum of the parts is important, not the parts on their own.

They're both important. At the end, the final product is what matters, but a fundamental part of what I've been arguing for is exactly the deconstruction people do here. If we speak of trade-offs, then for every sacrifice there should be an effort to make up for it. Much of these trade-offs we can actually quantify.

I haven't read that thread, but I imagine that opinions are divided, or there is a vocal minority that disagrees. But once again, do you have one set of criteria for 2D platformers, and sub sets of criteria for sprite based vs rasterizer? Is there another set of criteria for racers, and one for games that are supposed to look like cartoons, split for subtypes that are rpgs, action games or kids games?

Well, like any good discussion, opinions are divided, sure. There's also no intention to actually publish a list of 'standard' features at any point, so it's less focused that it could be. (Which isn't a bad thing, of course.)

Well, yeah. This is all limited to 3d. 2d may have its own set of criteria; though 2d is so advanced that I don't even know if the rasterizer techniques that were wizardry in the 90s would even be useful.

I have no problem with reviewers that want to delve into a little more of the technical aspect, but I don't think there is a way to standardize it, or make qualitative judgments based on a check list.

I disagree. I think each publication could compile a list and follow it consistently. I think consistency would be a breath of fresh air compared to what we have currently, which is a 'fire and forget' approach, in which the publication uses bylines to bypass any issues with accountability, while reviewers themselves use 'it's just an opinion' to do the same thing. The small list I brought up, for instance, is a good start. With more technical know-how you could actually go over many different games and see what other aspects are standard.

 

[dobwal]

I have no problem with reviewers that want to delve into a little more of the technical aspect, but I don't think there is a way to standardize it, or make qualitative judgments based on a check list.

You don't have to make a qualitative judgement based on a check list. But its no excuse not to include one.

If I were to stereotypically judge gamers based on the typical game sites with huge readerships, I would think that gamers are the most depthless bunch of hobbyists around. Typical review sites are just new sites with very shallow reviews.

I am sorry but I refuse to believe that these sites, that don't even try to cater to the more sophiscated gamers, employ the most efficient and effective method of reviewing titles.

http://autos.msn.com/research/vip/review.aspx?year=2010&make=Ford&model=Fusion&cp-documentid=875880

http://autos.msn.com/research/vip/review.aspx?year=2010&make=Ford&model=Fusion&cp-documentid=1081701

These are a reviews of a Ford Fusion. A Ford Fusion!!! Your average car buyers gets excited about a Ford Fusion as much as your average gamers gets excited about a typical crappy EA game. 95% of people who bought this car could care less about some of the info provided by these reviews because most buy these cars because its affordable and get them from point A to B.

Nevertheless, these reviews, while being subjective and objective, provide a plethora of information in a concise manner with 10X amount more technical data then your average multi page game review provides. If a no frills car can get this much attention to its review from a site thats in no way a diehard car site, then why can't GameSpot or IGN give a similar or better treatment to even big name releases.

Its simple. GameSpot and IGN know that their typical reader isn't someone who's your typical forum contributor of B3D. Ever visit their onsite forums? Generally, review sites tend to attract the younger and less mature crowd, who don't typically care about AA or AF. I doubt whether the majority of its users even know what the terms mean. And I haven't encountered a lot of review sites that even try to educate their web goers. They cater to a different crowd that typically younger and less serious about gaming as a hobby.

I believe that if more serious and financially well off sites with a more mature and knowledgeable readership really gave it a go in terms of reviewing games, they would produce way better quality reviews with a plethora of technical info and could describe differences in quality that wasn't purely subjective in nature.

 

[Acert93]

Well, depth of field isn't really different than any other feature.

True

Agreed.

but Joshua has a problem with Depth of Field particularly, so I'd rather move away from it.

No, I don't have a problem with DOF.

I quite like the effect when used correctly--I loath it when tacted on as a check-box to appease the "Lens Flare" folks and / or poorly implimented. I have a problem with DOF being used as a pet criteria as much as any of the other effects I mentioned. So Scott's point stands.

 

[Acert93]

These are a reviews of a Ford Fusion. A Ford Fusion!!! Your average car buyers gets excited about a Ford Fusion as much as your average gamers gets excited about a typical crappy EA game. 95% of people who bought this car could care less about some of the info provided by these reviews because most buy these cars because its affordable and get them from point A to B.

Nevertheless, these reviews, while being subjective and objective, provide a plethora of information in a concise manner with 10X amount more technical data then your average multi page game review provides. If a no frills car can get this much attention to its review from a site thats in no way a diehard car site, then why can't GameSpot or IGN give a similar or better treatment to even big name releases.

http://xbox360.ign.com/objects/142/14281102.html

CoD4 MW2 Review.

Maybe we are looking at different sites/pages but IGN gives all sorts of technical data on the game (cost, release date in various territories, platforms, number of players on a console, number of players online, general idea of typical game length, IGN Review Rating, Press Average Rating, over 900 User Reviews, 40+ videos and another 40+ screen shots, etc). On top of a 3 page text review and summary they also have a full fledged video review and over 100 (!) news articles, features, previews, etc concerning

Sure, they aren't giving you certain details (how many terrorists are killed in the game? Civilians? Better yet, what is the sample rate and decibel level of an explosion at 10ft game space? At 100ft? Of the handfuls of HDR approaches which one do they use?) But then again when you have over an *hours* worth of gameplay footage to preview a lot of questions are answered--it is like a test drive. CoD4 MW1 had a proper test drive (beta) and MW2 is the sequal to a very successful title.

This generation, more than any other, due to free online demos of many titles negates a lot of the need for media sites to tell consumers about a games visuals, sound, controls, etc. Things like game length, how is the quality (does it get better or putter out?), story progression, etc become much more relevant.

Consider games are a $60 product no one depends on for safety or transportation to work (like a car--that costs over $20,000 new) the amount of outlets reviewing games, dispersing media, etc is actually quite amazing.

I also don't think consumers need 10 pages of technical details on every title to quantify whether a game is visually good or not.

I also don't need 10 pages to quantify the quality of the sound. I am QUITE surprised that this entire stream of arguements hasn't filtered down to Sound because these same issues plague the review of audio in games--more so.

Anyhow, if we were reviewing software renderers and had more access to technical data and had a direct input on asset quality and creation a LOT of technical reviews would surface. But games are to be played. People don't apply the method to review a car, a book, or cheesecake to how they review a game.

 

[the ignoramus]

Data about fps averages, and input latency should be included in the reviews from major sites. These kinds of things are not subjective at all. Low framerates (sub 30 fps averages) and GTA IV/HS/KZ2 levels of input latency are not acceptable compromises in any way, shape or form. Image quality is trickier, but any decision that results in horrible image quality (sub HD and no AA) and sub 30 fps performance isnt an acceptable compromise either.

 

[Phil]

Here's some food for thought:

If we look at the main gaming market as a whole, we have to assume that most consumers are not of the technical orientated mind. They hardly know the concept of framerate, tearing, DOF, Anti Aliasing techniques or any features. They might see it if you point it out to them, but most of them will simply not care and judge graphics, sound, even the game-mechanics in a very subjective and non-objective manner.

Reviewers will bear this in mind and to what kind of crowd they are writing their reviews for. Suffice to say, technical minded B3d folks or hardcore gamers that spend most of their times writing about games in forums are perhaps not their core audience. The average Joe they're writing for, doesn't understand most technical issues. For him, the biggest factor if a game looks good is down to art-assets and direction. Certainly not framerate.

Despite all this, and for the sake of gaming, I still would have to agree with Obonicus's argument that more objectivity and a more technical orientated metric should be required in reviews. Why? Because for one, they can be measured in black or white just like the technical specs of a car in a car-magazine review. It also makes the reader more aware to what there is to graphics and that there is more to it than just art and how it comes together on the screen.

Right now, gaming is heading in a direction where visuals is the biggest factor to selling a game. See the Mike Acton/Insomniac interview to see that they themselves are complaining about the little technical knowlege in the gaming population and that to get good reviews (and thus sell more) they have to up the visuals and sacrifice framerate. Perhaps, if more reviewers cared to mention framerates and other technical orientated features in an objective manner, people would actually understand why the latest 60 fps title looks worse than most 30 fps games. But at the same time, they'll also come to realize that the game on the other hand offers better playability by delivering a smoother framerate. If reviewers don't touch on these factors in their reviews more often, how will the average consumer ever understand what seperates most games?

I guess it's not too much to ask for more objective black/white numbers in our reviews.

 

[dobwal]

http://xbox360.ign.com/objects/142/14281102.html

CoD4 MW2 Review.

Maybe we are looking at different sites/pages but IGN gives all sorts of technical data on the game (cost, release date in various territories, platforms, number of players on a console, number of players online, general idea of typical game length, IGN Review Rating, Press Average Rating, over 900 User Reviews, 40+ videos and another 40+ screen shots, etc). On top of a 3 page text review and summary they also have a full fledged video review and over 100 (!) news articles, features, previews, etc concerning

Sure, they aren't giving you certain details (how many terrorists are killed in the game? Civilians? Better yet, what is the sample rate and decibel level of an explosion at 10ft game space? At 100ft? Of the handfuls of HDR approaches which one do they use?) But then again when you have over an *hours* worth of gameplay footage to preview a lot of questions are answered--it is like a test drive. CoD4 MW1 had a proper test drive (beta) and MW2 is the sequal to a very successful title.

This generation, more than any other, due to free online demos of many titles negates a lot of the need for media sites to tell consumers about a games visuals, sound, controls, etc. Things like game length, how is the quality (does it get better or putter out?), story progression, etc become much more relevant.

Consider games are a $60 product no one depends on for safety or transportation to work (like a car--that costs over $20,000 new) the amount of outlets reviewing games, dispersing media, etc is actually quite amazing.

I also don't think consumers need 10 pages of technical details on every title to quantify whether a game is visually good or not.

I also don't need 10 pages to quantify the quality of the sound. I am QUITE surprised that this entire stream of arguements hasn't filtered down to Sound because these same issues plague the review of audio in games--more so.

Anyhow, if we were reviewing software renderers and had more access to technical data and had a direct input on asset quality and creation a LOT of technical reviews would surface. But games are to be played. People don't apply the method to review a car, a book, or cheesecake to how they review a game.

You don't need ten pages of technical data. But a standard chart would be nice. And a summary on how a title may be negatively affected or able to overcome any existing technical flaws. There is no reason not to devote a paragraph of two to the technical aspects of a game. Average framerate, resolution, AA, AF, etc and and other technical don't require alot of realestate to adequately cover.

Furthermore, as a reviewer you shouldn't be assuming that all their readers have played a demo. As for it being a $60 game, I can find more tech specifications on reviews for cheap ear phones. And ear phones are for listening but yet there are provisions of technical data so the reader can be more informed.

http://reviews.cnet.com/headphones/...-7877_7-32161239.html?tag=contentMain;compare

Reviews are suppose to inform the reader of more than just the subjective opinion of a reviewer's experience. No one reads reviews because they are entertained by how John Doe So and So likes or dislikes his experience with a title. They are meant to help readers judge the quality of a title and there exists technical and objective aspects of games that can effect a reader's own experience so there is no reason not to include and comment on them.

 

[Acert93]

I don't think your analogy works at all.

Earphones are for Outputting Audio, so you review the Device's ability to do that. Music is to be Listened to, so you review the artistry, theme, lyrics, instrumentation (I cannot think of a music review that detailed how every track was used, the exacting recording percision of each instrument, etc instead generalities). Games are for Playing, you review the gaming experience. Renderers are for Outputting Images, you review their ability to do that.

Games aren't Renderers.

And yet we still get info about tearing, framerate, and graphical issues that impact the experience. A chart may be nice, but #'s don't always tell the story (one game can be smooth at 30Hz while another is NOT smooth at 30Hz).

And as pointed out you guys keep going back to certain features (e.g. AA) when there is more than one way to skin a cat. Is no one going to challenge the premise that a game with out MSAA or low levels can have less visible aliasing than a game with a higher MSAA level?

So in a review I could say:

* "Game ABC has a lot of jaggies;" and
* "Game XYZ has a crisp clean look"

And communicate clearly. Using the "chart" method of SUBJECTIVE SELF SELECTION OF FEATURES you could get the wrong impression:

* "Check-Box: Game ABC has 4xMSAA;" and
* "Check-Box: Game XYZ has 2xMSAA"

And yet there is more to the story. Is it adaptive AA? Is it applied to every object in game? Is it only applied to certain surfaces (e.g. GeoW2 is on static objects)? What filter / pattern is used (NV and ATI use different filters which CAN cause more visible aliasing at the SAME MSAA level)? Has a shader been employed to soften edges? Are MB and DOF used in ways that minimize visible aliasing? How about MRT, are they ordered in a way that promotes aliasing (see 4xMSAA games with very aliased bloom)? What impact does the asset design have on aliasing? Is the art direction high contrast or lower contrast, and does this affect the visibility of aliasing? Is there super sampling? Is MSAA applied to cube maps? Particles? And what resolution are the source buffer for the particles? -- Simply put, telling consumers the MSAA-level doesn't tell them anything about what they will see on screen.

I don't see anyone here addressing the actual complexity of the above paragraph and showing how it is even POSSIBLE to write an OBJECTIVE review that takes into account all these variables to objectively give the reader all the evidence, in a game review, that remains a game review.

It is niave to think a "chart check box" accurately reflects the game. And a running *commentary* on the technical implimentation has little place in a *game review*.

Further, anyone with the knowledge to understand the hardware can understand a more generalized statement. e.g. "[--Game--] has a generally clean look. The textures are crisp with a lot of detail although the beautiful 3D rocks on the road are blurry at a distance. The game also has minimial jaggies but can be spotted in areas with intense light. The game has beautiful lighting and shadows that really add to the atmosphere, our only gripes being shadows cast by the flashlight are a little rough and there seems to be some delay in the loading of some ambient shadows (which look beautiful). Framerate is a solid 30fps until the later levels where there is some visible tearing."

Converting this is a chart of "technical check boxes enables" can be deceptive. e.g.

* AF: 16xAF
* AA: 4xMSAA
* Shadows: ESM
* Indirect Illumination: PRT, SSAO
* Framerate: Ave: 27, Min:15, Max: 30
* Tearing: 5% Torn

Great, right? Now it all falls apart:

* AF: 16xAF => Actually NOT applied to POM. Further, the developer used surface dependant application and users can find surfaces without AF
* AA: 4xMSAA => MSAA not applied to alpha or cube maps. Sorting order borked causing extreme alaising on HDR effects. Textures lack AA effects. Art is high contrast bright colors with long straight edges which emphasizes aliasing, especially in well lit areas due to HDR issues.
* Shadows: ESM => Baked for static geometry, ESM corner cases not fixed with low filtering, shadow map resolution only 256x256, dynamic lights usually cast low resolution unfiltered shadow maps
* Indirect Illumination: PRT, SSAO => PRT only for static objects, SSAO slowly updates and done on a quarter buffer. No work around for large objects not in screenspace
* Framerate: Ave: 27, Min:15, Max: 30 => First 8 of 10 chapters are 29.98/27/30. Last 2 chapters are 24/15/30 with a couple stretches of significant drops that don't affect gameplay as the focus shifts to realtime cinematic like effects
* Tearing: 5% Torn => Less than 1% frames torn first 8 chapters and usually not in a prime visible area, last 2 chapters exhibit significant tearing in certain segments but it typically not a problem

It is easy to see how this bunny trails necessarily runs into other areas if we are to be objective and not self selecting. e.g. Cube maps can be a problem in some games (are they reversed, resolution, anti-aliased, update rate, latency).

Every game has different criteria, too, but to be fair and objective you need to cover every detail (e.g. Cube Maps: None... oh wait, but the game didn't need them! And yet the paragraph before found in general reviews communicates without obstructing with details that may not in themselves be even accurate, "[--Game--] has a generally clean look. The textures are crisp with a lot of detail although the beautiful 3D rocks on the road are blurry at a distance. The game also has minimial jaggies but can be spotted in areas with intense light. The game has beautiful lighting and shadows that really add to the atmosphere, our only gripes being shadows cast by the flashlight are a little rough and there seems to be some delay in the loading of some ambient shadows (which look beautiful). Framerate is a solid 30fps until the later levels where there is some visible tearing."

If I were reviewing a renderer a review of a much more technical nature is in view.

But gamers are buying a game. The issue is to communicate what their audiance needs.

Maybe there is a market for technical reviews--but be warned, I have already seen the unwillingness from you guys to do so. And an "objective" technical review that is not fully objective, self selecting, and technically in error is gonna be criticized for being inaccurate and biased.

The fact Gamersyde, IGN, Gametrailers, GameSpot, etc actually put up video of said games actually goes a looong way to address consumer questions, "What will the real game look like?"

Do consumers need to know anything more than, "What does it look like on screen?"

Doesn't sound like people want a game review, and don't care what the game looks like, but want a technical data sheet on a very limited number of Buzz Words. The goal for more objective reporting is good, but this bunny trail won't give you what you want.

 

[dobwal]

Earphones are for Outputting Audio, so you review the Device. Music is to be Listened to, so you review the artistry. Games are for Playing, you review the gaming experience. Renderers are for Outputting Images, you review their ability to do that.

Games aren't Renderers.

Regardless, you reviewing an experience. How well a game stands up technically has a great amount of influence on the potential of one's overall experience.

Since you like to go extreme. You, I and everyone else would readily investigate the technical aspects of how any particular artist's music was recorded if a quarter of music on Walmart's shelves was produced by playing the music on a 8 track player which was then recorded by a boom box which was played back on the back of moving truck in the middle of downtown traffic and then finally recorded on a iphone and subsquently burned to a CD.

Would a movie's framerate not be important aspect of any movie review if they typically ranged from 2-24 fps?

What's naive is thinking that the only way to provide a relevant amount of technical information would require one to generate a large amount of it and discuss every point. Some technical data is more relevant then others. Thus, while you may have a standardized technical chart with alot data, only those that have an appreciable effect on the experience needs to be discussed within the body of the review.

In most reviews it isn't the technical discussion thats draws out the review, its often the subjective part as saying "average fps is 60 fps, never dropping below 45 fps" is way more clean and informative than descriptions produced subjectively because even using a simple such as "smooth" are not definitely defined. Meaning you have to get way more wordy to allow the reader to interpret the writer's definition of "smooth".

 

[zed]

So where is this all important check list that every game should be held to and extensive details given in the written review?

I can even start:

1. Resolution
2. Anti-aliasing
- Technique(s)

etc

I agree Ive written about that here before, These things that quantifiable should be written down and added up
I forget what what numbers I came up with
lighting 60% of the final mark
resolution 10%
etc

As it stands now (I dont know if it was the same guy reviewing both games)
but killzone2 9.5 graphics + MW2 10 for graphics, with kz2 being available 6+ months previously is a joke

 

[Scott_Arm]

...Some technical data is more relevant then others. Thus, while you may have a standardized technical chart with alot data, only those that have an appreciable effect on the experience needs to be discussed within the body of the review.

Ok, which data is more relevant than others? I actually want to know an answer to this. Frame rate keeps coming up, and to me, it is fairly relevant, but honestly the way reviews are written it is usually addressed in terms the average person can understand. What else guys? I think Josh has pretty much trampled why discussing AA and AF is not relevant or useful in a review, and why summarizing features in a list isn't going to tell you how good a game actually looks. And the same goes for shaders and post processing effects, which is what Josh and I have been arguing all along. You make choices to reach an intended image, which is what is important. One feature is not negative and one is not positive. One is not better than the other, even. They are different.

 

[obonicus]

They've been brought up multiple times, Scott. The slippery slope argument of listing dozens of different criteria as if there's no way to select a subset of those doesn't work. You yourself have framed these things as 'trade-offs'. There are factors that are objectively better than others -- you haven't even disagreed on this. The main disagreement here is that you don't think a game can be deconstructed at all (I don't think you've actually thought out the repercussions on this notion). This is such a fundamental point of contention that we might as well be speaking different languages.

 

[Scott_Arm]

I agree Ive written about that here before, These things that quantifiable should be written down and added up
I forget what what numbers I came up with
lighting 60% of the final mark
resolution 10%
etc

As it stands now (I dont know if it was the same guy reviewing both games)
but killzone2 9.5 graphics + MW2 10 for graphics, with kz2 being available 6+ months previously is a joke

You are incorrect. Lighting should be 61.875 percent of the final score, and SSS should be about 4.9321 since it is not common or expected yet. We can reconvene and review those numbers on an 8 week basis as certain features become more/less desirable.

Joking aside, a few posts up you have an explanation as to why quantifying AA etc is a useless exercise, and you should probably read it. AA of the same "quantity" can have vastly different results in different games. So is 4x MSAA in one game equal in score to 4x MSAA in another game?

 

[Scott_Arm]

They've been brought up multiple times, Scott. The slippery slope argument of listing dozens of different criteria as if there's no way to select a subset of those doesn't work. You yourself have framed these things as 'trade-offs'. There are factors that are objectively better than others -- you haven't even disagreed on this. The main disagreement here is that you don't think a game can be deconstructed at all. This is such a fundamental point of contention that we might as well be speaking different languages.

Ok, then. Choose a subset. Someone do it. I want to know what I should think is important.

No, I do disagree with you entirely.

Tradeoffs. Do I want one post processing effect or another? One is not inherently worse than the other, they just achieve different results. Do I want high res shadows or low res shadows plus something else? Are high res shadows better than low res shadows? I wouldn't definitively say they were, because maybe the low res, with what you are trying to achieve is sufficient, and maybe that extra effect you gain improves the overall impression.

I'd love to play a game that was 480p that attempted photo-realism. Why not? In that situation is 480p worse than 720p? I'd say no. It's just a different choice. You can deconstruct games. I just don't think it is a useful exercise unless you're actually working in the industry as a programmer. I like to read about it because I find it interesting. Does it have any bearing on what I like to look at when I'm playing a game? No.

 

[Acert93]

That is subjective and allows for significant bias that abstracts certain points to the detriments of others, exactly the issue being criticized.

The reader is still to the discernment and whims of the reviewer who may overlook, skew, ignorantly pass over, or biasly interpret data or present such that gives a false sense of objectivity because it is veiled in the false pretense of objectivity.

I don't think my point is extreme. What it is, though, is pointing out how subjectivity reigns in regards to what is "objective" and what is "relevant" in terms of something as peripheral as a game review.

Classic example of this flaw is technical specs in the hardware themselves. e.g. a TBDR and IMR operate differently and therefor will consume resources in different patterns. AMD's and NV's GPU have widely different performances in shader bound apps that defy linear relationships to flops and even go beyond shader architectures into what are non-shader bound dependancies that surprise. Consoles are the same: What happened to all those 80+ GFLOPs in NV2A? Are we to compare those to a Intel C2D? Look how most developers say Xenos is faster than RSX, yet it has lower FLOPs and TMUs and RSX has more pixel writing ability on the GPU. This generation has been yet another learning lesson that you cannot just say, 'An SPE has more flops per core than a C2D core, so lets just put the #s side-by-side for consumers' -- that "objectivity" isn't objective at all! What is your work load? How well adapted is your algorhythm for the architectures? Are there dependencies? Is the problem better suited for Caches or LS that may cause significant performance issues regardless of flops?

Technical Numbers without a context is dancing around the pretence that some technical number, enabled/disabled, etc can determine anything relevant. Without a context it is irrelevant.

And in games the context is the visual image on the screen, which invariably converges with the artistic characteristics of what is onscreen.

And to limit discussion to what the author feels is "relevant" is extremely subjective.

I feel see how only those things having an "appreciable effect on the experience" is any different from what reviewers are already providing.

I would really like to see a review from one of you guys demonstrating how such a review (a) targets consumers and is comprehensible (b) deftly handles these technical issues in addressing visual issues that affect the experience while (c) being consistantly objective and (d) without diminishing the artistic elements of the game or the interaction of graphics with the gameplay experience.

If you can write 5 reviews of games that can pass muster without being spotted for inconsistencies, subjective analysis, self selecting of features relevant to the reviewer while skewing others readers my inquire concerning, etc I would be very impressed--and suggest you start your own reviewing website because I would read it

But as my examples show, even with my meager knowledge of 3D rendering issues (far above your casual gamer) I see no end to the bunny hole if I am to claim "objectivity" and to be considerate of issues consumers may have, regardless of my own perception.

I don't find it surprising that me, the detractor, is the only one willing to generate a check list with some variables.

Why is no one up to that challenge? Is that not what you guys are calling for? Should not the proponets be able to present a clear case?

 

[obonicus]

No, I do disagree with you entirely.

Tradeoffs. Do I want one post processing effect or another? One is not inherently worse than the other, they just achieve different results. Do I want high res shadows or low res shadows plus something else? Are high res shadows better than low res shadows? I wouldn't definitively say they were, because maybe the low res, with what you are trying to achieve is sufficient, and maybe that extra effect you gain improves the overall impression.

I'd love to play a game that was 480p that attempted photo-realism. Why not? In that situation is 480p worse than 720p? I'd say no. It's just a different choice.

No one's talking about post-processing, Scott. You're skirting the actual features I did bring up. When I asked if, on its own, 640p is ever better than 720p your response was that you can't even look at such things on their own. You didn't actually answer the question I asked. Instead, you disagree with the very premise that you can deconstruct graphics at all. This is fundamental, we can't reach any sort of consensus. This also applies to your shadow question; there's a clear idea of superiority (but the question of whether some feature is actually a negative is, again, based on a previously agreed-upon standard) you say that you can't isolate one aspect of graphics from the others. But you also can't isolate the 'I like it' factor from the gameplay, or from personal bias (which itself will influence the entire review).

So I'll throw a different question at you, which I hope you'll actually answer: given that all graphics analysis must be entirely subjective, how do you extract information about a game's graphics from a review written by a reviewer you don't know?

Edit: Just saw your edit. As to the subset, it's been presented multiple times. I've reduced criteria to a very base set which you have nonetheless rejected because you don't think any of these things can be separated from the whole.