Metacritic in 2014

[Shifty Geezer]

I agree, but it'd have to be a unformily applied policy. And I bet good money that user reviews of broken games which were afforded high scores by the gaming press aren't anything like as forgiving as those professional reviewers.

Halo:MCC collection, no negative reviews from the press, 28% negative from players grumbling about broken online.
DriveClub, one negative review from press, 37% negative from players grumbling about it being broken.

The professional, filtered Metacritic score is, seems to me, less representative than the user scores. And it's this filtered score that determines developer bonuses rather than the actual players scores, I guess because it's the initial professional scores that determine whether a game sells or not and what makes the money. Whether the people buying the game like or not doesn't so much matter (to the suits, but it will do!).

 

[Nesh]

But the score doesn't, and Metacritic presumably doesn't factor in analysis unless every article's read and (subjectively) weighted. Chances are they just rate some sources as more important than others without reading the content.

It's not irrational, just a different perspective. I agree with what you're saying, that some reviews are less rational than others. Effectively, sometimes idiots review stuff and make illogical arguments. However, the people that are going to play the game are also going to include the same types of people. One illogical review complaining about Dead Space and giving it a 3/10 is going to resonate with similarly illogical players who'd find the same illogical issues with the game. Therefore, if 2% of reviews rate it 3/10, chances are 2% of players would rate the game 3/10, and so the number used to represent the appeal of the game to the wider audience is going to have to reflect that. And if you don't want to consider outliers, you can filter them out statistically - you don't need to use a weighting system. Point is, without knowing precisely what Metacritic's score is trying to be, we don't whether some reviews are less important than others. If the assumption is that Metacritic's job is just to aggregate public opinions into an average to represent game appeal to the general populace, then it should treat all reviews as equal. IF they are wanting to produce a score representative of a particular subset of the populace, than they should filter their data accordingly to represent that populace.

How it is not irrational to claim all are equal when it is acknowledged that some games are reviewed by "idiots" or a review is "illogical"? That alone demonstrates that not all reviews are equal. Of course a badly written review may resonate with some people. Outliers will always exist. The perspective of such outliers though is the perspective that is not shared by the majority of the targeted audience.

I am not supporting metacritic as being flawless. To the contrary I agreed that we dont know their methods and hence cannot comment on its level of accuracy. I agree with the logic though of using weights. Who knows? The weights may have even resulted after a regression they did on an actual statistical model.

This is how we derive weights scientifically with the least bias possible. We make models using hypothesis, we test for bias and minimize errors and we get statistical significances and values that act as weights (i.e this variable/this reviewer holds less statistical significance/has less value of effect on the dependent variable)

Furthermore, those professional reviews you speak of are at times nonsense themselves anyway. I've read reviews with a grumbling tone that get a higher score than the wordage describes. I've also read reviews where the reviewer clearly doesn't like the genre or franchise. Or introduces a criteria to a game that others aren't judged by. eg. LBP3 on EG was downrated for being buggy. Probably deserved, but other games like ACU weren't downrated for being buggy. Hence their scores are higher where the scoring isn't objective and equitable.

There are too many variables in play when a review is created, so it's probably best just to run with consensus across all opinions, maybe with some statistical removal of outliers.

Yes exceptions can exist in all cases. These arent always 100% free from bias either.
The frequency and how much they share that bias though also play their role.

 

[Nesh]

The professional, filtered Metacritic score is, seems to me, less representative than the user scores. And it's this filtered score that determines developer bonuses rather than the actual players scores, I guess because it's the initial professional scores that determine whether a game sells or not and what makes the money. Whether the people buying the game like or not doesn't so much matter (to the suits, but it will do!).

User score reviews on metacritic can also hold bias that often can be even worse. Anyone can comment and provide a review with the sole purpose if they want to make a smear campaign against a game (for whatever reasons). You can see scores ranging from 10 to 0 for the same game.

 

[Shifty Geezer]

How it is not irrational to claim all are equal when it is acknowledged that some games are reviewed by "idiots" or a review is "illogical"?

That's one perspective! I know people by and large have great difficulty understanding alternative perspectives as opposed to alternative interpretations/conclusions, so maybe it's not possible for you to see this from another viewpoint which is that Billy comes from? I'll give it one more shot and then drop it.

There is one perspective that rates different people's viewpoints as being more or less valid based on a criteria such as smarts. By that criteria, some people are 'idiots' and their viewpoints are readily dismissed due to being irrational/incoherent. When forming a game score from viewpoints, the 'idiots' views should not be included.

There is another perspective that says, "peoples is peoples" and doesn't discriminate on views based on a selection criteria because, whatever the view, no matter how crazy it may seem, it's how someone felt and someone else out there will agree with it. An irrational viewpoint shared by 80% of people is going to reflect the views of those people, so that irrational score has to remain part of an aggregate of all people's opinions.

Both are legitimate approaches to the idea of creating an aggregate score with different end goals regards what their score represents. Neither is wrong in itself, and is only wrong when used incorrectly to try and attain the alternative representation.

Spoiler

It's actually akin to democracy - should you let everyone, even the hideously prejudiced, the uninformed, the 'too stupid to comprehend', have a say on policies? Or should you only allow a selection of the populace, those deemed worthy, to have a say on policy? If the latter, who do you filter out? The selection criteria is going to affect the policies. eg. Should we go to war? If we poll the general populace, 60% say 'no'. If we ask only those with an IQ over 180 as the only people smart enough to come up with a correct answer, the answer's 60% 'yes'. If we ask only those with an IQ of 100 or less as the only people with the social relevancy and investment to come up with a correct answer, the answer is 64% 'yes'. If we ask only mothers with a child of 2 or less as the only people with the emotional investment necessary to come up with a correct answer, we get an 84% 'no'. Each answer is a different perspective with it's own biases. And 'intelligent, rational' viewpoints are biased even if one doesn't think they are. An intellectual interpretation isn't the only valid one - an emotional response is just as 'human', and vice versa.

You're not wrong, but you're right only if one accepts the bias that you'd introduce to the score is wanted. That bias seems obviously the right thing to do from your POV, but in cosmic, philosophical terms it's just one option. For Billy, the idea of a completely unbiased score (biased by the contributions of the individual viewpoints) is the correct approach.

So the question of 'right' and 'wrong' regards approach comes down to deciding what the score should represent. Given that it's not possible to 100% accurately capture a filtered score to satisfy Perspective 1 and come up with a weighted score that shows the value of a game based on some reviewer criteria (experience, intelligence, etc.), but it is 100% possible to capture a general populace aggregate to show what the general populace - the 'smart' ones, the 'dumb' ones, the scientific ones and the artistic ones, the biased ones and the neutral ones, the gamers and the casuals - there's a strong reason to just go with Perspective 2 because it'll be an entirely accurate representation of a decidedly, and perfectly 'human', inaccurate measure. That's certainly what I presumed Metacritic was doing.

 

[Arwin]

After all this discussion I almost feel like making a new metacritic where you could set your own preference to trust or distrust certain sites or authors or even combinations of authors and genres, set corrections on certain site's scores, etc.

 

[Shifty Geezer]

Interesting concept, but it'd require a lot of knowledge from the users to select their trusted sources. You'd effectively have to learn all the sources and even individual users. Perhaps a better way would be to put in your own ratings for games you've played and the system finds reviewers who scored similarly. Where there is a pattern between a reviewer's (site's) scores and a user's, that reviewer (or site) could be added to the 'trusted' list and used for future scores for unowned games.

Hmm, that's actually a pretty awesome idea. It'd work just as well with movies and other stuff. Effectively a 'find like minds' process. If applied to something like a social network (Facebook), you could produce a 'similarity index' between users, and use that to generate recommendations. That'd be a trillion times better than some of the current category based or simple popularity based recommendations we see.

 

[ArcticCircle]

After all this discussion I almost feel like making a new metacritic where you could set your own preference to trust or distrust certain sites or authors or even combinations of authors and genres, set corrections on certain site's scores, etc.

Something like that would be great.

For example for me the greatest ever platformer or platinum style action game is worth zero but buggy, high production value AAA open world game is worth almost full price from steam sales if it gets at least somewhat good reviews. I'd say we could go even beyond genres, like allowing certain features like ironman (no deaths or game ends), speedrun friendly (in game timers, global rankings), allow z-axis movement (not tied to 2-feet high walls) ect ect. In general I love Steams "walking simulator" type of descriptions, even if they are not that specific.

Now that I think about it. I'd love if there was "review", that only would list all game mechanic features, art style, rules, world setting and brief synopsis of plot (if any). Something boring that only engineer would appreciate

 

[Deleted member 11852]

Now that I think about it. I'd love if there was "review", that only would list all game mechanic features, art style, rules, world setting and brief synopsis of plot (if any). Something boring that only engineer would appreciate

Reviews of old used to describe game mechanics in detail so you had a fuller idea of how a game played but modern reviews are very poor in this respect. Just take a look at reviews of GTA V or Shadow of Mordor, they almost all reference previous version (GTA) or other games (SoM) like Batman and Assassin's Creed. Reviews that work as a standalone reference for what a game is like to play are exceedingly rare.

I now read multiple gaming sites not out of choice but because if you read more of them you'll have more information to piece together to make an informed decision.

 

[Arwin]

Now that I think about it. I'd love if there was "review", that only would list all game mechanic features, art style, rules, world setting and brief synopsis of plot (if any). Something boring that only engineer would appreciate

That part is basically what I created www.techingames.net for

But I'll definitely consider the metacritic bit, I'll think about it ...

 

[Silent_Buddha]

How it is not irrational to claim all are equal when it is acknowledged that some games are reviewed by "idiots" or a review is "illogical"? That alone demonstrates that not all reviews are equal. Of course a badly written review may resonate with some people. Outliers will always exist. The perspective of such outliers though is the perspective that is not shared by the majority of the targeted audience.

Who determines whether someone is an "idiot" or not for reviewing something as subjective as a game? Or as subjective as how good a game looks?

Everything about a game is subjective. An opinion/review to one person might seem brilliant as it agree's with their own personal bias/desires while it seems idiotic to another.

All reviews are equal because all reviews aren't equal. There is no review site that is better than any other review site to every person on the internet. Just because you or I find a review particularly good, doesn't mean it is particularly good to your next door neighbor, co-worker, relative, or person halfway around the world.

With Metacritic in particular, it isn't the weighting that is a problem, it is that they don't reveal how they weight reviews. Hence, we do not know what bias Metacritic is imposing on it's aggregate review score. If it was revealed and I did not agree with their particular bias, I can just ignore them. If I agree with their bias, I would continue to use them.

We don't know if their particular bias is based on payment from companies, personal bias [Ethical? Moral? Sexist? Feminist? Religious? Other?], political bias, some distorted and/or delusional bias where they think what they are doing is scientific, or whatever other sort of bias.

I just choose to err. on the safe side and assume their bias isn't the same as mine and ignore them.

Unfortunately Joe Schmoe on the internet who doesn't look too deeply into Metacritic's biased ratings will just assume that's a fair assement of a game and likely miss out on a game he might have liked but got a bad Metacritic score due to the bias involved.

Yes exceptions can exist in all cases. These arent always 100% free from bias either.
The frequency and how much they share that bias though also play their role.

Not only can they exist in all cases. They 100% do exist in all cases. In something as subjective as games, media, music, art, what color is best, what is the ideal temperature, blah, blah, blah. It's all 100% biased. Just because the majority share that bias doesn't mean it isn't biased.

BTW - for a less biased post, feel free to replace all instances of bias with something more easy to read such as preference. I constantly used the word bias to show how absurd and childish it is to constantly use such a loaded and inflammatory word to attempt to influence how people think. However, if you wish to have a more inflammatory read of my above post, feel free to replace all instances of bias with a word such as agenda.

Regards,
SB

 

[Arwin]

Started working on making this idea concrete. Beta section for adding reviewers publications and reviews is already up, will work out the user filters and modifiers later. See the site thread for more details.

 

[Deleted member 13524]

After all this discussion I almost feel like making a new metacritic where you could set your own preference to trust or distrust certain sites or authors or even combinations of authors and genres, set corrections on certain site's scores, etc.

I think it would be more interesting if users could rate reviews instead of publications/reviewers. Then, each publication and/or reviewer would get a score based on an aggregated score of all their corresponding reviews. This way, it would be harder for a mob to upgrade/downgrade a certain website/reviewer due to social/political interests (i.e. I'll downvote this guy because he's a commie / he's SJW / he supports gamergate / he's a republican / he hates kitties / etc.).

This is more akin to what happens in the scientific community with the impact factor. Impact factor is never given directly to authors or publications. Their score comes directly from the performance of their articles (number of times they are referenced).

Consequently, an alternative meta-score for each game could be weighted according to this reviewer/publication score. A professional reviewer wouldn't be marginalized because of a single "badly written review" as everyone makes mistakes.
Also, scores from a crappy journalist/reviewer who is universally known for making crappy reviews wouldn't affect the meta-score much - if at all.

 

[Arwin]

Thanks for some good ideas. I'll see what combinations I can come up with that make some sense. Adding reviews, publications and reviewers seems to be working now btw. Hopefully I'll have some time tomorrow to take it a bit further.

 

[Rodéric]

I don't see the point, what's interesting in metacritic isn't the absolute press score but the range it's in and the difference with the user score...
Instead of allowing the user to bias press/critic score, wouldn't it be more interesting to have people give games a score and have the website find critics/publications that gave similar scores so the user finds out who/where (s)he can read reviews that would match his/her own taste ?

 

[Shifty Geezer]

Instead of allowing the user to bias press/critic score, wouldn't it be more interesting to have people give games a score and have the website find critics/publications that gave similar scores so the user finds out who/where (s)he can read reviews that would match his/her own taste ?

Great minds think alike.

However, I think Arwin's trying to get a different dataset, something like an auto-weighted average. Like it or not, companies are using Metacritic scores as a measure of accomplishment. If the community they sell to filters the results in favour of the communities' preferences, we should get a more representative score. At least, that ought to be by-product of Arwin's approach even if not the intention. If the intention is to help users choose games with a simple statistical test, I think the 'like minds' method would be more robust.

 

[Deleted member 13524]

I don't see the point, what's interesting in metacritic isn't the absolute press score but the range it's in and the difference with the user score...
Instead of allowing the user to bias press/critic score, wouldn't it be more interesting to have people give games a score and have the website find critics/publications that gave similar scores so the user finds out who/where (s)he can read reviews that would match his/her own taste ?

User scores become rather meaningless once each user is able to give 100 anonymous reviews to a game without ever buying or playing said game, which is what happens in Metacritic (see: any PC game from EA).
Only Steam, Origin and Uplay are capable of aggregating user reviews with a certain validation, thanks to their DRM.

You can, however, score a review. There are reviews where gameplay, graphics, sound, story, replayability, etc. are mentioned and evaluated. I would rate such reviews with a good score.
Likewise, there are reviews where the game itself is hardly mentioned and the reviewer limits himself to judge the content from a social/political perspective. I would rate that review really low since I don't read reviews to be told what I should/shouldn't like from someone's social/political point of view.

But maybe most people enjoy being told what they should/shouldn't like based on a social/political PoV, so even if my preferences don't align with most consumers, the reviewer/publication scores would still be more representative of their own "weight" in a meta-score.

 

[Shifty Geezer]

You can, however, score a review. There are reviews where gameplay, graphics, sound, story, replayability, etc. are mentioned and evaluated. I would rate such reviews with a good score.

How many users will review a well-written review a low score just because they disagree with its message? Going by up/downvotes on comment threads associated with articles, reviews of reviews will be every bit as biased as every other piece of social opinion.

But maybe most people enjoy being told what they should/shouldn't like based on a social/political PoV, so even if my preferences don't align with most consumers, the reviewer/publication scores would still be more representative of their own "weight" in a meta-score.

Which is where a 'like minds' approach is preferable, because it moves you into a more focussed demographic rather than plopping you in with the mass consensus. You get a tailored fit to your opinions.

eg. A user like yourself values in-depth game dissections. You'd uprate reviewers who provide that sort of content. In future games, you get only opinions founded on such critiques, be they positive or negative.

Another user looks on AAA blockbusters with suspicion. They only rate reviewers who give the mainstream games lower scores and uprate quirky indie titles.

When you both look for new games and sort by review score, you'd get a filter of all games based on sensible critiques regardless of budget, and Other User would get a list of quirky indie titles that also includes AAA blockbusters when their favourite reviews found merit in them. So for Other User, where they might ignore COD16 ordinarily, the fact that favourite sources 'Women's Weekly' and 'Juegos de Juego' and YouTuber 'SplatchHeadPlonk17' rated COD16 9/10 means they could trust it'd actually appeal to them.

It's a system that provides subjective context to the numbers, and enriches the data with human-relevant meaning via association. It's the true concept of a meta-critic, IMO - a gestalt persona offering advice based on human subjective opinions instead of unrelated objective criteria. I suppose the closest equivalent at the moment is the 'people who liked this also liked...' on various sites.

 

[Rodéric]

...I suppose the closest equivalent at the moment is the 'people who liked this also liked...' on various sites.

Precisely, it's also similar to Steam curators, figures that people listen to to make a choice based on their tastes (and critic ?).

 

[Shifty Geezer]

It's better than curators though because it's not limited to one person. And it'd be automated, instead of having to individually review curators.

 

[Arwin]

I agree. The basics for www.techingames.net to support an experiment are in place - you can add reviews, and the review average is shown. Now it's a matter of adding a lot of reviews to a few good test games and then adding some nice filters. The question is of course how to best do that. Perhaps it would be best to start with adding your own reviews / scores for a few typical games, and then start matching those against existing review scores to see who you align with. But as we have game features in there, we could also try to predict what new games you may like that share features. Perhaps by also adding an option to rate features? And that could also help predict if a game becomes more valuable after certain features have been added in a new version, or the impact of certain issues that a game is experiencing to at least temporarily suggesting to steer clear from a game. For instance, if you're all about online, then MCC and DriveClub were pretty much no-go areas for a long while, but if that's a minor thing to you, then they were great titles for you at launch.