Peer review really only works properly when the peers are a) roughly equally knowledgeable, b) have a commonly held understanding of the reason for the existence of the peer review process and c) have no specific axe to grind.
Definitely. Ympoint is the tools isn't at fault, only how it's wielded. Where a reputation is like a hammer, it's down to those who have it whether they build something, destroy something, or hit each other over the head with it.
There was a massive range in rep levels amongst folks here. Some of the highest repped members being Ordinary Joes with no special connection to the graphics or gaming industry other than they spend a lot of time hanging around on forums...
That's very true, and as one of those Ordinary Joes I should know! But Reputation isn't just about technical know-how, but also what a person contributes to discussion. Someone who knows very little but asks useful questions that gets useful answers is a valued member of the community, and those sorts of people I've handed out rep to. Rep shouldn't be a measure of how clever you are or how much experience in the field you have, but how much you contribute to the community. By that same token, a dev who knows everything about a certain piece of hardware...let's say Ken Kutaragi joins and posts open info on PS3...but only posts 3 times a year, isn't contributing much to the forum. That to me is why the devs didn't have the fully-filled rep bars. Their insights are invaluable and make up the meat of the forum, but the number of posts often don't warrant reputation. Unless you go giving positive rep to short posts of no real merit, you won't get their reputation up. Rep is handed out per post.
Maybe the rep-per-post of the Knowledgeable Guy was higher than Ordinary Joe, but that wasn't what was displayed and it's not what this discussion is about. All-in-all rep was too easily handed out in large quantities by too many clueless people for it ever to become a serious measure of the average quality of any given members posts.
I'm sure it wasn't wielded properly in all cases. However, there is the question of how you measure whether rep was handed out cluelessly or not. If 10 people give Mr. X neg rep for a post that Mr. X thinks perfectly valid, does that mean those ten people were clueless, or Mr. X's idea of what's acceptible doesn't coincide with what others on the board consider acceptible? As a community one needs to fit in with the community mores, as poorly defined as they might be. Reputation was quite democratic with everyone (enabled to hand out rep) having their say. It was slanted in favour of the 'old boys' as they could sway a person's numerical rating, but if someone got more neg rep than positive, it's because that someone was irking the population into voting against them. And if that someone contributed positively, they were voted up in rep.
Without seeing all the reputations given out to and by everyone, I can't form my own idea of how poorly it was wielded. On the whole though, the prinicple of the system was fair. If you didn't say or do anything to offend people, you wouldn't get - rep. Maybe what people took offence to wasn't what we'd want of B3D, but that's their community if so. eg. If every time a poster posts something negative about Brand Q they got a load of negative reputation, that would show that the voting community want a community where Brand Q is never criticised. Not very smart, but if that's what the voting public want...
Perhaps the major issue was concentration of votes? If everyone
had to pass judgement on every post, +ve, -ve or neutral, you wouldn't get such an imbalance? That way where perhaps 10 people would vocally -rep a person for criticising Brand Q, there'd be the vast majority of 5000 votes who voted it neutral and that would show the community
as a whole are okay with criticism.
Still, if B3D can find the perfect solution, it will have amazing consequences for the entire world. The issue here is exactly the same of politics and social structure. Do you have ruling classes? An open democracy where all have equal say? A single monarch? No-one's managed to find the perfect solution yet, and it'll be magic if someone does!
