The hugely scientific console reliability poll

[patsu]

The main point is that the sample size, and the poll data at hand is not big enough (nor a good enough representative of the average console) to give you any numbers of statistical significance, and is therefore irrelevant to be used as an indicative, or as data for any real world analysis.

Not really. The data here (if collected more fully, including date of purchase) are valid for any statistical analysis of 360's reliability.

Because the data at hand is not a representative of the consumer population, the question about consumer distribution and manufacturing characteristics is irrelevant, and something which cannot be answered tbh.

My question was: How does B3D's composition affect the failure rate of their 360s ? If it's because B3D posters buy more launch consoles, it just means that we have better/more data for launch consoles.

The result does not necessarily have to depend on whether the buyers are "representative of the consumer population"

 

[macabre]

Really? My ps1 still works

Launch units had a problem with the power supply being too close to the CD laser mechanic,, resulting in disc read errors.
Mine still works though when I turn it upside down.

Disc read errors are a big issue on the early 360s as well. Out of 4 people I know who have a launch unit 2 are having this. 1 of them is unusable for games now, I`d call that one dead.

 

[deepbrown]

Launch units had a problem with the power supply being too close to the CD laser mechanic,, resulting in disc read errors.
Mine still works though when I turn it upside down.

Disc read errors are a big issue on the early 360s as well. Out of 4 people I know who have a launch unit 2 are having this. 1 of them is unusable for games now, I`d call that one dead.

Oh i see. Now I have had the disc reading error on my ps2 for some new games (GOW) and DVD's...but I just use a laser disc cleaner and it works straight after...I just thought it was age (but I suppose it happened to people when the console was young)

 

[Cheezdoodles]

Not really. The data here (if collected more fully, including date of purchase) are valid for any statistical analysis of 360's reliability.

No its not. Sample size is to small, and the sample is not a representative of the average X360 console, the method of acquiring the data (website polling) is also not reliable, all in all, its not worth anything as far as statistical analysis goes.

Even if you collect data more specifically, it would still be invalid for any real statistical use, why? Because people with working consoles are less likely to seek out threads about reliability and less likely to vote. People by nature, like to complain about bad things rather than compliment good things. And again, the sample data being so small, leads to big chance of the data being inaccurate.

My question was: How does B3D's composition affect the failure rate of their 360s ? If it's because B3D posters buy more launch consoles, it just means that we have better/more data for launch consoles.

And i already answered this about 10 times now. Since there is a higher consistency of early adapters, we would get an inflated failure rate. Therefore the data you would collect, would be flawed in terms of any general failure rate conclusions.

The result does not necessarily have to depend on whether the buyers are "representative of the consumer population"

No, but it has to be a representative of the consoles, and it has to be in a big enough sample. Even if we would do as you suggest, and collect the data as date of purchase from the voters, its not good for any real use, because people have to actively seek out this poll in order to vote. And no matter what argument you make, the sample size will still be far to small to even give you an indication of anything, statistically speaking.

437 dead consoles, 14 on life support. 359 individual posters.

I thought il add to this, since you figure that this "official x360 death" topic is such a great piece of source data RolfN, here a little math based on it. Since we do not know the total amount of people with X360's on Neogaf, therefore we have no ways of knowing the actual reliability rate of that population, the only meaningful statistic we could get for reliability would be to look at the reliability of the replacements.

Apparently, 359 people have had 437 dead consoles. Assuming that these broken consoles have been replaced by the warranty, this would mean out of 359 replacement consoles, only 78 consoles died, the rest is working.

76\359 = 21,71% failure rate. So now RolfN, in regards to this:

2. There is not a single sample population that shows failure rates under 30%.

we now have a sample population that do show less than 30% failure rate.

Of course, this data is just as flawed, as all the other data we have been discussing, (yet less flawed than your example that just showed "total" failed consoles on neogaf)

 

[PatrickL]

Well beyond3d population is a particular subset of the population. That sample bought consoles with a ratio of 2 PS3 for one Wii. As we know that worldwide the sales are rather the opposite we can say that B3D population is not representative of the average population.
Does that have an effect on the the reliabity results? In a perfect world, it should not. But do we live in a perfect world ?

 

[Tap In]

let's not forget that it's human nature to be motivated to vote (or make your voice heard) when you HAVE had a problem compared to if you have NOT had a problem.

I would guess that a higher percentage of 360 owners who has had a problem has voted than every 360 owner with a perfectly fine running machine. that's just human nature.

 

[Todd33]

I think the results are beyond spinning. Compared to the other thread and the PS2, which was well known to be unreliable and has been out far longer, the 360 is looking really bad. This re-enforces what people read all over the net and the fact they extended the warranty. So unless there is a huge conspiracy I think MS shipped some shoddy hardware.

So buy extended warranties

At this point the PS3 and Wii have a good reputation for reliability, which was not true for the 360 at the same point and time in 2006.

 

[Kryton]

let's not forget that it's human nature to be motivated to vote (or make your voice heard) when you HAVE had a problem compared to if you have NOT had a problem.

I would guess that a higher percentage of 360 owners who has had a problem has voted than every 360 owner with a perfectly fine running machine. that's just human nature.

It's not human nature at all. It's human nature to moan, but this is a neutral "Did it fail or did it work?" question.

 

[Todd33]

It's not human nature at all. It's human nature to moan, but this is a neutral "Did it fail or did it work?" question.

We are also looking at relative failures, if it were human nature to only report failures in polls than all consoles would have similar ratios.

 

[Jaeyden]

"The hugely scientific console reliability poll"

Correct me if I'm wrong but.....doesn't the title of this thread actually imply that this poll is NOT meant to be torn to shreds by the analytical mind? It's a simple survey. I didn't think that Shifty was trying to accomplish statistical nirvana to begin with. I take the results with a grain of salt. Xboxes break down more than the other consoles, it doesn't take a computer/rocket scientist to figure that one out. BTW - I couldn't vote in this poll.

 

[Tap In]

It's not human nature at all. It's human nature to moan, but this is a neutral "Did it fail or did it work?" question.

I'm not faulting the poll (Shifty is a smart guy), just the results.

I know for a fact that not everyone who has a working machine voted. :smile: Not to mention that there is no way to verify if all of the voters actually own a machine or if some would try to skew the poll.

yes, you are correct it is human nature to moan when things go wrong. *Googles human behavioral studies*.

 

[dobwal]

There is a less than scientific way to conduct a poll and there is a reason why (sampling) this poll shouldn't be extrapolated to denote the failure rate outside of the experience of the respondents.

"The major distinguishing difference between scientific and unscientific polls is who picks the respondents for the survey. In a scientific poll, the pollster identifies and seeks out the people to be interviewed. In an unscientific poll, the respondents usually "volunteer" their opinions, selecting themselves for the poll.

The results of the well-conducted scientific poll can provide a reliable guide to the opinions of many people in addition to those interviewed -- even the opinions of all Americans. The results of an unscientific poll tell you nothing more than simply what those respondents say."

http://www.publicagenda.org/polling/polling_20q.cfm

"When conducting a survey, how a researcher selects participants is just as important as how many participate...Representative samples must be selected carefully and without bias. For example, samples made up of self-selected responders, such as people who participate in a survey or poll by calling an 800 number, are almost certainly biased samples.

In a scientific survey, researchers choose samples through some random process that is usually mentioned in the survey background materials. In statistical terms, a random event is one that occurs with a certain, measurable chance or probability of happening. For example, under the simplest circumstances, where each member of a population has one chance of being sampled, the probability of getting selected for a survey can be calculated just by knowing a population size and desired sample size.

Nonrandom samples include those that select members of the population based on their proximity, availability or through referrals by friends. There are sometimes scientific reasons to conduct non-random surveys, but they are often unscientific and should not be used to generalize statistically to larger populations.

http://nsf.gov/news/special_reports/survey/index.jsp?id=trust

"In a bona fide survey, the sample is not selected haphazardly or only from persons who volunteer to participate. It is scientifically chosen so that each person in the population will have a measurable chance of selection. This way, the results can be reliably projected from the sample to the larger population."

http://www.amstat.org/sections/srms/pamphlet.pdf

 

[patsu]

No its not. Sample size is to small, and the sample is not a representative of the average X360 console, the method of acquiring the data (website polling) is also not reliable, all in all, its not worth anything as far as statistical analysis goes.

These are "real" data. You can combine them with another non-overlapping sample.

Even if you collect data more specifically, it would still be invalid for any real statistical use, why? Because people with working consoles are less likely to seek out threads about reliability and less likely to vote. People by nature, like to complain about bad things rather than compliment good things. And again, the sample data being so small, leads to big chance of the data being inaccurate.

Do they have to seek out this thread to post ? This thread just fall from the sky when I woke up one day and I participated. We already know there are no/few new posters in the poll (Check the nicknames of the people polled). So the data does not contain people who voluntarily sign up to vote.

This is not an opinion/preference poll. So I don't know if asking the question to a 360 12-year-old or 70-year-old makes a difference. If their 360 died, it died.

If you run an offline survey, people with working console may also refuse to participate (for whatever reasons), or lie. Over here, you can see their gamer tags.

And i already answered this about 10 times now. Since there is a higher consistency of early adapters, we would get an inflated failure rate. Therefore the data you would collect, would be flawed in terms of any general failure rate conclusions.

What we have collected would be representative (but not necessarily precise) of an early Xbox 360 owner's experience. They are not "inflated" per se. As long as the context is clear, I don't see a problem with that.

 

[Bigus Dickus]

Ok, so you're saying:
(1) B3D folks are more likely to complain if their consoles break and keep quiet if their consoles work ?

Yep, that's human nature. I'm also saying, quite importantly, that the people on B3D may not remotely represent the average consumer 360 population in general.

(2) If so, doesn't the same behavior apply to PS3 and Wii as well ?

Yep, with the modification of inertia, which I have explained before. This has been seen to occur in the past, where certain ideas gain weight and momentum, and thus become much more visible and discussed.

 

[Bigus Dickus]

1. There's no correlation between frequenting forums and having your stuff fail on you, no matter how hard the hand-waving gets.

I wouldn't be surprised if that turned out to be a false claim. I have no proof, but intuition leads one to believe that the B3D (and in general forum frequenting population) might be more hard-core gamers, and thus put more hours on their machines. That is a direct tie-in to MTBF. I would not expect this to be the case for everyone, of course, but if I were a wagering man I'd put money on a weak but statistically sound correlation. Hardly the most important point here, though.

2. There is not a single sample population that shows failure rates under 30%. Nowehere. None. Never happened. So it's not even cherry-picking the wrong forums and gaming the polls. There is no known species on this planet that has experienced acceptable XBox 360 failure rates.

First, I'd disagree about the no sample population under 30%... MS certainly knows their failure rate, and I can guarantee you it isn't 30% or more. I'm not a wagering man, but even I would put money on that one and laugh all the way to the bank. Now, acceptable failure rate? Very subjective definition, but I would agree that it is higher than normal, not something MS is/was happy about, is certainly something they have expended resources towards trying to correct, and is higher than is likely the case for PS3/Wii (and many other consoles of past generations).

3. Look at the NeoGAF tally (full list of names, tracks multiple failures, tracks near-deaths).And it's hard to manipulate polls on NeoGAF. The registration process takes months to go through and trolling juniors are usually banned without warning.

Even more meaningless than the B3D "poll" since it has no way to account for working 360's.

Key questions:

What is your sample size?
How did you arrive at your sample population?
What is its demographic, and how does that relate to the total population demographic?
What question are you asking?
How are you gathering data?

You're going to pretty much fail every single one of those key questions with every single forum "poll" or survey or whatever you link to. They aren't all equally flawed... they are all horrible, but some are more horribly flawed than others.

And the response?
*LaalaalaaaIcan'thearyouuuuu*

Nope. The response is pretty consistent from the crowd with a head on their shoulders (read all of the posts in this thread and you will see this clearly). It goes pretty much like this: "360's have higher than expected/normal failure rates, more likely worse on the earlier production units, and likely lingering still... MS has definitely noticed, taken steps to correct this both in terms of warranties and hardware changes, and it has cost them money and mindshare. However, the failure rate is nowhere near 30%. 5% seems reasonable, if huge, and 10% would be the upper bound of all rationality."

How do you read that as "I don't hear you?"

 

[Cheezdoodles]

These are "real" data. You can combine them with another non-overlapping sample.

ROFL. Seriously, read up on statistics, and samples.

Of course you cannot do this, specially not when the data isn't properly collected in the first place. If your going to MEASURE anything, regardless of what it is, and what to get scientific about it (as in being able to actually claim things as facts) the data needs to be collected in a proper fashion. You need to have data representing averages. You cannot start to begin combining samples, unless all the samples have all been extracted in the same proper mathematical way, and your 100% the data isn't overlapping, (the data would also need to be taken from different demographic groups, or, both samples need to be representative of the consumer average, or in this case, console average).

Do they have to seek out this thread to post ? This thread just fall from the sky when I woke up one day and I participated. We already know there are no/few new posters in the poll (Check the nicknames of the people polled). So the data does not contain people who voluntarily sign up to vote.

Of course you have to seek out this poll. I don't visit every thread on b3d, i visit the topics that have a title that is interesting to me. I don't have any interest in reading about a Gamecube VGA cable for example, or PS1 downloadable games, so i stay way from those topics.

Your also misunderstanding whats being said. The biggest difference is that people VOLUNTARILY vote. They seek out this thread, and they vote. This alone makes the data you collect tainted.

This is not an opinion/preference poll. So I don't know if asking the question to a 360 12-year-old or 70-year-old makes a difference. If their 360 died, it died.

Your taking things far to literary, if you go and ask launch owners only, you would get a completely different result, than if you say, go ask people who recently bought the console and aren't using it for anything else than playing live arcade.

If you run an offline survey, people with working console may also refuse to participate (for whatever reasons), or lie.

Wow. Your just arguing for the sake of arguing arent you? Of course they can, of course they can lie. However, the chance of somebody lying in a survey where the poller "randomly" selects people in order to get a proper average as explained in dobwalls post, is much less likely than when somebody volunteers to post.

What we have collected would be representative (but not necessarily precise) of an early Xbox 360 owner's experience. They are not "inflated" per se. As long as the context is clear, I don't see a problem with that.

Did you even bother to read the explanations that dobwal linked to?

Wait this is a rhetorical question.

 

[NovemberMike]

...The people who know anything about statistics aren't even arguing a certain point in the console wars thing. We are simply saying that this poll is 100% useless for anything other than having a good laugh.

If you run an offline survey, people with working console may also refuse to participate (for whatever reasons), or lie. Over here, you can see their gamer tags.

Btw, having people not respond is call "Nonresponse Bias" and if it happens too much you have to throw the results of the survey out.

 

[patsu]

ROFL. Seriously, read up on statistics, and samples.

Of course you cannot do this, specially not when the data isn't properly collected in the first place. If your going to MEASURE anything, regardless of what it is, and what to get scientific about it (as in being able to actually claim things as facts) the data needs to be collected in a proper fashion. You need to have data representing averages. You cannot start to begin combining samples, unless all the samples have all been extracted in the same proper mathematical way, and your 100% the data isn't overlapping, (the data would also need to be taken from different demographic groups, or, both samples need to be representative of the consumer average, or in this case, console average).

This is not a sales/opinion/preference poll, so the answer is pretty clear cut (i.e., no ambiguity in interpreting whether an Xbox died or not).

When it comes to Xbox 360 population (NOT consumer population !) selection, I already acknowledged that date of purchase is missing. Bigus Dickus suggested hours of use as another possible variable, which I agree (since I highlighted MTBF and PS3 Folding as an example). What else do you think will affect the poll result ? These additional data points can help to explain the data but they won't affect the outcome (of whether the box has indeed failed).

Filtering out overlapping data is not as difficult as you thought in this case... via gamer tags, if someone really wants to tally these up. This is one of the rare cases where I think online polling has some advantages.

Of course you have to seek out this poll. I don't visit every thread on b3d, i visit the topics that have a title that is interesting to me. I don't have any interest in reading about a Gamecube VGA cable for example, or PS1 downloadable games, so i stay way from those topics.

Your also misunderstanding whats being said. The biggest difference is that people VOLUNTARILY vote. They seek out this thread, and they vote. This alone makes the data you collect tainted.

True, you have to click into the thread to see what's going on.... but isn't this the same as offline polling where people stop/call you and ask to participate in some survey about 360 failure ? I have rejected/particiapted in a few. You don't actively seek out the poll to propagate personal agenda (e.g., register as a new member to participate, pull data from customer support to review called in cases).

The short falls are:
* B3D community is not big
* Insufficient info collected to normalize/explain the dataset
* Entries should be per-box instead of per-user. Some users may exchange their 360s with new ones via store warranty, or have gone through multiple failed Xboxes.

Your taking things far to literary, if you go and ask launch owners only, you would get a completely different result, than if you say, go ask people who recently bought the console and aren't using it for anything else than playing live arcade.

Not really... I'm seeking the context since those supporting data were not collected. I have always highlighted that the numbers are not precise and we cannot extrapolate any trend from 1 poll like this (yet).

As for taking things far too literary, don't you think even real world polling has its many flaws too ? The trick lies in building in mechanisms to detect and reason about the data...

Wow. Your just arguing for the sake of arguing arent you? Of course they can, of course they can lie. However, the chance of somebody lying in a survey where the poller "randomly" selects people in order to get a proper average as explained in dobwalls post, is much less likely than when somebody volunteers to post.

See Bigus Dickus' reference to hot topics (e.g., Xbox 360 failures). It is not uncommon for people surveyed to coerce with the expected outcome just to prove that they are smart... The same behaviour will work in offline or online poll, but I have digressed. In this case, in the online world, we at least have gamer tags to follow up.

Don't always write off online polls. All polling mechanisms have their flaws. If we use them correctly, we should be able to get what we want. But perhaps B3D is not the platform to explore this.

Did you even bother to read the explanations that dobwal linked to?

Wait this is a rhetorical question.

Do you understand what I'm saying ?

 

[Onlooker1]

Wow, so the poll is "useless" but not so useless that 2 or 3 characters don't feel the need to devote hours of complex multiple response typing, in an effort to refute the (obvious to anyone) conclusion that xbox360 is very unreliable vs its competition.

I'd not be surprised if microsoft pays PR agencies to do forum damage control when topics like this threaten to blow up from a smoke to fire. If they don't, then some people sure behave like MS pays them for anxious full-time refutation duty ..

 

[Cheezdoodles]

Wow, so the poll is "useless" but not so useless that 2 or 3 characters don't feel the need to devote hours of complex multiple response typing, in an effort to refute the (obvious to anyone) conclusion that xbox360 is very unreliable vs its competition.

Nobody is refuting the fact that the X360 has a high unreliability rate vs its competitors (i think everybody here has agreed to that), we pointed out that the source data is in no way good to give you any indicative of anything, except for the reliability rate of the consoles that are owned by the questioned population sample.

Then people have tried to by varius amounts of broken logic try to prove that wrong.