Critical PS3 internal clock error and its aftermath

[SG79]

Yeah, exactly 24 hours after it happened, the units went back to normal.

*snip*

Exaggerate much? There were people talking about massive hardware recalls and numerous hardware issues out the wazzoo. According to Pachter (talking to MSNBC), this issue would've lingered on for a week before it's fixed and some sites claimed that slim units and save data was corrupted. Compare that with a "it's a clock problem and we hope to get it fixed within 24 hours". Which sounds better?

It wasn't a pleasant experience but given how short it lasted and the fact that it wasn't permanent , it's not a PR nightmare. For your average Joe, Sony did fix the problem but we know what happened. The issue as simple it was is solved, and they need a firmware update to remedy a similar incident 4 years from now.

 

[Cornsnake]

Yeah, exactly 24 hours after it happened, the units went back to normal.



Exaggerate much? There were people talking about massive hardware recalls and numerous hardware issues out the wazzoo. According to Pachter (talking to MSNBC), this issue would've lingered on for a week before it's fixed and some sites claimed that slim units and save data was corrupted. Compare that with a "it's a clock problem and we hope to get it fixed within 24 hours". Which sounds better?

It wasn't a pleasant experience but given how short it lasted and the fact that it wasn't permanent , it's not a PR nightmare. For your average Joe, Sony did fix the problem but we know what happened. The issue as simple it was is solved, and they need a firmware update to remedy a similar incident 4 years from now.

A few million PS3's broke themselves. I'd call that a PR nightmare. It doesn't matter that it could have been worse. Sony didn't fix anything, and for Joe average it just looks like the PS3 broke itself and start working again randomly.

Consoles have often been known for being unreliable. I don't think Sony wants that to happen to the PS3.

 

[patsu]

They said it was a PSN related issue when it clearly wasn't. And they said not to trust any source other then Sony.

And they were never able to solve the issue. Which is nice because now the PS3's name for reliability is forever down to the toilet. Because for anyone who doesn't follow this as close we do, it just seems their PS3's decided to break itself one day and start working again the next. Who knows what it's going to do tomorrow.

They better give us something nice if they want to get rid of this PR nightmare. Have I mentioned I wanted cross game chat?

Ha ha, I do think they should clear PSN's name since it is wrongfully blamed. I don't think PS3's reliability is affected because the problem only hit the Fat PS3s according to the official reports. Newer models are unaffected. I'm assuming a firmware patch later will address the root cause (and they should reassure us).

I am a little disappointed none of the big shots chimed in to talk to us directly (via Email or the Blog). Kaz Hirai's statement is for the press, and seems "for show" only. Yes, while clarifying previous PSN statement, they can also talk about PSN and what's to come in the near future. I wonder who's the new Sony Group online chief. It's starting on April 1st, so the head should be known already.

I am also interested in whether "behind the scene" informal update by the community managers can help to soften the impact (or make it worse. ).

 

[SG79]

A few million PS3's broke themselves. I'd call that a PR nightmare. It doesn't matter that it could have been worse. Sony didn't fix anything, and for Joe average it just looks like the PS3 broke itself and start working again randomly.

Consoles have often been known for being unreliable. I don't think Sony wants that to happen to the PS3.

They didn't really break since you're talking technicality, and Sony didn't do anything by letting the issue solve itself. As I remember, that was in fact MS's advice with the Zune. As I said earlier in the topic, it's kind of a delicate issue when the immediate fix required some tech savvy work (unlike a Zune for example).

Your average Joe probably checked an online source to see what's going on and goes by the official or unofficial line. I don't think there are owners out there sitting in tents without any net connections whatsoever

The only thing they can and SHOULD do is come clean with a proper press release (not a blog post) explaining what exactly happened. Then perhaps offer a few free games on PSN for a limited time.

I do agree that it's baffling that they acknowledged the clock issue and didn't revise the original claim of a PSN problem.

 

[deathindustrial]

This issue keeps getting blamed on the RTC but is not the RTC just a plain old counter with a stable clock? Any actual date calculations have got to be in Sony's firmware somewhere, right (same idea as POSIX time)? This has got to be an internal Sony problem and not a supplier one I would think.

 

[Cornsnake]

They didn't really break since you're talking technicality, and Sony didn't do anything by letting the issue solve itself. As I remember, that was in fact MS's advice with the Zune. As I said earlier in the topic, it's kind of a delicate issue when the immediate fix required some tech savvy work (unlike a Zune for example).

Your average Joe probably checked an online source to see what's going on and goes by the official or unofficial line. I don't think there are owners out there sitting in tents without any net connections whatsoever

The only thing they can and SHOULD do is come clean with a proper press release (not a blog post) explaining what exactly happened. Then perhaps offer a few free games on PSN for a limited time.

I do agree that it's baffling that they acknowledged the clock issue and didn't revise the original claim of a PSN problem.

I'm talking about what the average consumer would think when a gamesconsole doesn't play games, it's broken. I doubt they'll go online to see what's going on because as far as they know it's an isolated issue only effecting them. They'll probably tinker around with it trying to get it to work, give up, try later, hey it's working again, hope that never happens again.


Since Sony's response to the issue is so poor, have they actually officially announced that it's safe to turn on the PS3 again?

 

[patsu]

Since Sony's response to the issue is so poor, have they actually officially announced that it's safe to turn on the PS3 again?

Yes, They held a press conference. You see several sites reporting them. But I still prefer direct communication.

Their helpdesk will also know what to say now.

 

[Nesh]

I'm talking about what the average consumer would think when a gamesconsole doesn't play games, it's broken. I doubt they'll go online to see what's going on because as far as they know it's an isolated issue only effecting them. They'll probably tinker around with it trying to get it to work, give up, try later, hey it's working again, hope that never happens again.


Since Sony's response to the issue is so poor, have they actually officially announced that it's safe to turn on the PS3 again?

Eh? Broken?

Then kudos for Sony for making a console fix itself after it is broken because it now plays as if nothing happened and normally "broken" means it needs to be taken to a repair shop or Sony for repairs

 

[patsu]

Digital Foundry has more info:
http://www.joystiq.com/2010/03/02/apocalyps3-8-out-of-11-ps3-fat-skus-affected-ps3-shared-zune/

If this sounds suspiciously similar to the Z2K bug that suddenly bricked thousands (dozens?) of 30GB Zune players in 2008, it's because both the PS3 and Zune share an ARM CPU chip, says DF.

... and here's a Sony VP's comment: -_-
http://twitter.com/TheKevinButler/status/9887945951

 

[ShadowRunner]

From the response i have had from more casual people i know there is not much bad feeling at all, they are just happy there consoles are fine. I think the hysteria is/was mostly online, outside of forums the world has moved on

 

[patsu]

Yes, the best thing about the fix is "No user action required" ! It's the best fix possible (except for isolated data losses). I was relieved too when I heard that. That and the network works as if nothing has happened.

The "don't trust unofficial sources" and "don't update unless you're sure" are pretty standard responses. This time round, the forumites may have gotten it right. But there is no guarantee that it will always be the case (e.g., There could be multiple issues).

 

[PARANOiA]

I'm going to say the battery is charge when it's plug to the wall. So everytime you turn it off the battery stops. Kinda like a dead ps3 controller. They still work if it's always plug. You probably just need to set up the time at every start up. The guys that did pull the BIOS battery off the ps3 and back on had to set the OS clock again.

CMOS batteries don't work like that, though.

Games will work when the battery is empty, the clock will reset to a default value.The OS is prepared for that unlike the bug. You`ll probably loose your settings when you plug the system off so people will either
1. live with it
2. some will change the battery themselves
3. buy a PSThree for 99 bucks
4. already be on PS4

The problem here is the PS3 issue this week showed that the CMOS battery being out of sync with the system battery causes massive issues that aren't solved by simply changing the date. This included all downloaded content being unplayable, all online content failing, and all games with trophies (i.e. every game released between a few years ago and every game moving forward) being unplayable.

I don't see this as a small issue, and I hope they manage to patch whatever restrictions stop systems from fucking up entirely when the CMOS clock is incorrect - this is not a leap year issue, but rather an issue on very messed up DRM implementation.

 

[Strange]

Digital Foundry has more info:
http://www.joystiq.com/2010/03/02/apocalyps3-8-out-of-11-ps3-fat-skus-affected-ps3-shared-zune/



... and here's a Sony VP's comment: -_-
http://twitter.com/TheKevinButler/status/9887945951

I wonder if Freescale actually knew about this problem (which they should since December 2008) and told their other customers to check/prepare for this.

 

[Weaste]

This issue keeps getting blamed on the RTC but is not the RTC just a plain old counter with a stable clock? Any actual date calculations have got to be in Sony's firmware somewhere, right (same idea as POSIX time)? This has got to be an internal Sony problem and not a supplier one I would think.

No, an RTC keeps a time and date. You'll find one in a cheap CASIO watch.

 

[Weaste]

I wonder if Freescale actually knew about this problem (which they should since December 2008) and told their other customers to check/prepare for this.

This is the problem I am having, do we know where the RTC in the PS3 is actually located. It could be in the Cell for all we know.

 

[patsu]

I wonder if Freescale actually knew about this problem (which they should since December 2008) and told their other customers to check/prepare for this.

Yeah, that's why I mentioned they should revisit their process. I suspect the ball got dropped somewhere (e.g., someone got overwhelmed and forgot).

The problem here is the PS3 issue this week showed that the CMOS battery being out of sync with the system battery causes massive issues that aren't solved by simply changing the date. This included all downloaded content being unplayable, all online content failing, and all games with trophies (i.e. every game released between a few years ago and every game moving forward) being unplayable.

I don't see this as a small issue, and I hope they manage to patch whatever restrictions stop systems from fucking up entirely when the CMOS clock is incorrect - this is not a leap year issue, but rather an issue on very messed up DRM implementation.

I don't know whether the DRM implementation is messed up but the incident doesn't imply that conclusion. All (complex enough) security systems require some sort of timestamp. It's in the design/algorithm. Even a good implementation should fail in this case.

It is believable that the problem is in the clock.

 

[Grall]

... and here's a Sony VP's comment: -_-

Epic arrogance.

I'd fire him for showing such contempt to his own customers if it was within my powers. Maybe he thought he was being funny, except he wasn't.

Sony leadership should consider who exactly it is that pays their fat salaries...

 

[Richard]

I'm amused by the whole experience to be honest. I've not seen anything quite like this in consumer electronics before, though studying Computer Science you'd hear similar stories on a much larger scale financially (e.g. leaving out a semicolon in some simple bit of code means your satellite overshoots Mercury and you lose a few hundred million dollars).

This is a very widespread problem actually, but you only hear it for mass-consumer products. Testing software is for the weak; reinventing wheels is the stapple of the true programmer.

This issue keeps getting blamed on the RTC but is not the RTC just a plain old counter with a stable clock? Any actual date calculations have got to be in Sony's firmware somewhere, right (same idea as POSIX time)? This has got to be an internal Sony problem and not a supplier one I would think.

It's very likely this is a supplier problem actually. This type of thing (time) is too low-level and the PS3 too complex, for a company like Sony to bother with directly. Same problem with the Zune leap year bug actually.

Yes, the best thing about the fix is "No user action required" ! It's the best fix possible (except for isolated data losses). I was relieved too when I heard that. That and the network works as if nothing has happened.

Heh. I'm much more concerned (as an outside observer) about other bugs that might have been prevented. As often as leap-year bugs happen (despite being a completely understood problem) they're now at least acknowledged. Maybe there's a racing condition on the BR-JAVA or the cpu throttling logic, etc.

If they couldn't find the stuff that other people have tripped over (and continue to trip over) how confident are you that they'll find the problems no one else encountered yet?

 

[patsu]

Heh. I'm much more concerned (as an outside observer) about other bugs that might have been prevented. As often as leap-year bugs happen (despite being a completely understood problem) they're now at least acknowledged. Maybe there's a racing condition on the BR-JAVA or the cpu throttling logic, etc.

If they couldn't find the stuff that other people have tripped over (and continue to trip over) how confident are you that they'll find the problems no one else encountered yet?

Bah... leap year bug happens a day in 4 years. BD player is tested and used round the clock, all over the world. There has been cases where a movie refuses to play, but very visible problems like these are usually identified quickly after launch (like PS3's version of Assassin's Creed), or not caught by end users at all (because they can't be bothered -- no one uses that feature. ).

I do agree multi-threaded programs can get hairy though. And people get tripped over all sorts of overflow/underflow bugs over and over. But we still have great software from time to time.

 

[SG79]

Epic arrogance.

I'd fire him for showing such contempt to his own customers if it was within my powers. Maybe he thought he was being funny, except he wasn't.

Sony leadership should consider who exactly it is that pays their fat salaries...

I guess you're not in North America. That's just an actor who portrays a "VP" in commercials with humorous themes; look as his title; VP of first person shooter relations. They launched the campaign with him around the launch of the slim, and keep using him in commercials. I'm sure you'd find a few on youtube.