Exception handling and console lock ups

[speedsix]

When a console hard locks, i.e game freezes and requires a reboot, what is actually happening/happened? I am guessing an exception has occurred in the code (let's say not because of hardware malfunction) that is either caught and nothing can be done about it(maybe after several repeat attempts at doing what it's trying to do) or a completely unhandled exception.

Obviously with a console you're unlikely to throw up an error dialog to the user like most software. The only case I can think of this happening is for 'dirty disc' errors. I assume in these scenarios (I don't know how the Xbox apis work) you would request to read in an asset from the disc and after the nth failure the apis itself would generate the 'disc cannot be read' message to the user.

 

[AlphaWolf]

I don't believe either system would (normally) lock hard on a disc read error.

 

[corysama]

Although the consoles are capable of exception handling, the console manufacturers strongly advise against enabling it. The in-order PowerPCs take a large speed hit from it. I don't recall the specific details on why. But basically, pretty much any exception that occurs will be unhandled. Thus the crash.

The simple truth is that modern console games are incredibly complicated pieces of software made under extremely tight time and manpower constraints. A single mistake in any of millions of interacting rules can bring the whole system down instantly. Even though most games go through thousands of hours of testing, there frequently still are subtle errors hiding in shipped games.

Frankly, after working deep in their guts for so long, I'm amazed every day that computers work at all.

 

[makattack]

Couldn't a hard lock include a resource deadlock that somehow affects the OS? Memory, disk, etc? I've locked up a computer well and good within a protected sandbox (Virtual Machine) with a program that just consumed all resources (CPU and memory) which made things run so slowly, I couldn't kill the process and all it's run-away threads.

 

[speedsix]

So how does a crash affect a multi-threaded application, would it halt all threads? I know on the 360 a proper hard lock will mean you can't access the system blade.

Saying that, a lot of these reports of GTA crashes say the sound continues to play as normal but everything else is completely locked?

 

[-tkf-]

So how does a crash affect a multi-threaded application, would it halt all threads? I know on the 360 a proper hard lock will mean you can't access the system blade.

Saying that, a lot of these reports of GTA crashes say the sound continues to play as normal but everything else is completely locked?

Wild guess would be something like the game actually crashes a "system driver" that just happens to be flawed depending on the PS3 model? afaik we had a FCC document where we could see something changed on the NIC side at least on the wireless side.

At least you can work around the bug, but how this came through 9 months of extra work is kind a strange. Something tells me that Rockstar was nowhere near to being ready with the game when they decided to delay it.

 

[DeanA]

Although the consoles are capable of exception handling, the console manufacturers strongly advise against enabling it.

I'm not sure this is right.. for debugging purposes, you can enable exception handling on consoles (PS3, for example), and there's no performance penalty at all. Certainly nothing in-order-specific about it either. It's just a very handy thing to be able to do while you are developing a title. Of course, you don't want custom exception handlers kicking off in retail titles, as you may as well have crashes be consistent to the end user - ie: a black screen and a reboot.

The one exception (pun not intended) to this is the denormal exception on Altivec/VMX which rounds denormal values. It slows down Altivec stuff quite significantly. But then that exception is disabled on PS3 by the OS.

Cheers,
Dean

 

[B1663r]

The only times Ive encountered ~any~ problems with the 360 is right before you call 1-800-4my-xbox and talk to max.

 

[ERP]

I'm not sure this is right.. for debugging purposes, you can enable exception handling on consoles (PS3, for example), and there's no performance penalty at all. Certainly nothing in-order-specific about it either. It's just a very handy thing to be able to do while you are developing a title. Of course, you don't want custom exception handlers kicking off in retail titles, as you may as well have crashes be consistent to the end user - ie: a black screen and a reboot.

The one exception (pun not intended) to this is the denormal exception on Altivec/VMX which rounds denormal values. It slows down Altivec stuff quite significantly. But then that exception is disabled on PS3 by the OS.

Cheers,
Dean

The original poster is confusing C++ exception handling (where there is a performance hit because of the way the code is generated) and machine exception handling, wherer there isn't, unless your throwing a lot of exceptions.

Generally most developers have an exception handler that does a variation of a BSOD during development, with info to try and resolve the original cause. Most exceptions are not recoverable at run time, so often devs will simply have shipping handlers reboot the machine. Things like null pointer dereferences, divide by 0 etc will throw exceptions.

Hard locks usually ammount to infinite loops, deadlocks (infinite loops accross multiple processes), or GPU hangs (usually a result of passing bad data).

Disk errors aren't usually terminal, and most of the TRC's just insist you retry.

Games are hundreds of thousands or millions of lines of code today, they're complicated, and they will have bugs. Where I'll work we'll have perhaps a thousand hours of test on something close to final when we ship, if the game sells even vaguely well, users will put 10x that on it in the first hour.

 

[corysama]

The original poster is confusing C++ exception handling (where there is a performance hit because of the way the code is generated) and machine exception handling, wherer there isn't, unless your throwing a lot of exceptions.

You are correct. That's what I get for talking about something I never use. I'm barely familiar with C++ exception handling, but I'm not clear on the capabilities of machine exception handling. Is it just invoking your callback function on an interrupt or is there more to it? Are there any cases where you can use a handler to continue from a machine exception or in practice is it limited to error reporting?

 

[ERP]

You are correct. That's what I get for talking about something I never use. I'm barely familiar with C++ exception handling, but I'm not clear on the capabilities of machine exception handling. Is it just invoking your callback function on an interrupt or is there more to it? Are there any cases where you can use a handler to continue from a machine exception or in practice is it limited to error reporting?

Fundamentally it's just an interupt, sometimes with extra crap on the stack.
You can safely return from one, and you might from some subset.

Back in the good old hacking days, I've used them for wrapping buffers (position buffer at end of memory, and use the Illegal Address exception to do the "wrap), fixing outlying divide by zero errors. Because paying the cost of the exception on the rare occasions they occured was cheaper than doing the test every time in the code.

But those types of hacks aren't worth it any more, exceptions are expensive to process and you get to run a lot of instructions in a frame.

In most cases when an exception is thrown, it's unexpected and you likely have bad program state. The question isn't what do I do when I try and derefference a NULL pointer, it's how did I end up with a NULL pointer, so it's not practical to continue with any expectation of stability.

 

[Akumajou]

I don't believe either system would (normally) lock hard on a disc read error.

Well I do believe that my old XBox 1 and many others that I knew would randomly give dirty disc errors on brand new flawless games.

I have heard and seen on youtube that the X360 still suffers from this idiotic error.

 

[AlphaWolf]

Well I do believe that my old XBox 1 and many others that I knew would randomly give dirty disc errors on brand new flawless games.

I have heard and seen on youtube that the X360 still suffers from this idiotic error.

The problem is hardly limited to the xbox's. This thread is about the way they are handled by the consoles. Normally the 360 will pop up an unable to read disc message and return you to the dashboard, not lock up or force reboot.

 

[speedsix]

Well I do believe that my old XBox 1 and many others that I knew would randomly give dirty disc errors on brand new flawless games.

I have heard and seen on youtube that the X360 still suffers from this idiotic error.

Yeah I'm fairly convinced the original Xbox would throw up the dirty disc screen for errors other than an actual disc read error.