Panajev2001a: Thats some unreadable patent-gibberish..
That particular patent at least seemed clear to me... you try to access a memory area from the I/O device (in this case the GPU) and a Hardware unit on the CELL BE checks if it is accessing a valid page.
When you start Linux, which sits above the Hypervisor, the Hypervisor sets access rules for that HW unit and quite likely marks XDR areas with data you should not have access from either the PPU/SPU side or the GPU side as not accessible by those physical devices. You try to read that area of memory, the DMA access is denied and only the Hypervisor can decide which memory area you can access and which one you cannot.
RSX, with a Linux driver giving you access to it, would function normally, it just could not DMA to/from an area the Hypervisor has protected.
RSX cant easily switch states, you have the Sony-Context and you have the Linux-Context (for a lack of better terms).
Now under Sony-Context you have shaders running that can access Region A in XDR Ram and Region B in DDR-Ram. Under Linux-Context you shouldnt be allowed to access those regions.
[...]
What if the content is already cached - are the security-measures implemented at that low level?
Unless RSX can safely dump its state (like DX10 are just beginning to do), I cant think of a secure (and somewhat predictible) way to switch Context.
Switching context? Maybe I am being dense here...
From the way you are explaining those contexts it seems like one is the "environment" set-up for the Game OS and the other one for Linux, but
the two OS do not run side-by-side/concurrently and Linux does not even run really directly close to the metal but on top of the Hypervisor.
You want to run Linux? Game OS says "ok, have it your way...", it starts the reboot process and the boot flag is changed so the boot-loader can start the "Other OS"...
This is not like executing Linux inside a VM running as a fully privileged application on the Game OS.