Well - here's how I understand it after reading through the docs (and spufs sources):
When an SPU enters isolation mode, it fetches the encrypted / signed (with the hardware root key) binary (normally a loader) from memory (via DMA) into its Local Store, decrypts it there and runs it. When in isolated mode, only a very small window of its Local Store is accessible through DMA by the other components (other SPUs, PPU, RSX, etc) while outbound DMA requests aren't restricted.
The rest of the Local Store is protected and can only be accessed by the code running on the isolated SPU itself. So if the programmer of the isolated SPU code isn't a complete idiot and does perform proper sanity checks on the (probably malicious) data that enters the SPU through that window there's very little chance to mess with it.
The IBM isolation loader for example expects you to write the memory address where the encrypted binary (to be decrypted by the loader code already running) resides into the accessible Local Store window. After doing so the SPU fetches the encrypted binary via DMA, decrypts it into the inaccessible regions of its Local Store and runs it. It seems plausible that METLDR uses a similar mechanism.
I'd be interested in how you think this could be circumvented without the keys necessary to create own signed / encrypted binaries.
Originally Posted by SCE
As far as I see it, it's nearly impossible to tell the Linux kernel "not to use the first xx MB of RAM" since the Linux kernel uses LV1 calls to allocate memory. What looks like one big chunk of contiguous memory to the kernel might consist of many small parts (LPARs) spread over the entire physical memory range.
So in order to have such tight control over memory allocation I guess one would have to patch the hypervisor.