Older versions, no you couldn't run unsigned code. As for OtherOS, it isnt a sandbox per-se, it is running on the cell hardware, with access through the Hypervisor, it is just under a different access policy than the PS3 OS, so it has less power over what it can do.
- Yeah, it's the hypervisor the main issue... But I thought the hypervisor was a layer on top of OtherOS, preventing access to the ps3 graphics hardware.. And I was thinking that the Hypervisor was running only on top of OtherOS, so I was guessing the PS3's FW OS was not behind the hypervisor, which would ease breaking in through an exploit.. although I might be wrong, maybe the hypervisor is always running and the FW OS simply asks the Hypervisor to give it access to the hardware providing some key/authentication, in which case, we're back to square one. You're in a better position to know how this all works in reality.
And remember, the PS3 was also designed to prevent against any type of overflow, thats one of the points of having a Hypervisor, to look over everything, and make sure nobody tries to break in.
humm.. if there's buffer overflow protection from the cpu/kernel side, then yep, it's another story, of course, executing code in the .data section of an ELF can be easily checked by flagging memory segments with the executable flag, but if there's a buffer overflow that allows overwriting the code in the .text section, then that might be possible (change the stack ret value to go to a legitimate function/code from the OS FW that would call hypervisor_disable() or game_start or something.. and put the right values in the stack to pass it the right parameters, like the device path for example (a /dev/bd should be readable the same as a .iso file...))... although, again, Sony might have been smart enough to make their compiler wrap every function called by a stub which would check the stack before and after every function call... it adds overhead, but adds security against those types of overflow...
In theory emulators could be made I suppose, but it depends on how fast the VM actually is, I dont think it was designed in mind to have NES and Genesis games running thru it! Its like having an armed guard inside a bank safe, even if someone gets in, they will be stopped once the door is open!
hehe, I don't think an emulator would be fast enough on that VM and I don't think that our purpose is a SNES emulator (I can have that on my PC). The interesting part would be to get full access to the hardware... it's also much more challenging
By the way, I said earlier that the FW OS might run under the hypervisor and it just asks the hypervisor access to the hardware through key/auth/other alg. If that's the case, since we got firmware dumps, couldn't that algorithm be retreive by reverse engineering the assembly from the firmware and simply reproduce whatever it does to get the hypervisor disabled? It wouldn't be that hard as far as I can tell.. I mean, yes, if there's some complex algorithm involved, it might be hard to RE, but it would be much easier to just copy the raw binary of the firmware into an ELF and simply call the function pointers directly and let the code be run without even having the need to RE that ASM into C...
What are your thoughts about this ? Should I open a new thread about this to avoid off-topic-ing ? in which case, where should I open it (I can't access the dev section yet).