Well, in regards to the file - it is for Debug Systems only, and uses a different key, we have already looked into that.
Furthermore, we can change files on the flash, HOWEVER the certificates are stored inside an encrypted filesystem, which is stored like a file on the flash, so we can not (yet) alter contents inside of it.
In regards to the SELF, the basis for retail SELF's is a SCE header with certain flags, an ELF header with flags, some hashes of encrypted data, then the encrypted data, followed by relocated ELF data at the bottom, which is mostly tables to tell where data is stored inside the decrypted ELF. No, we can not run a SELF inside otheros.
As for any unencrypted self, its a development self.
i had your same idea about man-in-the-middle attack but i was so sad when i saw that ps3 don't allow unsigned certificates :|
i used stunnel to perform the attack providing a self made certificate to the ps3 and the error was "unknown_issuer"
but i have an idea to make the ps thinks that the certificate is valid....
pm me if you're interested.
Not to ruin your enthusiasm but this is hardly going to work. The PS3 most probably (as browsers and OSs do) stores hashes of the trusted certificate authorities (like VeriSign for example). The certificate of Sony servers are signed by these top-level authorities and PS3 can verify that what comes in is really a Sony certificate. It maybe even stores Sony's certificates directly.
So unless you can alter those stored trusted certificates (which you can't) or get a certificate that appears to be Sony's and sign by Verisign (or whoever signed their SSL certificates) you're out of luck - i.e. PS3 won't accept the incoming stuff as genuine and you end up with exactly the same message you have already received... One more option is to brute-force the keys