Umm, this might not be a huge help, as I'm pretty much a newbie, but hopefully someone might get an idea from this.
Had a look at some PSP RCO files and they always start with .PRFq [0050 5246 7100]. In fact the first 24 bytes are the same for all PSP RCO files.
The PS3 start with FRP [4652 5000]. The first 14 bytes appear to be the same. The dlna.rco has info linking to 'no media servers were found' and other errors that pop up within the dash. But I guess everyone already knew that. Sorry.
Tried to open PS3 RCO with RCO Editor and of cause, it spat out errors. Tried changing the hex header, but again no go. Tried to use the PSP to decrypt the PS3 RCO, but not really sure what I'm doing. Not even sure that I should be posting this even, with all the modding genius posting here. Tried Z33's Resurssiklunssi to decrypt but it just looked at me funny, lol.
There are some tools for the PSP to decrypt PRX, RCO, PBP files that some users may find useful. Found them here - used google. http://endlessparadigm.com/forum/archive/index.php?thread-4142.html
Sorry I couldn't be more help, I can only hope that it spurs someone in the right direction, sorry. I'll keep digging away. Good luck.