You have to have Python installed correctly. See techbliss.org/threads/setting-up-ida-with-pyside-and-pyqt-it-makes-big-difference.211/ for help (there are finished packages there somewhere).
First set system environment setting: itechtics.com/customize-windows-environment-variables/
Make new one called:
Turn Firewall off when debugging or pass it throug or it wont connect. Restart PC.
Okay fire up ida and load a spu.elf file. Choose spu proccessor
Right away after its loaded you would notice that we can selct debuggers, the original spu dosent support this.
I code so it start with the remote gdb debugger since this is what we need, we have to setup some small stuff before we start, press debugger options at the top.
Tick run program before debugging and choose configoration.
Now i have made two options we can choose from.
Debugger or dump ls. We start with dump since its the fun one.
Press ok ok ok. I will give a short explainasion later.
Press run the green arrow and it dumps the ls.bin
Now we test out the live debugger option. We chosse under debugger settings again.
IMPORTANT !! We have to have max package size set to 1024.
Tick stop on debugging start it seems to be the most stable. Then click edit exceptions in the button.
Locate SIGABRT and press right click edit and untick stop.
You press the green arrow again to run or set Breakpoint and run to cursor.
Success! We can now write custom anergistic to do specific needs. Remeber anergistic is basicly gdbserver in a shell.
Little more about the Dumper module. I use a custom version off Anergistic. That works with patching my way throug a spu file. When loaded any spu module it loads a spesial patcher module also. Credits to computerline
You find it under views>> subview >> patches. Press that.
Under edit >> patches you have two new options.
Select the jump in graph view and select edit selection.
Patch and go to patch windows >> right click and refresh.
To quote: Deci3dbg - Ida Pro debugger module for PlayStation 3
Whoever used to debug on PlayStation 3 knows that there is only one debugger available - SN Systems ProDG. It has some nice features (that I even miss in others debuggers) but overall... its not that good.
There also was a gdb client but it was pulled around 1.xx sdk (specification changed, there is no more step cmd, etc). So after some time that I spent with ProDG I realized that it just dont works for me and decided to get host debugger to communicate with Ida. It would have allowed me to use all those nice features like scripts, plugins, tracing, leaving comments in place, interface and hotkeys to which I am used to over the years.
Okay, I hear that someone of you is asking why I did Ida module and not some kind of gdb proxy instead, and reasons are simple:
I already have experience of making gdb proxys and hosts, but not had experience of making Ida debugger modules (actually not much who did this)
Ida's gdb client is not open source
ppc stub is buggy in 6.1. I do own the latest build of Ida Pro, but lets be honest here, most dont. So I was making this code around this version of Ida. Do not need to mention that Ida Pro 6.5 is backward compatible with 6.1, and even debugger module compiled with idasdk61 would work just fine (much better) with 6.5.
So I did it and it works pretty well. Was testing it for months, catching bugs. At the last time fixed some remaining bugs around half year ago, so it should be much better, but not used it much since then.
General and Float registers
Exceptions, Breakpoints, Step thru code
Hardware breakpoints (DABR)
Threads and Modules
Works with official Sony's Reference Tools and Debug Stations (DECR/DECH)
Also works with custom firmwares
It uses ProDG's TMAPI for communication over deci3 protocol with ps3. Its pretty good and even if deci3 specification docs are leaked its saved alot of time. Therefor, its supports only Windows platform.
Copy content of "C:\Program Files (x86)\SN Systems\PS3\sdk" of your ProDG installation to project folder
Copy deci3dbg.plw and deci3dbg.p64 to plugins folder inside your Ida Pro installation.
How to use
1. Load PPC binary in Ida Pro
2. Debugger -> Select debugger... -> DECI3 debugger plugin
3. Debugger -> Attach to process...
4. Select target and press OK (target may be showed as 'disconnected', bug of TMAPI)
5. Select process
Finally, from zadow28: Oct0xor only made prebuild for IDA 6.5 I rebuilded it for IDA 6.1 here.. you have to have DEX and TM target manager) installed also. I can't test it I fried my PS3 ages ago, and bought a PS4.
I have made a new version that includes so that the debugger shows in SPU proccessor also, just extract in ida folder owerwrite the old ones. If you have spu.plw and spu.p64 in the plugins folder delete that and use this insteed. Python needed. probably needs some more instructions to the code. It uses the ps3tm API so could be that it will work also for spu.