PS3 HDD Contents

[HanSooloo]

Hello GrandpaHomer, thanks for the valuable feedback. My comments are below:

1. Can you please let me know what scans (e.g. signature) and what dumps (suppose just partial ones) you need?

Initially I would like to have the regular scan mode (so if you don't mention anything, it will take all 0s).

2. Would it be possible to include another 2 command line parameters in your scan program:

A) Start position for scan
B) Bytes to scan

Good idea! I have been thinking about that one too. There is a 0.4 version I haven't released yet, so I will put this into that before the release probably.

It would be great if you can time stamp the verbose messages (like hddcntr is to help estimate remaining time of scan.

I thought of a "same line progress counter", but the display is not very nice when I put a "\r" at the end of the displayed line. I will think of something else. If you have some ideas on the coding side, I am open to them :-)

Oh - and if you can please print numbers (hard drive sizes etc.) with thousand separators it would greatly increase the readibility of numbers with so many places.

I always thought of having thousands separator for the numbers. Let me try to figure it out!

... Are you viping the drives before the test and if yes by what utility and to what content (e.g. all zeroes)?

I always wipe the hdd with zeros first, and THEN format the hdd with the PS3. This way I always know where the PS3 is modifying the data. To zero the hdd, you can use the following code:

dd if=/dev/zero of=/dev/<your_hdd_device> obs=1M

Allright kids, maybe we are onto something here. I have been formatting my poor hdds with all sorts of filesystems lately and taking images of them to compare against the PS3 FS.

Well, it seems like the layout "resembles" ReiserFS. Just to show some samples here:

PS3 HDD Layout:

Start        End         Length
0x0B7CD000   0x0B7CDFFF  0x10000
0x16F85000   0x16F85FFF  0x10000

ReiserFS HDD Layout:

Start        End         Length
0x0B7CD000   0x0B7CDFFF  0x10000
0x10007E00   0x10007E0F  0x00010
0x16F85000   0x16F85FFF  0x10000
0x18007E00   0x18007E0F  0x00010

These sections are from the first occurance of a repeating section. Notice that the 0x10000 length sections start at the EXACT same address on a 60GB hdd.

Interesting thing, too, is how ReiserFS's 0x00010 section is absent in the PS3 layout. Maybe it is consolidated to the beginning of the hdd. Who knows?

The next step is to copy the same file to both PS3 and the ReiserFS formatted hdd and see where the changes are.

Keep checking back in kids...

[GrandpaHomer]

OK - Here is the version 0.3.1 with modification described above ...

So now it's possible to specify starting position of the scan via -f parameter and number of bytes to scan via -l parameter. (I've not managed to implemet direct seek so at the moment it's simply reading the disk without any processing until desired start possition is reached. If someone could suggest any faster solution please let me know.)

Size of the hard drive is also displayed in GB.

The counter messages are now date / time stamped.

So far I've not managed to implement thousands separator.

I've also changed default inout device from sda to sdf and lowered the default buffer size from 256000000 to 110000000 to make it work with PS3 / YDL5. You can indeed changes both values in source or specify different ones via -h and -b commands.

Compiled bin file is for PS3.

[GrandpaHomer]

There is a 0.4 version I haven't released yet, so I will put this into that before the release probably.

I've uploaded here an "interim" version which I was using over past couple of days to spare other interrested of making the modifications by themselves. And indeed I look forward to your version 0.4.

I have a couple more suggestions for next version:

1. Log File (to mirror screen output) - could be (de-)activated by command line parameter as well

2. In case there is already results.txt file to rename it or just create results with some timestamp addition in the filename (or just number them in sequence - like results001.txt, results002.txt etc.) to avoid unwated owerwriting of previous results.

I Know that both of above are not critical but will increase comfort during multiply scan runs.

Also - what I would like to hear from you is what you want me to test / scan on all those hard drives and what's the priority - e.g should I start with the biggest ones and work down to the smallest etc. I'm ATM using PS3 to run the scan bit it's a bit slugginsh - mainly due to the limited memory (that's why I've changed that defaul buffer value). I'm just finishing installation of another 2 PCs so hopefully today evening or latest tomorrow I should be able to run several concurent scans / imagings / blankings at the same time to increse the "productivity" a bit.

So - now it's back to you and I'm awaiting instructions.

[HanSooloo]

OK - Here is the version 0.3.1 with modification described above ...

Just to make sure, when you are in the Linux side of PS3, the hypervisor will ONLY give access to the Linux formatted part of the hdd, and NOT the full 60GB (or whatever the size of the hdd). So, when you are running the program ON the PS3 Linux, you are not really scanning the full hdd.

Just for that, I got a 2.5" SATA to USB2 adaptor and have always scanned the hdd from a LiveCD Linux distro on CDROM or a USB stick. Pentoo Live CD 2006.1 is the best out there; can fit on a 256MB stick and will give you 5 to 6 MB empty space as well

Here are some comments:

I've also changed default inout device from sda to sdf and lowered the default buffer size from 256000000 to 110000000 to make it work with PS3 / YDL5. You can indeed changes both values in source or specify different ones via -h and -b commands.

I wouldn't use scan hdd on the PS3 itself, since the hypervisor will only "expose" the Linux part of the hdd to you and NOT the whole hdd. It is a better idea to take the hdd and plug it to your PC with a 2.5" SATA to USB2 cable; boot your PC with a LiveCD Linux distro (Pentoo Live is the best choice in my opinion); and then run the scan hdd program on the harddisk.

Also - what I would like to hear from you is what you want me to test / scan on all those hard drives and what's the priority - e.g should I start with the biggest ones and work down to the smallest etc. I'm ATM using PS3 to run the scan bit it's a bit slugginsh - mainly due to the limited memory (that's why I've changed that defaul buffer value). I'm just finishing installation of another 2 PCs so hopefully today evening or latest tomorrow I should be able to run several concurent scans / imagings / blankings at the same time to increse the "productivity" a bit.

I would ask you to do the following:

1. Make sure you are running Linux on a PC and not the PS3
2. Start with the smallest of the hdds
3. Zero the hdd using the "dd" command I posted before
4. Run scan hdd program using the defaults
5. Label the "results.txt" file with the size of the hdd (e.g.; results.60gb.txt)
6. Post the results

[GrandpaHomer]

Just to make sure, when you are in the Linux side of PS3, the hypervisor will ONLY give access to the Linux formatted part of the hdd, and NOT the full 60GB (or whatever the size of the hdd). So, when you are running the program ON the PS3 Linux, you are not really scanning the full hdd.

Indeed - that's why I have one HDD as a system one (from which the PS3 is booting) and another one connected via USB adaptor - sorry for not stating that but I supposed it would be clear from SDF device (other of course it would be SDA) ...

I'm pretty sure it is scanning the whole hard drive as far as results are the same like from other PC using the live distro.

Just for that, I got a 2.5" SATA to USB2 adaptor and have always scanned the hdd from a LiveCD Linux distro on CDROM or a USB stick. Pentoo Live CD 2006.1 is the best out there; can fit on a 256MB stick and will give you 5 to 6 MB empty space as well

Yep - done all that - I have USB to IDE, USB to SATA, SATA to IDE, PCMCI to IDE, IDE 3.5" to IDE 2.5, IDE to Compact Flash and plenty of others.

Here are some comments:

I wouldn't use scan hdd on the PS3 itself, since the hypervisor will only "expose" the Linux part of the hdd to you and NOT the whole hdd. It is a better idea to take the hdd and plug it to your PC with a 2.5" SATA to USB2 cable; boot your PC with a LiveCD Linux distro (Pentoo Live is the best choice in my opinion); and then run the scan hdd program on the harddisk.

I would ask you to do the following:

1. Make sure you are running Linux on a PC and not the PS3
2. Start with the smallest of the hdds
3. Zero the hdd using the "dd" command I posted before
4. Run scan hdd program using the defaults
5. Label the "results.txt" file with the size of the hdd (e.g.; results.60gb.txt)
6. Post the results

1. As explained above - doesn't matter (indeed as far as the system disk is not the scanned one).
2. OK
3. Indeed.
4. OK
5. Will do.
6 Ditto.

Just to explain my setup here - I've used a (slightly modified) SATA data + power backplate to feed the internal sata sockets out of the PS3 so now I can easily swap hard drive almost instantly without fidling with the cage, screws and cover all the time. It also allows me to attach 3.5" drives as well (thus I can go up to 750 GB at the moment) as wel as use IDE drives via SATA to IDE adaptor. So in theory I can try even some ridiculous sizes like 2.1 GB etc. ...

The definition of "freshly installed" is as follows:

1. Go to System Settings and choose Restore to Factory Defaults.
2. When the "Press X to restart" prompt comes on, press and hold the power button to power off the PS3.

The idea here is that the HDD data does not contain ANY customization / personalization and is as generic as it gets.

I don't think there is any customization written to the disk related to the system seting - at least not until first reboot after initialization of the disk. It's a bit pain to re-set evrytime all the settings ...

I might be indeed indeed mistaken but I'll try to verify that tonight.

What I'd suggest instead is:

1. Put the wiped / zeroed disk in

2. Switch on the PS3.

3. Confirm initialization / formating of the drive.

4. At the prompt after formating to press x to reboot switch off the PS3 by holding the power button. (I was also thinking about using the power switch because the PS3 is apparentlyu accessing the disk before power down but I'll try to verify if any changes are made ...)

There is definitely something happening to teh disk AFTER the first reboot - also please see bellow attached result and comment.

OK - I've done - maybe more for a laugh (but I personally believe that could be the "way in") some test with one of my smallest hard drives - only 3.2 GB ...

After initializing by PS3 it shows capacity of 3098 MB with 896 MB free ...

I've done scan for:

1. Zeroed disk (which - BIG surprise - is ... well zeroed )
2. Initialized disk (with PS3 NOT reset to defaults) but without rebout after initialization.
3. Initialized disk (with PS3 NOT reset to defaults) with rebout after initialization.

What I think might be usefull is to fill up those 896 available MBs with some predefined AV files and see what will show up in the images.

ALso - HanSooloo - if you want I can do full images of 3.2 GB drive with all possible "combinations" of initializing so you can have a look by yourself. Just let me know.

[GrandpaHomer]

So - here are the scan results for 3.2 GB (YES - three point two gigabytes!) hard drive - isn't PS3 just fantastic to be able to handle such drive ...

Also - there is a bug in scan-3.0.1 resulting in incorrect seek to start position. Please wait either for my (hopefully fixed) version 3.0.2 which I'm now testing or even better for an "offical" version 0.4 from HanSooloo ...

[HanSooloo]

I don't think there is any customization written to the disk related to the system seting - at least not until first reboot after initialization of the disk. It's a bit pain to re-set evrytime all the settings ...

I remember saying "Factory Defaults", but now I believe that was an unnecessary step. I don't think you need to do it anymore. I would say, "don't hit the power switch at the back", since we want the hdd in a consistent state after the format is finished. Because, the PS3 might be writing some critical info to the hdd at that time. Just my 2 pennies :-)

I would prefer the scan results from Option-2, with the PS3 powered down at the "Press X to Restart" prompt by pressing and holding down the power key on the front of the unit. Thanks for your help.

[GrandpaHomer]

I remember saying "Factory Defaults", but now I believe that was an unnecessary step. I don't think you need to do it anymore. I would say, "don't hit the power switch at the back", since we want the hdd in a consistent state after the format is finished. Because, the PS3 might be writing some critical info to the hdd at that time. Just my 2 pennies :-)

There is no problem with that as far as I can see any activity on "externally" connected drive anyway so I'd indeed wait till the disk is idle before powering the PS3 down. There is definitely some disk activity when you'll press and hold power button on the front panel. Just for the peace of mind I can do all 3 cases again and make a full image and post it here (compressed) so you can have a look by yourself as well.

I would prefer the scan results from Option-2, with the PS3 powered down at the "Press X to Restart" prompt by pressing and holding down the power key on the front of the unit. Thanks for your help.

I'd also say that that's the best option of all 3 - there should be as less "unnecessary" data written to the disk as possible.

So - just to reiterate the "proper" procedure to initialize disk for scan:

1. There is no need for reseting to the default PS3 configuration.

2. Wipe / zero the disk using the dd command:

dd if=/dev/zero of=/dev/<your_hdd_device> obs=1M

3. Insert (attach) the wiped disk to your PS3 and power it up.

4. Confirm the initialization / formating of the hard drive. Do NOT press x at the end of the initialization!

5. Press and hold the front panel power button to shutdown the PS3.

6. Remove / detach the disk and perform scan / create image.

I'm just finishing the test of 3.0.2 "interim" version - the only changes are fixed bug with seek to start position and added prompt to confirm that you'd like to proceed with scan of the disk (after the size in GB is displayed) which allows you to abort in case you've by accident selected the wrong SDx device. There is also additional -c command line parameter to force the prompt (or not to prompt) in case the scan will be used as a part of the script.

I have a couple more suggestions to scan utility - if you'd like to incorporate any of those in next version:

1. Prompt to proceed with scan (as above).

2. Command line parameter -c (as above)

3. Using as a source of the scan also the image file to allow us repeated scan of specific "disk" configuration.

4. Put the line in the results.txt stating the date / time and size of the disk

And finally - couple of thoughts about various possible directions of future research:

1. I quite like the idea of using really small drive (like this 3.2 GB one) for various experiments with filling up the disk by pre-defined content as far as there is a very limited space to scan / look through.

2. Also I would like to experiment with "downloading" some predefined files to PS3 via proxy - it will indeed crash / fail upon file integrity check but it should help us to discover which area(s) of the drive are used for the "temporary" files and what's the structure. There might be need of forcing the process to end prematurely or even powering the PS3 at specific point of the download to avoid deletion of the temporary file but again - it could lead to some additional valuable info about the hard drive allocations.

3. I was also toying with idea of filling up the disk almost to the top and the monitor the PS3 behavior when it will run out of space during various situations - e.g. downloading and installing some demos or trailers, uploading new AV files to HDD and so on. I know it's a bit "of the course" here but it's still related to above points. Also - I know people are a bit skeptical regarding all "crashing the PS3" attempts but in controlled environment it could lead to some interesting results as well.

[GrandpaHomer]

All right - here is the version 0.3.2

The seek error should be fixed now.

[HanSooloo]

A new version of scan hdd is released. In this version I tried to implement changes suggested by GrandpaHomer. His 0.3.2 is forked from 0.3 of the original release, so we still need to integrated the code back into the main trunk. But that is just release management.

From the ChangeLog

2007-04-23 0.4.1
> Added new parameters based on feedback from GrandpaHomer
-f position to start scanning the hdd e.g.; 0
-l number of bytes to scan from start position e.g.; 1000
-u units to use in displaying counters (1 char) e.g.; K(B), M(B), G(B)

GrandpaHomer's code is used mostly as is in this version of the code
Big thank you goes to him!

2007-04-16 0.4
> Added 2 new command line options for exact match signature matching
-m 0: Non-match of blocks reported, 1: Matches of blocks reported
-t Free form signature made up of continuoys 2-character hex values e.g.; 0001020304

GrandpaHomer: before your next release, can you please give me a chance to incorporate your latest changes so we don't have different versions of the code out there?
My idea is to incorporate your 0.3.2 changes into a 0.4.2 and from then on go on a synchronized release schedule.