Sponsored Links

Sponsored Links

Results 1 to 2 of 2



  1. #1
    Contributor naturesbane's Avatar
    Join Date
    Jul 2006
    Posts
    26
    Sponsored Links

    Thumbs Up NPD header format (and compression notes)

    Sponsored Links
    NPD header format observations:

    HEX OFFSET
    00h: NPD magic = "4E 50 44"
    80h: "80h" (current observations indicate fixed)
    83h: compressed = 01; uncompressed = 00
    86h: block size indicator (see below)
    8Ch - 8Fh: uncompressed data length

    The length of header can be dependent upon the chosen block size. Should the data be compressed, the header length is increased by 16 bytes. At the end of the header one will find this new 16 bytes of information added when compression is used:

    Bytes "0-4" : 00's
    Bytes "5-8" : start offset of a block of compressed data
    Bytes "9-12": length of compressed data (1-3 sets of 00's can be included at the end of compressed data block)
    Bytes "13-16": 00 if compression algorithm not applied to that block of data; 01 if compression algorithm is used on that block

    For example, notice in NPD_header2 JPG file (using decimal offsets in this picture), that at offsets 310-311 (decimal) the hex values are "0230", which corresponds to 560 decimal. The block starts at offset 560 decimal. Furthermore, notice the length of the compressed data in bytes is 2Ch, or 44 decimal.

    The block size input, which can be an argument into programs making NPD files (e.g. -b [1,2,4,8,16,32), influences the number of blocks of compressed data. The larger arguments result in larger blocks of compressed data. So, the -b 1 argument would result in a larger number of smaller blocks as shown in the example NPD_header2 JPG file. At 86h in the header, that byte can indicate the block argument used:

    -b 1 -> "04h"
    -b 2 -> "08h"
    -b 4 -> "10h"
    -b 8 -> "20h"
    -b 16 -> "40h"
    -b 32 -> "80h"

    If anyone is interested in understanding the reason for this input, read up on blocking factors and their influence on record (block) sizes. e..g. type "gzip blocking factor" in Google.

    The green circle of hex values shows a pattern commonly found in some PS3 files. This pattern includes input used to determine were compressed data is located and the size of compressed blocks of data (a.k.a. records).
    Attached Thumbnails<br><br> Attached Thumbnails

    npd header.jpg   npd header2.jpg  

  2. #2
    Contributor naturesbane's Avatar
    Join Date
    Jul 2006
    Posts
    26
    Sponsored Links
    Sponsored Links
    The following provides some updated information on this header. The mechanisms for creating the files and compressing data are much better understood.

    Wanted to clarify on information thats already been disclosed here.

    First, the headers. The standard header is always 256 (0x100h) bytes long with the following information:
    00-03h: NPD magic
    80-83h: Compression constant + Compression flag. Specifically, 0x80000000 if not compressed and 0x80000001 if compressed.
    84-87h: Block size indicator (blocking factor - see below)
    88-8Fh: uncompressed (input file) data length

    An extended header follows the standard header. The extended headers size depends upon compression. Specifically:

    Compression Size
    NO

    The following provides some updated information on this file format. The mechanisms for creating the files and compressing data are much better understood.

    Wanted to clarify on information that was previously disclosed here.

    First, the headers. The standard header is always 256 (0x100h) bytes long with the following information:
    00-03h: NPD magic
    80-83h: Compression constant + Compression flag. Specifically, 0x80000000 if not compressed and 0x80000001 if compressed.
    84-87h: Block size indicator (blocking factor - see below)
    88-8Fh: uncompressed data length

    An extended header follows the standard header. The extended headers size depends upon compression. Specifically:

    Compression Size
    NO 0x10h * NUMBER_BLOCKS
    YES 0x20h * NUMBER_BLOCKS

    NUMBER_BLOCKS is related to the blocking factor (and size of input file). Therefore, the statement, "At the end of the header one will find this new 16 bytes of information added when compression is used" was not complete. It is 32 bytes per block when compressed and 16 bytes per block uncompressed with a stated blocking factor. The information captured per my original post was correct for compression scenario, just that the 16 bytes full of information on a block followed 16 leading bytes of 0's.

    The extended header is followed by the uncompressed or compressed data. For compressed data, a block of data always ends on a 16 byte alignment. For example, if you had 0x1Dh bytes of compressed data, the next block of data would start at 0x20h an intern bytes filled with 0's. In this case, start with your length of compressed data block 0x1Dh and add 0x0Fh, then logical AND that result with 0xFFFFFFF0h -> you will get 0x20h.

    In my last post, I discussed how the blocking factor argument was stored in a translated way in the file. That relationship is a double left shift. Take "-b 4" argument. 0x04h is 00000100 in binary. If you left shift twice, you get 00010000, which is 0x10h. Refer to my original post to see this relationship with the other argument values. The value after the left shifting is stored in the standard header and, later, used in the calculation of the size of (compressed data) blocks.

    If you find this information useful, let us know!
    Last edited by naturesbane; 07-09-2007 at 02:59 AM Reason: Automerged Doublepost

 
Sponsored Links

Advertising - Affiliates - Contact Us - PS3 Downloads - PS3 Forums - Privacy Statement - Site Rules - Top - © 2015 PlayStation 3 News