Halo 3 Retail Game Research

Prey · Post by **Prey** » Mon Sep 24, 2007 1:51 pm

This post will be kept updated with the latest findings related to Halo 3 files. Currently:

Mon Sep 24, 2007 - Topic started.
Tue Oct 2, 2007 - Post re-organised and re-structured to allow for all findings to have a place within.
Wed Oct 3, 2007 - Updated .map header, and added more structures.
Fri Oct 12, 2007 - Updated .blf (and .mapinfo) struct(s) with headerlen and JFIF info. Updated map section with hash location. Added 'Engineer [beta]'.
Sat Oct 13, 2007 - .map header updated slightly.
Wed Oct 17, 2007 - Updated .map section with MapMagic and a mass of other structures.
Thu Oct 18, 2007 - Slight update to TagRef and LoneID structs. Corrected location of Locale pointers in matg.
Mon Oct 22, 2007 - Added 'Johnson' to the Applications list. Updated map layout overview.
Tue Oct 23, 2007 - Added in 'What's what' to .map section.
Sun Dec 09, 2007 - Added to map section: Scripts, Assets and Security sections, Locale codes, and 'Engineer'.
Sun Jan 13, 2008 - Updated TagInfo, TagRef, LoneID and Reflexive structures.
Sun Feb 10, 2008 - Added 'Johnson' to the Applications list.
Mon Feb 25, 2008 - Added BLFHeader and EOFFooter layout.
Mon Mar 03, 2008 - Updated BLFHeader and EOFFooter. 'Unk's now have offset in name for ease purposes.

Note that code representations are done in C#, and so follow the C# syntax. This can be slightly confusing with the namings of some datatypes, so just to clear them up:

Int: 32bit integer, or 'long'.
String: ASCII/Unicode-encoded text of variable length.

BLF Header
Description: Commonly found to be the header in most Halo 3 related files.
Extra: Always 68 bytes long. Big Endian.
Structure:

Code: Select all

    struct BLFHeader
    {
        string Word_blf;        // Len4
        int HeaderSize;
        short Unk8;
        short Unk10;
        short Unk12;
        string FileDescription; // Len34, Not present in all files.
        int Filesize;
        byte[] Unk52;           // Len8
        int FileContentsSize;
    }

Contributors: Prey, Supermodder911, Anthony

EOF Footer
Description: Commonly found to be the footer in most Halo 3 related files.
Extra: Always 273 bytes long.
Structure:

Code: Select all

    struct EOFFooter
    {
        string Word_eof; // Len4
        int FooterSize;
        byte[] Unk8;     // Len(FooterSize - 8)
    }

Contributors: Prey

Root -> maps -> images
File Extension: .blf
File Description: The body of these files is either in PNG, or JFIF format. The header is a BLFHeader and the footer is an EOFFooter.
Extra: The JFIF file format is part of the JPEG family. The PNG and JFIF are also both recognised formats.
Structure:

Code: Select all

    struct BLFImageFile
    {
        BLFHeader blfHeader;
        byte[] ImgBuffer;  // Either PNG or JFIF
        EOFFooter eofFooter;
    }

Contributors: Prey, LuxuriousMeat

Root -> maps -> info
File Extension: .mapinfo
File Description: The body of these files contain the name and description of the corresponding map (ie. salvation.mapinfo -> salvation.map), in the several different languages supported by the 360, along with an extra variation of Spanish. The header is a BLFHeader and the footer is an EOFFooter.
Structure:

Code: Select all

    struct MapInfoFile
    {
        BLFHeader blfHeader;
        byte Zero1;
        string EnglishName;    // All strings are unicode. All are Len64
        string JapaneseName;
        string GermanName;
        string FrenchName;
        string SpanishName;
        string LatinAmericaSpanishName;
        string ItalianName;
        string KoreanName;
        string ChineseName;
        byte[] Zero2;
        string PortugueseName;
        byte[] Zero3;
        string EnglishDesc;    // All strings are unicode. All are Len256
        string JapaneseDesc;
        string GermanDesc;
        string FrenchDesc;
        string SpanishDesc;
        string LatinAmericaSpanishDesc;
        string ItalianDesc;
        string KoreanDesc;
        string ChineseDesc;
        byte[] Zero4;
        string PortugueseDesc;
        byte[] Zero5;          // Len255
        string InternalName1;  // Len256
        string InternalName2;  // Len256
        byte[] Zero6;          // Varying length
        EOFFooter eofFooter;
    }

Contributors: Prey, Tural

Root -> maps
File Extension: .map
File Description: These files, for example, contain within them all the content used within that specific level (ie. guardian.map will contain the textures used in Guardian). Although some "special" maps contain assets that are used within more than one map, and thus referenced by others. These special, or "shared" maps are unplayable (ie. shared.map). Everything is sorted into "tag" structures within the map file, and these tags each have a pointer to their defining properties, or 'meta' (ie. the Mauler weapon tag stores how much ammo it may carry at any one time). Other miscellaneous stuff that is contained includes the Locale tables, Scripts, etc.
Extra: These files are of the byte order 'Big Endian'. Also near all objects within are padded to 4096. The padding is calculated by doing (4096 - (ObjectSize % 4096)) % 4096.

What's what:

Code: Select all

.map FileName | Corresponding Map
====================================
005_intro ----- Arrival
010_jungle ==== Sierra 117
020_base ------ Crow's Nest
030_outskirts = Tsavo Highway
040_voi ------- The Storm
050_floodvoi == Floodgate
070_waste ----- The Ark
100_citadel === The Covenant
110_hc -------- Cortana
120_halo ====== Halo
130_epilogue -- Epilogue
chill ========= Narrows
construct ----- Construct
cyberdyne ===== The Pit
deadlock ------ High Ground
guardian ====== Guardian
isolation ----- Isolation
mainmenu ====== Main Menu
riverworld ---- Valhalla
salvation ===== Epitaph
shrine -------- Sandtrap
snowbound ===== Snowbound
zanzibar ------ Last Resort
campaign ====== [Unplayable, Used for SinglePlayer resources]
shared -------- [Unplayable, Used for MultiPlayer resources]

General overview of map layout:

Header - Len12288
StringTableIndex - Padded
StringTable - Padded
FileTable - Padded
FileTableIndex - Padded
Assets..?
Meta
TagInfoHeader
TagInfo
TagClassIndex
IndexHeader
Unk
Meta
Locale Tables and Indices - Padded

Structures:
Header: The very first thing in the map file.

Code: Select all

    struct Header
    {
        string WordHead;            // Len4
        int Version;                // 11 For H3 Maps
        int Filesize;
        byte[] Zero1;               // Len4
        int IndexOffset;            // Address
        int VirtSegmentStart;       // Guess
        int VirtSegmentSize;        // Guess
        byte[] Zero2;               // Len256
        string BuildInfo;           // Len32
        short MapTypeIndex;         // 0=SP 1=MP 2=MM 3=Shared
        byte[] Unk318;              // Len26
        int StringTableCount;
        int StringTableSize;
        int StringTableIndexOffset; // Address
        int StringTableOffset;      // Address
        byte[] Zero3;               // Len4
        int Unk364;
        int Unk368;
        byte[] Zero4;               // Len24
        string InternalName;        // Len32
        byte[] Zero5;               // Len4
        string ScenarioName;        // Len256
        int Unk688;
        int FileTableCount;
        int FileTableOffset;        // Address
        int FileTableSize;
        int FileTableIndexOffset;   // Address
        int Checksum;               // Every 4 bytes xor'ed together after the header
        byte[] Unk712;              // Len32 - Constant
        int MapMagicBaseAddr;
        byte[] Unk748;              // Len128
        byte[] Hash;                // Len256
        int Unk1132;
        int MapMagicAddrMod1;
        int Unk1140;
        int LocaleTableAddrMod;
        byte[] Unk1148;             // Len12
        int MapMagicAddrMod2;
        byte[] Unk1164;             // Len11120
        string WordFoot;            // Len4
    }

StringTableIndex: Reached via StringTableIndexOffset found in the Header. Contains pointers to the starts of each string in the StringTable. They are in the same order as the strings, and are 4 bytes in length (int(32)) each time. They are also relevant to the StringTable (ie. string1's pointer will be 0, string2's pointer will be the length of string1 + 1 for the null terminator, etc).

StringTable: Reached via the StringTableOffset found in the Header. Comprised of ASCII strings, which are each ended with a null terminator (0x0).

FileTableIndex: Reached via the FileTableIndexOffset found in the Header. Follows the same structure as the StringTableIndex.

FileTable: Reached via the FileTableOffset found in the Header. Follows the same structure as the StringTable.

Index: Reached via the IndexOffset found in the Header.

Code: Select all

    struct Index
    {
        int TagClassCount;
        int TagClassIndexOffset;  // Address
        int TagCount;
        int TagInfoOffset;        // Address
        int TagInfoHeaderCount;
        int TagInfoHeaderOffset;  // Address
        int TagInfoHeaderCount2;
        int TagInfoHeaderOffset2; // Address
        byte[] Unk32;             // Len4
        string WordTags;          // Len4
    }

Extra: TagInfoHeaderOffset and TagInfoHeaderOffset2 are normally equal, and in those cases the second count is 0 (which, seeing as the count is always 0, may mean the value is completely depreciated.)

TagClassIndex: Reached via the TagClassIndexOffset found in the Index.

Code: Select all

    struct TagClass
    {
        string Class;            // Len4
        string ParentClass;      // Len4
        string GrandParentClass; // Len4
        int Identifier;
    }

TagInfo: Reached via the TagInfoOffset found in the Index.

Code: Select all

    struct Tag
    {
        short ClassIndex;
        int Identifier;   // Read as 16bit (short)
        int MetaOffset;   // Address
    }

Extra: ClassIndex references the TagClass at that index in the TagClassIndex.
Also because tag identifiers are actually 32bit, we have to convert our identifier from 16bit using the following method:

Code: Select all

Identifer <<= 16;
Identifier |= TagIndex; // Index in TagInfo list

[/size]

TagInfoHeader: Reached via the TagInfoHeaderOffset(2) found in the Index.

Code: Select all

    struct TagInfoHeaderItem
    {
        string Class; // Len4
        int Unk4;     // Len4
    }

Magic
Addresses must have other modified addresses added/subtracted from them in order for them to point at their corresponding object in the map file.. opposed to in xbox memory. Current methods that have been found that calculate these "magic" values are as follows:

HeaderMagic: StringTableIndexOffset - HeaderLength (12288).
Usage In Header:
- VirtSegmentStart -= HeaderMagic
- StringTableIndexOffset -= HeaderMagic
- StringTableOffset -= HeaderMagic
- FileTableOffset -= HeaderMagic
- FileTableIndexOffset -= HeaderMagic
MapMagic: MapMagicBaseAddr - (MapMagicAddrMod1 + MapMagicAddrMod2)
Usage In Header:
- IndexOffset -= MapMagic
Usage In Index:
- TagClassIndexOffset -= MapMagic
- TagInfoOffset -= MapMagic
- TagInfoHeaderOffset -= MapMagic
- TagInfoHeaderOffset2 -= MapMagic
Usage In TagInfo:
- MetaOffset -= MapMagic

Tag Meta
The meta of a tag is reached by following it's MetaOffset. Several structures exist inside the meta, here is a list of what you can expect: (Note that this list is currently incomplete)

TagRef: A reference to another tag.

Code: Select all

    struct TagRef
    {
        string Class;   // Len4
        byte[] Zero1;   // Len8
        int Identifier; 
    }

LoneID: A reference to another tag, but only by ID.

Code: Select all

    struct LoneID
    {
        int Identifier;
    }

Reflexive: A pointer to a list somewhere else in the map file, or 'tag block'. The pointer is translated by subtracting the MapMagic. The count of items in the list is also included in the structure. These items are usually referred to as 'chunks', and it is also possible to get reflexives inside of chunks; these are normally referred to as nested or inner reflexives.

Code: Select all

    struct Reflexive
    {
        int ChunkCount;
        int Pointer;    // Address    
        byte[] Zero1;   // Len4
    }

Locale Tables
To reach these, you must first go to the "matg - globals\globals" tag meta + 0x1C4, and then there will be 12 structures, one after the other, each holding information for a different language. The structure is as follows:

Code: Select all

    struct Locale
    {
        int Count;
        int Size;
        int TableIndexOffset; // Address
        int TableOffset;      // Address
        byte[] Unk16;         // Len52
    }

The addresses are translated by adding the LocaleTableAddrMod found in the Header. The TableIndex's structure is as follows:

Code: Select all

    struct LocaleTableIndex
    {
        int Unk0;
        int StringIndex;
    }

The TableOffset must be added to the StringIndex to point to the string in the map. The Table follows the same structure as the StringTable. The strings also contain with-in them 'codes', which are translated to images by the engine before being displayed on screen. The codes are as follows: (incomplete)

Code: Select all

   Start: EE848C20
A Button: EE848020
B Button: 20EE8481
X Button: EE848220
Y Button: 20EE8483

Scripts
To reach these you must first go to the only "scnr" tag meta, and read in the following:

Code: Select all

0x3E0 Int Table Size
0x3E8 Int Table Offset
0x3F4     Scripts Reflexive
0x4A4     ScriptSyntaxes Reflexive

The Script's chunks structure is as follows:

Code: Select all

    struct ScriptChunk
    {
        string Name;
        short ScriptType;
        short ReturnType;
        short ExpressionIndex;
        short ChunkIndex;
        byte[] Unused; // Len12
    }

The ScriptType is an index into the following list:

Code: Select all

enum ScriptType : short
    {
        Startup,
        Dormant,
        Continuous,
        Static,
        Stub,
        Command_Script
    }

And as is the ReturnType into this list:

Code: Select all

enum ReturnType : short
    {
       UnParsed,
       SpecialForm,
       FunctionName,
       PassThrough,
       Void,
       Boolean,
       Real,
       Short,
       Long,
       String,
       script,
       string_id,
       unit_seat_mapping,
       trigger_volume,
       cutscene_flag,
       cutscene_camera_point,
       cutscene_title,
       cutscene_recording,
       device_group,
       ai,
       ai_command_list,
       ai_command_script,
       ai_behaviour,
       ai_orders,
       starting_profile,
       conversation,
       structure_bsp,
       navpoint,
       point_reference,
       style,
       hud_message,
       object_list,
       scenary,
       effect,
       unit,
       looping_sound,
       animation_graph,
       object_definition,
       bitmap,
       shader,
       UNK,
       render_model,
       structure_definition,
       lightmap_definition,
       game_difficulty,
       actor_type,
       hud_corner,
       model_state,
       value,
       network_event,
       value2,
       unit_name,
       vehicle_name,
       weapon_name,
       device_name,
       scenery_name,
       object_name
    }

The ScriptSyntaxes' chunks structure is as follows:

Code: Select all

    struct ScriptSyntaxChunk
    {
        short ExpressionID;
        short Identity;
        short ValueType;
        short ExpressionType;
        short SiblingPointer;
        short SiblingIndex;
        int ScriptStringOffset;
        byte[] Value; // Len4
        short Unk18;
        short Unk20;
    }

Click to be directed to post + download

4.
Name: Engineer

Click to be directed to post + download

3.
Name: Johnson
Author(s): Prey
Version: 1.3
Released: Mon Oct 22, 2007
Description: Can open map files, files from the 'images' folder, and files from the 'info' folder. Main purpose is to aid research into the map file, so expect lots of conveniences.

Download: Binary, Source

2.
Name: Engineer [Beta]

Click to be directed to post + download

1.
Name: Mango
Author(s): Prey
Version: 1.0
Released: Mon Sep 24, 2007
Description: Can open files from the 'images' and 'info' folders, as well as .map files.

Download: Binary, Source

E
Feel free to post any of your findings in this thread (of course related to Halo 3 files), and if I deem appropriate: I'll update this post with your research and add your name to the appropriate Contributors list(s).

shade45 · Post by **shade45** » Mon Sep 24, 2007 1:56 pm

Here add these offsets to the header

Code: Select all

int string_id_count; //344
int file_table_count; //692

Prey · Post by **Prey** » Mon Sep 24, 2007 1:58 pm

They're already there shade.

shade45 · Post by **shade45** » Mon Sep 24, 2007 2:02 pm

Wow I must be blind as a bat

Iron_Forge · Post by **Iron_Forge** » Mon Sep 24, 2007 2:06 pm

0x10 is the index offset (magic'd of course)...From first glance, the maps seem pretty straight forward...More or less the same as H2...

galvination · Post by **galvination** » Mon Sep 24, 2007 2:09 pm

Why would they change what's worked in the past. >___>
I'm going to need to look into this more ::dl's mango::

shade45 · Post by **shade45** » Mon Sep 24, 2007 2:12 pm

Cool now we need to figure how to calculate that magic.

Anyways once you get the index offset its structured like this (same as in beta)

Code: Select all

        public struct Index
        {
            int tag_type_count;
            int tag_type_index_offset;
            int meta_count;
            int tag_info_offset;
            int unkown; //Always 6
            int main_tags;
            int unkown_zero;
            int main_tags_2;
            int unknown;
            string tags;
        }

Tural · Post by **Tural** » Mon Sep 24, 2007 2:22 pm

Languages on mapinfo files.

English
---
German
French
Spanish
Spanish (Again?)
Italian
---
---
Portuguese

hay guyz im helpin wit h3 reserch lololol

Iron_Forge · Post by **Iron_Forge** » Mon Sep 24, 2007 2:29 pm

Oh, magic is just 3218235392 subtract the size of the map...But if the size of the map / 16384 ends with greater than .5, you have to round it up, adding 16384 to the size of the map, or just subtract that from the constant...

Assuming my maths right...

Also, after filetable count, is file table offset, file table size, and file table index, just like in H2...These offsets seem to use a different magic, but the distance between them is correct, so thats what they are...

Prey · Post by **Prey** » Tue Sep 25, 2007 12:10 pm

Noice work so far guys

...I'll get round to updating my post with the new info sometime soon.

It seems we still need a magic value to translate the pointers in the index though...unless that's the same as the magic you figured I_F? I've yet to try so I'm not sure.

And lulz, nice work Tural. To complete the list: (I think)

English
Japanese
German
French
Spanish
Spanish again it seems
Italian
Korean
Chinese
Portuguese

kornman00 · Post by **kornman00** » Tue Sep 25, 2007 12:52 pm

The supported locales on the 360 are the exact same from the first xbox so I don't see how or why you are finding a "2nd" spanish. Are you sure you're not getting confused with italian? (yes I know you have it listed)

Tural · Post by **Tural** » Tue Sep 25, 2007 12:59 pm

There are two Spanish descriptions in every mapinfo file. Unless you know some language that is identical to Spanish in each of these name and description translations (Meaning it would be Spanish >_>), it is Spanish twice. =x

Latin America Spanish and Spain Spanish?

Prey · Post by **Prey** » Tue Sep 25, 2007 1:12 pm

[quote="Tural"]There are two Spanish descriptions in every mapinfo file. Unless you know some language that is identical to Spanish in each of these name and description translations (Meaning it would be Spanish >_>), it is Spanish twice. =x

Latin America Spanish and Spain Spanish?

kornman00 · Post by **kornman00** » Tue Sep 25, 2007 1:33 pm

[quote="Tural"]
Latin America Spanish and Spain Spanish?

StalkingGrunt911 · Post by **StalkingGrunt911** » Tue Sep 25, 2007 1:37 pm

Well in Spanish last year when I took it my teacher said to never use Vosotros only people in Spain use Vosotros. So maybe if you look and find Vosotros in the Spain section that could be it. I don't have the .map files so I can't really look so I guess I will sound like an idiot if this isn't right. Nosotros/Vosotros mean we. We only had to use Nosotros.

Tural · Post by **Tural** » Tue Sep 25, 2007 1:48 pm

Tural wrote:identical to Spanish in each of these name and description translations

Prey wrote:Just happens that the words/phrases used were the same in both types, both spelling and grammatically wise.

Agent ME · Post by **Agent ME** » Tue Sep 25, 2007 8:48 pm

StalkingGrunt911 wrote:Well in Spanish last year when I took it my teacher said to never use Vosotros only people in Spain use Vosotros. So maybe if you look and find Vosotros in the Spain section that could be it. I don't have the .map files so I can't really look so I guess I will sound like an idiot if this isn't right. Nosotros/Vosotros mean we. We only had to use Nosotros.

Vosotros is the plural of you - in English people often say "you guys" to mean the same thing.
In Latin America 'vosotros' isn't used often, instead usually using the formal form 'ustedes'.

ScottyGEE · Post by **ScottyGEE** » Tue Sep 25, 2007 8:54 pm

Ok, lets put it this way, there are either two spanish languages inside halo 3 of either different kinds or the same thing for unknown reasons...

I doubt it will have any impact on the way we mod (eg "nooo I cant mod the bubble shield because I'm unsure of why there are two spanishs listed! Curse you Bungie!) so talking about Spanish and its variations and reasons for being in halo 3 shouldn't really be discussed.

So back to the modding

Tural · Post by **Tural** » Tue Sep 25, 2007 8:57 pm

This isn't a Spanish discussion thread. In all honesty, who cares? It can go in the program as Spanish twice. There is absolutely no downside to it, and it really doesn't warrant all this discussion.

Beat you Tural <3 SGEE

shade45 · Post by **shade45** » Tue Sep 25, 2007 8:59 pm

Prey wrote: It seems we still need a magic value to translate the pointers in the index though...unless that's the same as the magic you figured I_F? I've yet to try so I'm not sure.

Yes this magic is also used everywhere else in the map

You should be able to load the meta tree the same way as in the beta