Halo 3 Retail Game Research
Posted: Mon Sep 24, 2007 1:51 pm
This post will be kept updated with the latest findings related to Halo 3 files. Currently:
BLF Header
Description: Commonly found to be the header in most Halo 3 related files.
Extra: Always 68 bytes long. Big Endian.
Structure:
Contributors: Prey, Supermodder911, Anthony
EOF Footer
Description: Commonly found to be the footer in most Halo 3 related files.
Extra: Always 273 bytes long.
Structure:
Contributors: Prey
Root -> maps -> images
File Extension: .blf
File Description: The body of these files is either in PNG, or JFIF format. The header is a BLFHeader and the footer is an EOFFooter.
Extra: The JFIF file format is part of the JPEG family. The PNG and JFIF are also both recognised formats.
Structure:
Contributors: Prey, LuxuriousMeat
Root -> maps -> info
File Extension: .mapinfo
File Description: The body of these files contain the name and description of the corresponding map (ie. salvation.mapinfo -> salvation.map), in the several different languages supported by the 360, along with an extra variation of Spanish. The header is a BLFHeader and the footer is an EOFFooter.
Structure:
Contributors: Prey, Tural
Root -> maps
File Extension: .map
File Description: These files, for example, contain within them all the content used within that specific level (ie. guardian.map will contain the textures used in Guardian). Although some "special" maps contain assets that are used within more than one map, and thus referenced by others. These special, or "shared" maps are unplayable (ie. shared.map). Everything is sorted into "tag" structures within the map file, and these tags each have a pointer to their defining properties, or 'meta' (ie. the Mauler weapon tag stores how much ammo it may carry at any one time). Other miscellaneous stuff that is contained includes the Locale tables, Scripts, etc.
Extra: These files are of the byte order 'Big Endian'. Also near all objects within are padded to 4096. The padding is calculated by doing (4096 - (ObjectSize % 4096)) % 4096.
What's what:
General overview of map layout:
Header: The very first thing in the map file.
StringTableIndex: Reached via StringTableIndexOffset found in the Header. Contains pointers to the starts of each string in the StringTable. They are in the same order as the strings, and are 4 bytes in length (int(32)) each time. They are also relevant to the StringTable (ie. string1's pointer will be 0, string2's pointer will be the length of string1 + 1 for the null terminator, etc).
StringTable: Reached via the StringTableOffset found in the Header. Comprised of ASCII strings, which are each ended with a null terminator (0x0).
FileTableIndex: Reached via the FileTableIndexOffset found in the Header. Follows the same structure as the StringTableIndex.
FileTable: Reached via the FileTableOffset found in the Header. Follows the same structure as the StringTable.
Index: Reached via the IndexOffset found in the Header.Extra: TagInfoHeaderOffset and TagInfoHeaderOffset2 are normally equal, and in those cases the second count is 0 (which, seeing as the count is always 0, may mean the value is completely depreciated.)
TagClassIndex: Reached via the TagClassIndexOffset found in the Index.
TagInfo: Reached via the TagInfoOffset found in the Index.Extra: ClassIndex references the TagClass at that index in the TagClassIndex.
Also because tag identifiers are actually 32bit, we have to convert our identifier from 16bit using the following method:[/size]
TagInfoHeader: Reached via the TagInfoHeaderOffset(2) found in the Index.
Magic
Addresses must have other modified addresses added/subtracted from them in order for them to point at their corresponding object in the map file.. opposed to in xbox memory. Current methods that have been found that calculate these "magic" values are as follows:
The meta of a tag is reached by following it's MetaOffset. Several structures exist inside the meta, here is a list of what you can expect: (Note that this list is currently incomplete)
TagRef: A reference to another tag.
LoneID: A reference to another tag, but only by ID.
Reflexive: A pointer to a list somewhere else in the map file, or 'tag block'. The pointer is translated by subtracting the MapMagic. The count of items in the list is also included in the structure. These items are usually referred to as 'chunks', and it is also possible to get reflexives inside of chunks; these are normally referred to as nested or inner reflexives.
Locale Tables
To reach these, you must first go to the "matg - globals\globals" tag meta + 0x1C4, and then there will be 12 structures, one after the other, each holding information for a different language. The structure is as follows:
The addresses are translated by adding the LocaleTableAddrMod found in the Header. The TableIndex's structure is as follows:
The TableOffset must be added to the StringIndex to point to the string in the map. The Table follows the same structure as the StringTable. The strings also contain with-in them 'codes', which are translated to images by the engine before being displayed on screen. The codes are as follows: (incomplete)
Scripts
To reach these you must first go to the only "scnr" tag meta, and read in the following:
The Script's chunks structure is as follows:
The ScriptType is an index into the following list:
And as is the ReturnType into this list:
The ScriptSyntaxes' chunks structure is as follows:
For now, further help with Scripts can be gained from the following Halo 2 resources: (Just keep in mind these were made for Halo 2, not Halo 3)
Soldier of Lite's Halo 2 Scripting Guide
xbox7887's Completed Script Database
Assets.. or 'raw'
Definite pointers and sizes have yet to be found for the assets, but this is a compilation of all the posts in this topic that may help us find them: (newest to oldest)
http://forums.halomods.com/viewtopic.ph ... 219#724219
http://forums.halomods.com/viewtopic.ph ... 872#713872
http://forums.halomods.com/viewtopic.ph ... 101#712101
http://forums.halomods.com/viewtopic.ph ... 849#710849
http://forums.halomods.com/viewtopic.ph ... 999#709999
http://forums.halomods.com/viewtopic.ph ... 904#709904
http://forums.halomods.com/viewtopic.ph ... 852#709852
http://forums.halomods.com/viewtopic.ph ... 351#707351
http://forums.halomods.com/viewtopic.ph ... 800#703800
Map Security
The map file is protected by a series of hashes, and only one/a select few know where they are and how to disable the check in Halo3's default.xex file. They are keeping the knowledge to themselves to prevent abuse of the XBL service. There is no evidence as of yet that anyone has a method for reproducing the hashes, but here are the posts in this topic that may help us to figure it out: (newest to oldest)
http://forums.halomods.com/viewtopic.ph ... 952#708952
http://forums.halomods.com/viewtopic.ph ... 942#708942
http://forums.halomods.com/viewtopic.ph ... 495#701495
http://forums.halomods.com/viewtopic.ph ... 752#700752
http://forums.halomods.com/viewtopic.ph ... 640#700640
http://forums.halomods.com/viewtopic.ph ... 619#700619
http://forums.halomods.com/viewtopic.ph ... 436#700436
http://forums.halomods.com/viewtopic.ph ... 355#700355
http://forums.halomods.com/viewtopic.ph ... 080#695080
Contributors: Prey, Iron_Forge, Anthony, shade45, LuxuriousMeat, -DeToX-
Applications
Currently released applications that open at least one of the above:
5.
Name: Johnson
Click to be directed to post + download
4.
Name: Engineer
Click to be directed to post + download
3.
Name: Johnson
Author(s): Prey
Version: 1.3
Released: Mon Oct 22, 2007
Description: Can open map files, files from the 'images' folder, and files from the 'info' folder. Main purpose is to aid research into the map file, so expect lots of conveniences.
Download: Binary, Source
2.
Name: Engineer [Beta]
Click to be directed to post + download
1.
Name: Mango
Author(s): Prey
Version: 1.0
Released: Mon Sep 24, 2007
Description: Can open files from the 'images' and 'info' folders, as well as .map files.
Download: Binary, Source
E
Feel free to post any of your findings in this thread (of course related to Halo 3 files), and if I deem appropriate: I'll update this post with your research and add your name to the appropriate Contributors list(s).
- Mon Sep 24, 2007 - Topic started.
- Tue Oct 2, 2007 - Post re-organised and re-structured to allow for all findings to have a place within.
- Wed Oct 3, 2007 - Updated .map header, and added more structures.
- Fri Oct 12, 2007 - Updated .blf (and .mapinfo) struct(s) with headerlen and JFIF info. Updated map section with hash location. Added 'Engineer [beta]'.
- Sat Oct 13, 2007 - .map header updated slightly.
- Wed Oct 17, 2007 - Updated .map section with MapMagic and a mass of other structures.
- Thu Oct 18, 2007 - Slight update to TagRef and LoneID structs. Corrected location of Locale pointers in matg.
- Mon Oct 22, 2007 - Added 'Johnson' to the Applications list. Updated map layout overview.
- Tue Oct 23, 2007 - Added in 'What's what' to .map section.
- Sun Dec 09, 2007 - Added to map section: Scripts, Assets and Security sections, Locale codes, and 'Engineer'.
- Sun Jan 13, 2008 - Updated TagInfo, TagRef, LoneID and Reflexive structures.
- Sun Feb 10, 2008 - Added 'Johnson' to the Applications list.
- Mon Feb 25, 2008 - Added BLFHeader and EOFFooter layout.
- Mon Mar 03, 2008 - Updated BLFHeader and EOFFooter. 'Unk's now have offset in name for ease purposes.
- Int: 32bit integer, or 'long'.
- String: ASCII/Unicode-encoded text of variable length.
BLF Header
Description: Commonly found to be the header in most Halo 3 related files.
Extra: Always 68 bytes long. Big Endian.
Structure:
Code: Select all
struct BLFHeader
{
string Word_blf; // Len4
int HeaderSize;
short Unk8;
short Unk10;
short Unk12;
string FileDescription; // Len34, Not present in all files.
int Filesize;
byte[] Unk52; // Len8
int FileContentsSize;
}
EOF Footer
Description: Commonly found to be the footer in most Halo 3 related files.
Extra: Always 273 bytes long.
Structure:
Code: Select all
struct EOFFooter
{
string Word_eof; // Len4
int FooterSize;
byte[] Unk8; // Len(FooterSize - 8)
}
Root -> maps -> images
File Extension: .blf
File Description: The body of these files is either in PNG, or JFIF format. The header is a BLFHeader and the footer is an EOFFooter.
Extra: The JFIF file format is part of the JPEG family. The PNG and JFIF are also both recognised formats.
Structure:
Code: Select all
struct BLFImageFile
{
BLFHeader blfHeader;
byte[] ImgBuffer; // Either PNG or JFIF
EOFFooter eofFooter;
}
Root -> maps -> info
File Extension: .mapinfo
File Description: The body of these files contain the name and description of the corresponding map (ie. salvation.mapinfo -> salvation.map), in the several different languages supported by the 360, along with an extra variation of Spanish. The header is a BLFHeader and the footer is an EOFFooter.
Structure:
Code: Select all
struct MapInfoFile
{
BLFHeader blfHeader;
byte Zero1;
string EnglishName; // All strings are unicode. All are Len64
string JapaneseName;
string GermanName;
string FrenchName;
string SpanishName;
string LatinAmericaSpanishName;
string ItalianName;
string KoreanName;
string ChineseName;
byte[] Zero2;
string PortugueseName;
byte[] Zero3;
string EnglishDesc; // All strings are unicode. All are Len256
string JapaneseDesc;
string GermanDesc;
string FrenchDesc;
string SpanishDesc;
string LatinAmericaSpanishDesc;
string ItalianDesc;
string KoreanDesc;
string ChineseDesc;
byte[] Zero4;
string PortugueseDesc;
byte[] Zero5; // Len255
string InternalName1; // Len256
string InternalName2; // Len256
byte[] Zero6; // Varying length
EOFFooter eofFooter;
}
Root -> maps
File Extension: .map
File Description: These files, for example, contain within them all the content used within that specific level (ie. guardian.map will contain the textures used in Guardian). Although some "special" maps contain assets that are used within more than one map, and thus referenced by others. These special, or "shared" maps are unplayable (ie. shared.map). Everything is sorted into "tag" structures within the map file, and these tags each have a pointer to their defining properties, or 'meta' (ie. the Mauler weapon tag stores how much ammo it may carry at any one time). Other miscellaneous stuff that is contained includes the Locale tables, Scripts, etc.
Extra: These files are of the byte order 'Big Endian'. Also near all objects within are padded to 4096. The padding is calculated by doing (4096 - (ObjectSize % 4096)) % 4096.
What's what:
Code: Select all
.map FileName | Corresponding Map
====================================
005_intro ----- Arrival
010_jungle ==== Sierra 117
020_base ------ Crow's Nest
030_outskirts = Tsavo Highway
040_voi ------- The Storm
050_floodvoi == Floodgate
070_waste ----- The Ark
100_citadel === The Covenant
110_hc -------- Cortana
120_halo ====== Halo
130_epilogue -- Epilogue
chill ========= Narrows
construct ----- Construct
cyberdyne ===== The Pit
deadlock ------ High Ground
guardian ====== Guardian
isolation ----- Isolation
mainmenu ====== Main Menu
riverworld ---- Valhalla
salvation ===== Epitaph
shrine -------- Sandtrap
snowbound ===== Snowbound
zanzibar ------ Last Resort
campaign ====== [Unplayable, Used for SinglePlayer resources]
shared -------- [Unplayable, Used for MultiPlayer resources]
- Header - Len12288
- StringTableIndex - Padded
- StringTable - Padded
- FileTable - Padded
- FileTableIndex - Padded
- Assets..?
- Meta
- TagInfoHeader
- TagInfo
- TagClassIndex
- IndexHeader
- Unk
- Meta
- Locale Tables and Indices - Padded
Header: The very first thing in the map file.
Code: Select all
struct Header
{
string WordHead; // Len4
int Version; // 11 For H3 Maps
int Filesize;
byte[] Zero1; // Len4
int IndexOffset; // Address
int VirtSegmentStart; // Guess
int VirtSegmentSize; // Guess
byte[] Zero2; // Len256
string BuildInfo; // Len32
short MapTypeIndex; // 0=SP 1=MP 2=MM 3=Shared
byte[] Unk318; // Len26
int StringTableCount;
int StringTableSize;
int StringTableIndexOffset; // Address
int StringTableOffset; // Address
byte[] Zero3; // Len4
int Unk364;
int Unk368;
byte[] Zero4; // Len24
string InternalName; // Len32
byte[] Zero5; // Len4
string ScenarioName; // Len256
int Unk688;
int FileTableCount;
int FileTableOffset; // Address
int FileTableSize;
int FileTableIndexOffset; // Address
int Checksum; // Every 4 bytes xor'ed together after the header
byte[] Unk712; // Len32 - Constant
int MapMagicBaseAddr;
byte[] Unk748; // Len128
byte[] Hash; // Len256
int Unk1132;
int MapMagicAddrMod1;
int Unk1140;
int LocaleTableAddrMod;
byte[] Unk1148; // Len12
int MapMagicAddrMod2;
byte[] Unk1164; // Len11120
string WordFoot; // Len4
}
StringTable: Reached via the StringTableOffset found in the Header. Comprised of ASCII strings, which are each ended with a null terminator (0x0).
FileTableIndex: Reached via the FileTableIndexOffset found in the Header. Follows the same structure as the StringTableIndex.
FileTable: Reached via the FileTableOffset found in the Header. Follows the same structure as the StringTable.
Index: Reached via the IndexOffset found in the Header.
Code: Select all
struct Index
{
int TagClassCount;
int TagClassIndexOffset; // Address
int TagCount;
int TagInfoOffset; // Address
int TagInfoHeaderCount;
int TagInfoHeaderOffset; // Address
int TagInfoHeaderCount2;
int TagInfoHeaderOffset2; // Address
byte[] Unk32; // Len4
string WordTags; // Len4
}
TagClassIndex: Reached via the TagClassIndexOffset found in the Index.
Code: Select all
struct TagClass
{
string Class; // Len4
string ParentClass; // Len4
string GrandParentClass; // Len4
int Identifier;
}
Code: Select all
struct Tag
{
short ClassIndex;
int Identifier; // Read as 16bit (short)
int MetaOffset; // Address
}
Also because tag identifiers are actually 32bit, we have to convert our identifier from 16bit using the following method:
Code: Select all
Identifer <<= 16;
Identifier |= TagIndex; // Index in TagInfo list
TagInfoHeader: Reached via the TagInfoHeaderOffset(2) found in the Index.
Code: Select all
struct TagInfoHeaderItem
{
string Class; // Len4
int Unk4; // Len4
}
Addresses must have other modified addresses added/subtracted from them in order for them to point at their corresponding object in the map file.. opposed to in xbox memory. Current methods that have been found that calculate these "magic" values are as follows:
- HeaderMagic: StringTableIndexOffset - HeaderLength (12288).
Usage In Header:- VirtSegmentStart -= HeaderMagic
- StringTableIndexOffset -= HeaderMagic
- StringTableOffset -= HeaderMagic
- FileTableOffset -= HeaderMagic
- FileTableIndexOffset -= HeaderMagic
- MapMagic: MapMagicBaseAddr - (MapMagicAddrMod1 + MapMagicAddrMod2)
Usage In Header:- IndexOffset -= MapMagic
- TagClassIndexOffset -= MapMagic
- TagInfoOffset -= MapMagic
- TagInfoHeaderOffset -= MapMagic
- TagInfoHeaderOffset2 -= MapMagic
- MetaOffset -= MapMagic
The meta of a tag is reached by following it's MetaOffset. Several structures exist inside the meta, here is a list of what you can expect: (Note that this list is currently incomplete)
TagRef: A reference to another tag.
Code: Select all
struct TagRef
{
string Class; // Len4
byte[] Zero1; // Len8
int Identifier;
}
Code: Select all
struct LoneID
{
int Identifier;
}
Code: Select all
struct Reflexive
{
int ChunkCount;
int Pointer; // Address
byte[] Zero1; // Len4
}
To reach these, you must first go to the "matg - globals\globals" tag meta + 0x1C4, and then there will be 12 structures, one after the other, each holding information for a different language. The structure is as follows:
Code: Select all
struct Locale
{
int Count;
int Size;
int TableIndexOffset; // Address
int TableOffset; // Address
byte[] Unk16; // Len52
}
Code: Select all
struct LocaleTableIndex
{
int Unk0;
int StringIndex;
}
Code: Select all
Start: EE848C20
A Button: EE848020
B Button: 20EE8481
X Button: EE848220
Y Button: 20EE8483
To reach these you must first go to the only "scnr" tag meta, and read in the following:
Code: Select all
0x3E0 Int Table Size
0x3E8 Int Table Offset
0x3F4 Scripts Reflexive
0x4A4 ScriptSyntaxes Reflexive
Code: Select all
struct ScriptChunk
{
string Name;
short ScriptType;
short ReturnType;
short ExpressionIndex;
short ChunkIndex;
byte[] Unused; // Len12
}
Code: Select all
enum ScriptType : short
{
Startup,
Dormant,
Continuous,
Static,
Stub,
Command_Script
}
Code: Select all
enum ReturnType : short
{
UnParsed,
SpecialForm,
FunctionName,
PassThrough,
Void,
Boolean,
Real,
Short,
Long,
String,
script,
string_id,
unit_seat_mapping,
trigger_volume,
cutscene_flag,
cutscene_camera_point,
cutscene_title,
cutscene_recording,
device_group,
ai,
ai_command_list,
ai_command_script,
ai_behaviour,
ai_orders,
starting_profile,
conversation,
structure_bsp,
navpoint,
point_reference,
style,
hud_message,
object_list,
scenary,
effect,
unit,
looping_sound,
animation_graph,
object_definition,
bitmap,
shader,
UNK,
render_model,
structure_definition,
lightmap_definition,
game_difficulty,
actor_type,
hud_corner,
model_state,
value,
network_event,
value2,
unit_name,
vehicle_name,
weapon_name,
device_name,
scenery_name,
object_name
}
Code: Select all
struct ScriptSyntaxChunk
{
short ExpressionID;
short Identity;
short ValueType;
short ExpressionType;
short SiblingPointer;
short SiblingIndex;
int ScriptStringOffset;
byte[] Value; // Len4
short Unk18;
short Unk20;
}
Soldier of Lite's Halo 2 Scripting Guide
xbox7887's Completed Script Database
Assets.. or 'raw'
Definite pointers and sizes have yet to be found for the assets, but this is a compilation of all the posts in this topic that may help us find them: (newest to oldest)
http://forums.halomods.com/viewtopic.ph ... 219#724219
http://forums.halomods.com/viewtopic.ph ... 872#713872
http://forums.halomods.com/viewtopic.ph ... 101#712101
http://forums.halomods.com/viewtopic.ph ... 849#710849
http://forums.halomods.com/viewtopic.ph ... 999#709999
http://forums.halomods.com/viewtopic.ph ... 904#709904
http://forums.halomods.com/viewtopic.ph ... 852#709852
http://forums.halomods.com/viewtopic.ph ... 351#707351
http://forums.halomods.com/viewtopic.ph ... 800#703800
Map Security
The map file is protected by a series of hashes, and only one/a select few know where they are and how to disable the check in Halo3's default.xex file. They are keeping the knowledge to themselves to prevent abuse of the XBL service. There is no evidence as of yet that anyone has a method for reproducing the hashes, but here are the posts in this topic that may help us to figure it out: (newest to oldest)
http://forums.halomods.com/viewtopic.ph ... 952#708952
http://forums.halomods.com/viewtopic.ph ... 942#708942
http://forums.halomods.com/viewtopic.ph ... 495#701495
http://forums.halomods.com/viewtopic.ph ... 752#700752
http://forums.halomods.com/viewtopic.ph ... 640#700640
http://forums.halomods.com/viewtopic.ph ... 619#700619
http://forums.halomods.com/viewtopic.ph ... 436#700436
http://forums.halomods.com/viewtopic.ph ... 355#700355
http://forums.halomods.com/viewtopic.ph ... 080#695080
Contributors: Prey, Iron_Forge, Anthony, shade45, LuxuriousMeat, -DeToX-
Applications
Currently released applications that open at least one of the above:
5.
Name: Johnson
Click to be directed to post + download
4.
Name: Engineer
Click to be directed to post + download
3.
Name: Johnson
Author(s): Prey
Version: 1.3
Released: Mon Oct 22, 2007
Description: Can open map files, files from the 'images' folder, and files from the 'info' folder. Main purpose is to aid research into the map file, so expect lots of conveniences.
Download: Binary, Source
2.
Name: Engineer [Beta]
Click to be directed to post + download
1.
Name: Mango
Author(s): Prey
Version: 1.0
Released: Mon Sep 24, 2007
Description: Can open files from the 'images' and 'info' folders, as well as .map files.
Download: Binary, Source
E
Feel free to post any of your findings in this thread (of course related to Halo 3 files), and if I deem appropriate: I'll update this post with your research and add your name to the appropriate Contributors list(s).