Welcome
Ladies and Gents:

These forums are now closed and registration disabled.

Please join us at our new forum on Proboards. Our hope is that these new forums are more stable, provide more and better features, and allow continuation of the project forums in a safer, more secure, long term environment.

me3explorer.proboards.com

--The ME3Explorer Team

Current Research: DLC SFAR Fileformat

Technical research related to the structure of Mass Effect game files.

Current Research: DLC SFAR Fileformat

Postby WarrantyVoider » 21 Aug 2013, 20:33

here some info about dlc sfar format, afaik:

Raw Overview:
1.Header
2.Filelist
3.BlockSizeList
4.Blocks


1.Header
public uint Magic; // 0x53464152 = RAFS = SFAR reverse
public uint Version; // 0x00010000 always
public uint DataOffset;
public uint EntryOffset; // 0x20 always, start of FileTable
public uint FileCount;
public uint BlockTableOffset;
public uint MaxBlockSize; // 0x00010000 always
public char[] CompressionScheme;
//None = 0x6E6F6E65
//Lzma = 0x6C7A6D61
//Lzx = 0x6C7A7820


2.FileTable
this is a list of Entries which are 0x1E long
public uint HashA, HashB, HashC, HashD;
public uint BlockSizeIndex;
public uint UncompressedSize;
public byte UncompressedSizeAdder;
public uint DataOffset;
public byte DataOffsetAdder;


now comes the hard part, each fileentry refers to a list of blocks in the blocktable, each entry there is 2 bytes long (ushort). Also the UncompressedSize and DataOffset have to be calculated like following:
RealUncompressedSize = UncompressedSize + UncompressedSizeAdder << 32;
RealDataOffset = DataOffset + DataOffsetAdder << 32;


now if the BlockSizeIndex == -1 (0xFFFFFFFF) then the blocklist only has 1 entry, with RealUncompressedSize (@amarok, shouldnt that be called CompressedSize?) as BlockSize and RealDataOffset as BlockOffset. if its more than that, then the first offset is again RealDataOffset, but the size has to be read from the BlockSizeTable, to calculate the position in the table use following helper function:
private long getBlockOffset(int blockIndex, uint entryOffset, uint numEntries)
{
return (long)((blockIndex * 2) + entryOffset + (numEntries * 0x1E));
}

so seek to that position and read 2 byte wise the sizes of each block, here my implementation, to sum this up:
public void Serialize(SerializingFile con, HeaderStruct header)
{
Header = header;
MyOffset = (uint)con.GetPos();
HashA = con + HashA;
HashB = con + HashB;
HashC = con + HashC;
HashD = con + HashD;
BlockSizeIndex = con + BlockSizeIndex;
UncompressedSize = con + UncompressedSize;
UncompressedSizeAdder = con + UncompressedSizeAdder;
RealUncompressedSize = UncompressedSize + UncompressedSizeAdder << 32;
DataOffset = con + DataOffset;
DataOffsetAdder = con + DataOffsetAdder;
RealDataOffset = DataOffset + DataOffsetAdder << 32;
if (BlockSizeIndex == 0xFFFFFFFF)
{
BlockOffsets = new long[1];
BlockOffsets[0] = RealDataOffset;
BlockSizes = new ushort[1];
BlockSizes[0] = (ushort)RealUncompressedSize;
BlockTableOffset = 0;
}
else
{
int numBlocks = (int)Math.Ceiling((double)UncompressedSize / (double)header.MaxBlockSize);
if (con.isLoading)
{
BlockOffsets = new long[numBlocks];
BlockSizes = new ushort[numBlocks];
}
BlockOffsets[0] = RealDataOffset;
long pos = con.Memory.Position;
con.Seek((int)getBlockOffset((int)BlockSizeIndex, header.EntryOffset, header.FileCount), SeekOrigin.Begin);
BlockTableOffset = con.Memory.Position;
BlockSizes[0] = con + BlockSizes[0];
for (int i = 1; i < numBlocks; i++)
{
BlockSizes[i] = con + BlockSizes[i];
BlockOffsets[i] = BlockOffsets[i - 1] + BlockSizes[i];
}
con.Seek((int)pos, SeekOrigin.Begin);
}
}

well now just read the blocks, decompress and chain them. an entry with following hash will contain the filenames, but thats next on list...
0xB5, 0x50, 0x19, 0xCB, 0xF9, 0xD3, 0xDA, 0x65, 0xD5, 0x5B, 0x32, 0x1C, 0x00, 0x19, 0x69, 0x7C


greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Current Research: DLC SFAR Fileformat

Postby WarrantyVoider » 22 Aug 2013, 23:05

ok here comes the hash to filename challenge: one of the entries in the filelist has the above hash, I dont know the name that generates this hash, but you can find it with the signature from previous post, its content looks like this:

/BIOGame/DLC/DLC_CON_END/PCConsoleTOC.bin
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/Mount.dlc
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/Default_DLC_CON_END.bin
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/ConditionalsDLC_CON_END.cnd
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/ConditionalsDLC_Shared.cnd
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/DLC_CON_END_DEU.tlk
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/DLC_CON_END_ESN.tlk
/BIOGame/DLC/DLC_CON_END/CookedPCConsole/DLC_CON_END_FRA.tlk


now you read this (text) file line wise and generate the hash with following code (ported&adjusted from amorks code):
public static byte[] ComputeHash(string input)
{
byte[] bytes = new byte[input.Length];
for (int i = 0; i < input.Length; i++)
bytes[i] = (byte)Sanitize(input[i]);
var md5 = System.Security.Cryptography.MD5.Create();
return md5.ComputeHash(bytes);
}

public static char Sanitize(char c)
{
switch ((ushort)c)
{
case 0x008C: return (char)0x9C;
case 0x009F: return (char)0xFF;
case 0x00D0:
case 0x00DF:
case 0x00F0:
case 0x00F7: return c;
}
if ((c >= 'A' && c <= 'Z') || (c >= 'À' && c <= 'Þ'))
return char.ToLowerInvariant(c);
return c;
}


now just run through the filelist again, find the entry with the same hash and you have its filename :D

next thing is decompressing an entry, this is a bit heavier,so ill paste just my implementation (based on amaroks code again). here what happens in short: it takes the blockoffsets and sizes from the entry, decompresses them and chains them into a memorystream:
public MemoryStream DecompressEntry(int Index)
{
MemoryStream result = new MemoryStream();
FileEntryStruct e = Files[Index];
int count = 0;
byte[] inputBlock;
byte[] outputBlock = new byte[Header.MaxBlockSize];
long left = e.RealUncompressedSize;
FileStream fs = new FileStream(MyFileName, FileMode.Open, FileAccess.Read);
fs.Seek(e.BlockOffsets[0], SeekOrigin.Begin);
byte[] buff;
if (e.BlockSizeIndex == 0xFFFFFFFF)
{
buff = new byte[e.RealUncompressedSize];
fs.Read(buff,0,buff.Length);
result.Write(buff, 0, buff.Length);
}
else
{
while (left > 0)
{
uint compressedBlockSize = (uint)e.BlockSizes[count];
if (compressedBlockSize == 0)
compressedBlockSize = Header.MaxBlockSize;
if (compressedBlockSize == Header.MaxBlockSize || compressedBlockSize == left)
{
buff = new byte[compressedBlockSize];
fs.Read(buff, 0, buff.Length);
result.Write(buff, 0, buff.Length);
left -= compressedBlockSize;
}
else
{
var uncompressedBlockSize = (uint)Math.Min(left, Header.MaxBlockSize);
if (compressedBlockSize < 5)
{
throw new Exception("compressed block size smaller than 5");
}
inputBlock = new byte[compressedBlockSize];
fs.Read(inputBlock, 0, (int)compressedBlockSize);
uint actualUncompressedBlockSize = uncompressedBlockSize;
uint actualCompressedBlockSize = compressedBlockSize;
outputBlock = SevenZipHelper.Decompress(inputBlock, (int)actualUncompressedBlockSize);
if (outputBlock.Length != actualUncompressedBlockSize)
throw new Exception("Decompression Error");
result.Write(outputBlock, 0, (int)actualUncompressedBlockSize);
left -= uncompressedBlockSize;
}
count++;
}
}
fs.Close();
return result;
}


so far this was simple porting and adjusting amaroks code (which is based on gibbeds code), now I have to figure out the real hard part: compiling it all back with an modefied (possibly bigger sized) replaced/added entry!
the problem here is a fight of resources: if I simply recreate the entire file for a modification, alot of chained .mod jobs will be very very slow. If I collect the changes and compile them at the end (preferred method) I have more error sources to overview, changes could revert each other/have side effects, the "one-.mod-job-does-it-alone" philosophy is broken. I could also maybe add changes to the to the end of the file and clean it up afterwards, dunno, will see what I can do :P

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Current Research: DLC SFAR Fileformat

Postby WarrantyVoider » 26 Sep 2013, 01:15

theres a patch sfar called "Patch_01.sfar", pretty small and you cant open it in normal dlc editor. After some long research I fixed the offsets, to make it readable and also tried to make one file of it working, but it really contains "random" format changes. like adding a zero here and there, wrong block list sizes or generally wrong offsets... so weird, however who wants to play with it: here my modefied "patch" (dont try to run with this! not everything can be extracted)

Image

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Current Research: DLC SFAR Fileformat

Postby TankMaster » 13 Oct 2013, 04:53

I just wanted to make a small post to thank you for your continued work on the DLC SFAR editor(s). Using the new DLCEditor2, I was able to replace a couple PCC files (that had small changes) and not crash the game. Mainly the changes I made were to Patch_SFXWeaponFactory.pcc and Patch_SFXProfileSettings.pcc.

Basically, I changed SFXWeaponFactory.UpgradeLevels_NewGamePlus to 1 (default is 3), so when you do a NGP, it only levels up 1 level at a time instead of 3 (also removed the conditional that stops the upgrades past 2, but that's in the coalesced file). I also changed the SFXProfileSettings.MinReadinessRating to 100 (default is 50), this causes the game to always have 100% GAW assets, even offline. You can also set these in the game via the "Set" command.

FYI, I did not use your modified patch file, I used the default 1.04/1.05 (depending on if I am SP/MP). Just thought I would share some findings and thank you for your continued work.

TankMaster has been thanked by:
TankMaster
User
 
Posts: 173
Joined: 02 Nov 2012, 01:19
Location: Indiana
Has thanked: 6 time
Have thanks: 64 time

Re: Current Research: DLC SFAR Fileformat

Postby HolyDeath3000 » 13 Oct 2013, 15:19

Spoiler:
TankMaster wrote:I just wanted to make a small post to thank you for your continued work on the DLC SFAR editor(s). Using the new DLCEditor2, I was able to replace a couple PCC files (that had small changes) and not crash the game. Mainly the changes I made were to Patch_SFXWeaponFactory.pcc and Patch_SFXProfileSettings.pcc.

Basically, I changed SFXWeaponFactory.UpgradeLevels_NewGamePlus to 1 (default is 3), so when you do a NGP, it only levels up 1 level at a time instead of 3 (also removed the conditional that stops the upgrades past 2, but that's in the coalesced file). I also changed the SFXProfileSettings.MinReadinessRating to 100 (default is 50), this causes the game to always have 100% GAW assets, even offline. You can also set these in the game via the "Set" command.

FYI, I did not use your modified patch file, I used the default 1.04/1.05 (depending on if I am SP/MP). Just thought I would share some findings and thank you for your continued work.


Okay the galactic Readiness for the life of god. Please tell me how to do that in more detail. I really don't like the idea of having to play multiplayer just to get it high enough to get perfect ending and find it ridiculous that bioware forced multiplayer to give perfect ending in SP.


Moved To Help Section.
Last edited by HolyDeath3000 on 13 Oct 2013, 15:57, edited 1 time in total.
David Frost wrote:"Don't aim for success if you want it; just do what you love and believe in, and it will come naturally."
User avatar
HolyDeath3000
User
 
Posts: 88
Joined: 30 Sep 2013, 19:39
Location: South
Has thanked: 30 time
Have thanks: 10 time

Re: Current Research: DLC SFAR Fileformat

Postby WarrantyVoider » 13 Oct 2013, 15:51

HolyDeath3000 wrote:
TankMaster wrote:I just wanted to make a small post to thank you for your continued work on the DLC SFAR editor(s). Using the new DLCEditor2, I was able to replace a couple PCC files (that had small changes) and not crash the game. Mainly the changes I made were to Patch_SFXWeaponFactory.pcc and Patch_SFXProfileSettings.pcc.

Basically, I changed SFXWeaponFactory.UpgradeLevels_NewGamePlus to 1 (default is 3), so when you do a NGP, it only levels up 1 level at a time instead of 3 (also removed the conditional that stops the upgrades past 2, but that's in the coalesced file). I also changed the SFXProfileSettings.MinReadinessRating to 100 (default is 50), this causes the game to always have 100% GAW assets, even offline. You can also set these in the game via the "Set" command.

FYI, I did not use your modified patch file, I used the default 1.04/1.05 (depending on if I am SP/MP). Just thought I would share some findings and thank you for your continued work.


Okay the galactic Readiness for the life of god. Please tell me how to do that in more detail. I really don't like the idea of having to play multiplayer just to get it high enough to get perfect ending and find it ridiculous that bioware forced multiplayer to give perfect ending in SP.


@tank:thanks^^
@holydeath: No problem in asking, but unless you have related infos to add, please move this question to general help/chat. thanks

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section

WarrantyVoider has been thanked by:
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Current Research: DLC SFAR Fileformat

Postby HolyDeath3000 » 13 Oct 2013, 15:53

WarrantyVoider wrote:
HolyDeath3000 wrote:
TankMaster wrote:I just wanted to make a small post to thank you for your continued work on the DLC SFAR editor(s). Using the new DLCEditor2, I was able to replace a couple PCC files (that had small changes) and not crash the game. Mainly the changes I made were to Patch_SFXWeaponFactory.pcc and Patch_SFXProfileSettings.pcc.

Basically, I changed SFXWeaponFactory.UpgradeLevels_NewGamePlus to 1 (default is 3), so when you do a NGP, it only levels up 1 level at a time instead of 3 (also removed the conditional that stops the upgrades past 2, but that's in the coalesced file). I also changed the SFXProfileSettings.MinReadinessRating to 100 (default is 50), this causes the game to always have 100% GAW assets, even offline. You can also set these in the game via the "Set" command.

FYI, I did not use your modified patch file, I used the default 1.04/1.05 (depending on if I am SP/MP). Just thought I would share some findings and thank you for your continued work.


Okay the galactic Readiness for the life of god. Please tell me how to do that in more detail. I really don't like the idea of having to play multiplayer just to get it high enough to get perfect ending and find it ridiculous that bioware forced multiplayer to give perfect ending in SP.


@tank:thanks^^
@holydeath: No problem in asking, but unless you have related infos to add, please move this question to general help/chat. thanks

greetz WV


Will do then, I will move this to general help. Sorry forgot that this is a help/request and I will have it moved. :)
David Frost wrote:"Don't aim for success if you want it; just do what you love and believe in, and it will come naturally."
User avatar
HolyDeath3000
User
 
Posts: 88
Joined: 30 Sep 2013, 19:39
Location: South
Has thanked: 30 time
Have thanks: 10 time


Return to Technical Research

Who is online

Users browsing this forum: No registered users and 0 guests

suspicion-preferred