Welcome
Ladies and Gents:

These forums are now closed and registration disabled.

Please join us at our new forum on Proboards. Our hope is that these new forums are more stable, provide more and better features, and allow continuation of the project forums in a safer, more secure, long term environment.

me3explorer.proboards.com

--The ME3Explorer Team

XML Coalesced Compiler/Decompiler

Non-toolset utilities for the ME Trilogy developed by community coders. Note: Restricted to coder- or modder-oriented tools. User-oriented tools belong where they can be adequately supported (e.g., Nexusmods).

XML Coalesced Compiler/Decompiler

Postby TankMaster » 29 Jun 2014, 02:12

I made a post about this in the Research topic "Adding DLC" but figured I might as well give it an official thread so people could find it easier.
Beware, wall of text below...

I took gibbed's original JSON Coalesced compiler/decompiler program and changed it to support complex XML based projects. Been working on it for over a year and finally think it is in a good enough condition to share now.

Originally it started out as the same as the current JSON version, but started adding features and fixes to the original JSON compiler. It now supports indefinite XML includes (will get to the details of that later), almost emulates the way Unreal Engine handles the config files (Coalesced.bin). I have been able to take most of the content in the DLC & patch Coalesced.bin (Default_*.bin for DLC's/patches) and merge them into the main Coalesced.bin with minimal, if any, issues.

As far as I can tell, the code in ME3 Explorer has 1 minor issue for the DLC Coalesced files; each value has a "type" attribute, this tells the engine how to handle the value. In the traditional Engine.ini files, you had some properties that started with: -, +, . or !. A "+" symbol means to add if it is a unique value (does not already exist). "-" removes that value from the specified property, but must be an exact match or it won't be removed. "." basically just always adds that value regardless of already being present or not. "!' means to remove all values for that property (and technically the property itself). The traditional "=" does not have any of those symbols and will add the value or replace it depending.

From the best I can tell based on observation, type="1" (also tends to have the value "null") is equal to the "!" symbol; type="4" is equal to the "-" symbol; type="0" is equal to the "=" symbol; type="2' (also the default) I believe is equal to the "+" symbol; and type="3" is equal to the "." symbol.

The reason I give this information is because the main Coalesced does not usually deal with anything but type's 0, 1 and 2. Type's 3 & 4 are more frequent in DLC's and are more vital that they remain that type. The current Coalesced editor (both gibbed's and the one in ME3 Explorer) force all values to "type 2" except the values that are "type 1".

Alright, enough about that. I have included a readme (more or less notes on use and some of what I just mentioned) to help out. I also have included my original XML files I got from decompiling each Coalesced.bin file to give a quick start. It also includes a "Sample Mod" that demonstrates how one can break up the XML files into smaller chunks.

You can simply drag and drop an ".xml" or ".bin" over the exe file and it will compile/decompile based on file extension. However, it will not recognize a folder. This also requires .NET 4.5 to be installed.

http://1drv.ms/1jwmRdY

The basic structure of an XML project is:
[somename] means it is an attribute (as you can see from the XML files included).

CoalesceFile
[id] (Optional, not yet used)
[name] (Optional, not yet used)
[source] (Optional, not yet used)

Assets (Required)
Asset
[source] (Required, can be a relative or absolute path)

Settings (Optional)
Setting
[name] (Required)
[value] (Required)

CoalesceAsset
[id] (Optional, not really used yet)
[name] (Optional, not really used yet)
[source] (Required, also shouldn't have to change it from the default value)

Includes (Optional
Include (Optional)
[source] (Required, can be a relative or absolute path)

Sections (Optional)
Section (Optional)
[name] (Required)

Property (Optional)
[name] (Required)
[type] (Optional, defaults to 2) (Current supported types are -1, 0, 1, 2, 3 and 4) (Only use when there are no Value elements)
[ignore] (Optional, defaults to false) (Can only be "false" or "true")
[allowDuplicates] (Optional, defaults to false) (Can only be "false" or "true")

Value (Typically only needed when a property has multiple values, such as an array like the New Game Plus Player Variables)
[type] (Same as Property)
[ignore] (Same as Property)

Possible Settings:
ForceValueType = false|true (default is false)
ForcedValueType = 0|1|2|3|4 (ignored unless ForceValueType is true)


Hopefully that covers that portion of it, and makes it a little easier to understand.

Typically you won't use "ignore", "allowDuplicates" or the Settings elements; but some special cases may require one or more.

BioCredits is a perfect example of needing the "allowDuplicates" on it's property "credits":
<Property name="credits" allowDuplicates="true">

Currently, by default it will ignore values if they already exist as I have not properly handled the "type" in every scenario. Without this, it will easily mess the credits up as it have several duplicate entries (like headers in the credits that share same style).

type="-1" is a special type only meant for the compiler. Right now, whenever it comes across type="1" it will clear any values a property had before that, then add "null" and any values after it. I added type="-1" to give the same functionality but without adding "null" to the Coalesced.bin file, which makes it great if you want to overwrite an array (such as the items in a Store).
<Value type="-1">null</value>

The compiler also parses the Includes (if there are any) then it will handle the the "Sections" in the includer. I plan to add another attribute that tells when to parse an include (such as post-sections instead of pre-sections). Hopefully the included XML files will make things a little clearer than my wall of text.

Please let me know if anyone has any questions or if a bug appears that I have not fixed/found.

TankMaster has been thanked by:
TankMaster
User
 
Posts: 173
Joined: 02 Nov 2012, 01:19
Location: Indiana
Has thanked: 6 time
Have thanks: 64 time

Re: XML Coalesced Compiler/Decompiler

Postby TankMaster » 09 Jul 2014, 09:07

Wanted to add an update to this topic, I made a simple GUI application to ease the use (if in a case you don't want to drag & drop).

The program will detect what the source is and automatically set the destination type, so all you have to know is the path to the file(s)/folder(s).
Another thing is the checkbox "Change destination path after browsing for source" means that after clicking Source's "Browse" button and accepting the change (the "Open" button in the dialog's bottom right corner), it will change the Destination path from the source.

If anyone has any suggestions or if something doesn't work right let me know and I will try to fix it asap.

http://1drv.ms/1svSLgh
TankMaster
User
 
Posts: 173
Joined: 02 Nov 2012, 01:19
Location: Indiana
Has thanked: 6 time
Have thanks: 64 time

Re: XML Coalesced Compiler/Decompiler

Postby giftfish » 09 Jul 2014, 22:28

@TankMaster -- I've run into a couple issues as I'm testing the new ThaneMOD DLC (overall, going really well). All but one involve the coalesced, so I'm hoping you can provide some advice :]

1. Overwriting a war asset -- I've looked in all the other DLC coalesced to see how they handle this, and it doesn't look like any of them do it. We both overwrite and add an additional upgrade to the Hanar/Drell Asset. What's happening is this asset is listed twice on the terminal. And, it is not using the new image we specify, it's using the vanilla image. So, I'm guessing there is some issue with the overwrite. What I don't know is if I'm supposed to "allow duplicates" or try that (-1) trick. Or, if it has to do with the "type" being set to 3 rather than 2.

I should note that the new asset we add for Thane works fine, including the custom image.

2. Character Map Locations -- This is set in the sfxgame.sfxareamapdata > charactermapdata area of bioui.ini. I'm guessing this is also an overwrite issue, since that's what were doing. I specify two new conditionals to control Thane's appearance on the map, but they aren't working after the transition to the DLC mod format. I'm guessing that the game is using the vanilla ones instead.

Based on your response to those two issues, I have a third that is similar and that I should be able to figure out on my own.

Thanks much :)
giftfish
 

Re: XML Coalesced Compiler/Decompiler

Postby TankMaster » 10 Jul 2014, 02:00

Well, if you want to remove a single entry, use type="4". type="-1" and type="1" both clear the entire collection up to the point where that value is. I think right now, for it to work right on type="-1", null has to be there like it is on type="1".

IE: <Value type="1'>null</Value> or <Value type="-1">null</Value>

By default it will not add an entry if it already exists, this is what type="4" is for (in the traditional Unreal config files, it's the one with the -PropertyName=PropertValue). Also, you don't want to use allowDuplicates in this case, it's a special case for situations like the biocredits, since headers are often repeated.

For type="4", it has to be an exact match, except it does ignore case and before it even reaches that point, it strips out any extra whitespace, such as tabs, more than 1 space or new lines. However, if I remember right, it is sensitive to a single space, just not multiple spaces as it trims them down.

Example of type="4":

<Section name="sfxgame.sfxplayersquadloadoutdata">
<Property name="henchloadoutinfo">
<!-- Original entry in Coalesced.bin -->
<Value type="4">(ClassName=SFXPawn_Prothean,WeaponClasses=(LoadoutWeapons_HeavyPistols,LoadoutWeapons_Shotguns))</Value>

<!-- New Entry in DLC_HEN_PR -->
<Value type="3">(ClassName=SFXPawn_Prothean,WeaponClasses=(LoadoutWeapons_HeavyPistols,LoadoutWeapons_AssaultRifles))</Value>
</Property>
</Section>


If you wouldn't mind, could you post the xml for those 2 locations, would be easier to see the issues that way. Hope this helps.
TankMaster
User
 
Posts: 173
Joined: 02 Nov 2012, 01:19
Location: Indiana
Has thanked: 6 time
Have thanks: 64 time

Re: XML Coalesced Compiler/Decompiler

Postby giftfish » 10 Jul 2014, 02:39

@Tankmaster -- Sure. Here's the section of the xml:

<Section name="sfxgame.sfxareamapdata">
<Property name="charactermapdata" type="2">(Floor=AM_CitHosp,srCharacter=349396,srLocation=723573, nConditional=4250)</Property>
<Property name="charactermapdata" type="2">(Floor=AM_CitHosp,srCharacter=349396,srLocation=723575, nConditional=4249)</Property>
</Section>

These two entries control the appearance and location of Thane's name on the Citadel Map. They are identical to the vanilla ones, except the conditionals are different.


Spoiler:
<Section name="sfxgame.sfxgawassetshandler">
<Property name="allassets">
<Value type="3">(ID=185,AssetName="GAWAsset_HanarAndDrellForces",Type=GAWAssetType_Military,SubType=GAWAssetSubType_Fleet,StartingStrength=75,GUIName=715508,GUIDescription=715644,bIsExploration=False,UnlockPlotStates[0]=0,GUICategoryID=0,DebugConditionalDescription="Save the Hanar Homeworld",bShowNotificationOnAward=true,ImagePath="GUI_GameManual.Kinect_OptionNarrative_512x256",NotificationImagePath="GUI_Icons.Notifications.GM_CitSpace_256x128")</Value>
<Value type="3">(ID=300,AssetName="GAWAsset_KepralsTreatment",Type=GAWAssetType_Modifier,StartingStrength=0,GUIName=0,GUIDescription=140000078,bIsExploration=False,ModTargets[0]=(TargetID=185,Value=75),UnlockPlotStates[0]=0,GUICategoryID=0,DebugConditionalDescription="Read final Thane email",bShowNotificationOnAward=true,ImagePath="",NotificationImagePath="GUI_Icons.Notifications.GM_AFleet_256x128")</Value>

ID 185 is the main Hanar/Drell war asset. ThaneMOD changes the Starting Strength and ImagePath. Everything else is the same.

Asset 300 underneath it is new, and I've only included it in the post since it references the modified 185 entry. It's a second upgrade to the Hanar/Drell asset. The first upgrade is in the vanilla game and is unchanged by the mod.

I did try changing their value to "2" earlier which didn't fix anything. From your description, using "4" doesn't make sense to me, since the entries aren't the same as the vanilla, which is sort of the point. Basically, what I want to do is overwrite the vanilla versions of these 3 entries with the ThaneMOD versions.
-------------
One comment about the credits. We're only modding the "endcredits," not the main credits. I'm also not using the "_dlc" suffix on the Property name (or on any of the other Property names). I haven't used the "allowduplicates" yet. I think it's actually working without it. I frapsed the modded section today and it appeared fine, but I need to fraps the vanilla version and do a quick check in the TLK to be certain.
giftfish
 

Re: XML Coalesced Compiler/Decompiler

Postby TankMaster » 10 Jul 2014, 04:59

Ah okay, I see the issue. "charactermapdata" is any array (list), and the way the reader handles these is the "<Value>" elements. Otherwise it can cause some other weird issues. Basically, for lists, you use this setup -> Section -> Property -> Value, as shown below.

I went ahead and did some corrections to the first one. I will use the same example but in the format ME2 used, to show how type=4 is similar to it's "-<PropertyName>=<PropertyValue>" setup. Before I get to that, the way the compiler works (similar to how UE handles it), is when it hits this special type (specific to the Unreal Engine in general), it looks through the config cache (all the config files loaded up to that point, in this case it's the xml files). It looks at the cache and first finds the section, then the property, then if the value with type=4 exists, it removes it from the list.

The example I used in my last post shows what the original value for Javik's weapon loadout in the game, but in the DLC_HEN_PR Coalesced, it removes this value. Then it continues onto the next value, in this case a type=3 (right now type 0, 2, and 3 are pretty much the same as I have not finished mirroring UE's methods yet). Type 2 & 3 add values to the property (or properties to the section).

Mass Effect 3's Setup:
<Section name="sfxgame.sfxareamapdata">
<Property name="charactermapdata">
<!-- Remove existing property values that match; type=4 -->
<Value type="4">(Floor=AM_CitHosp,srCharacter=349396,srLocation=723573, nConditional=2514)</Value>
<Value type="4">(Floor=AM_CitHosp,srCharacter=349396,srLocation=723575, nConditional=2519)</Value>

<!-- Add new property values; type=3 -->
<Value type="3">(Floor=AM_CitHosp,srCharacter=349396,srLocation=723573, nConditional=4250)</Value>
<Value type="3">(Floor=AM_CitHosp,srCharacter=349396,srLocation=723575, nConditional=4249)</Value>
</Property>
</Section>

Mass Effect 2's Setup:
[sfxgame.sfxareamapdata]
; Type=4, Remove existing property values that match
-charactermapdata=(Floor=AM_CitHosp,srCharacter=349396,srLocation=723573, nConditional=2514)
-charactermapdata=(Floor=AM_CitHosp,srCharacter=349396,srLocation=723575, nConditional=2519)

; Type=3, Add new property values
+charactermapdata=(Floor=AM_CitHosp,srCharacter=349396,srLocation=723573, nConditional=4250)
+charactermapdata=(Floor=AM_CitHosp,srCharacter=349396,srLocation=723575, nConditional=4249)


You would have to do the same for the "allassets" property, otherwise, since that value does not exist, it will add it, but typically, Unreal Engine will pick the first one it reaches and ignore the newer value. That's why it shows the value and not the new one.
TankMaster
User
 
Posts: 173
Joined: 02 Nov 2012, 01:19
Location: Indiana
Has thanked: 6 time
Have thanks: 64 time

Re: XML Coalesced Compiler/Decompiler

Postby giftfish » 10 Jul 2014, 14:43

@TankMaster -- Thanks! I'll try doing it this way today. One thing I'm still confused about, though. I copied and pasted the formatting for this and all sections, directly from the From Ashes DLC. I basically opened the From Ashes' xml files, edited them, deleted all the unused sections, and checked them against JohnP's tutorial just to make sure I correctly kept all the "necessary" sections. But, now looking at the xml of the main game coalesced, I see that FA and the main game handles these sections differently. Any idea why?

Also, I'm now going to double check all my sections against the main game coalesced, just to see if there are other differences I need to be aware of.
giftfish
 

Re: XML Coalesced Compiler/Decompiler

Postby giftfish » 10 Jul 2014, 16:25

Update :]

I can verify that worked for everything, TankMaster. Thane's name now appears on the Citadel map correctly, the Hanar/Drell war asset only appears once, now has the proper image, and it's upgrade also works fine. I used the same strategy of the 4/3 value on a vanilla email the mod edits and that also worked.

Also, I can verify that when modifying the "endcredits" I didn't need to use an "allowduplicates" parameter. For mod authors who want to add their credits to BW's, the most logical place is in this "endcredits" section. At least in my opinion. That way it doesn't appear until after all of BW's credits are done. There's only one Title in this batch (Special Thanks) so as long as mod authors don't use the exact same title, they should be ok. At least that's my understanding from you you've said. The only thing that could make it complicated, perhaps, is if mod authors start using identical title fields in each other's mods...or something. Anyway, what's really cool is that all mod authors could now add their own credits, and with how the coalesced works, credits from multiple mods can exist simultaneously since they all just "add on". They would probably be ordered by the DLC priority order, I would guess.

I'm really glad you came out with this editor, TankMaster. With the ME3Explorer editor defaulting all those values to 2 and most folks not knowing what the significance of the values field was, nor what values were valid, there were definitely parts I wouldn't have been able to figure out for ThaneMOD's transition to a DLC mod. Still wondering if Wenchy handles it correctly or not. Maybe I'll investigate...later.

Now to try to figure out why the credits music from the main menu isn't working anymore...but that's another post in a different location, lol.
giftfish
 

Re: XML Coalesced Compiler/Decompiler

Postby gbaoye » 19 Jul 2014, 08:08

.xml Package Coalesced.bin
.xml → Coalesced.bin

I am sorry
Poor English
I hope you can use the most simple words
Describe it
Forgive Google poor English again
gbaoye
User
 
Posts: 1
Joined: 19 Jul 2014, 04:56
Has thanked: 0 time
Have thanks: 0 time


Return to Third-Party Utilities

Who is online

Users browsing this forum: No registered users and 1 guest

suspicion-preferred