Welcome
Ladies and Gents:

These forums are now closed and registration disabled.

Please join us at our new forum on Proboards. Our hope is that these new forums are more stable, provide more and better features, and allow continuation of the project forums in a safer, more secure, long term environment.

me3explorer.proboards.com

--The ME3Explorer Team

ME3 On The Hook

Technical research related to the structure of Mass Effect game files.

Re: ME3 On The Hook

Postby The Fob » 19 Apr 2013, 05:32

Hi WV,
Ok, here comes my question about the dump command. I tried to read it out of the code but I am just not enough of a coder to really get it so I am asking here.
When exactly does the dump command read out the objects? Is it when they are accessed on the hard drive?

I am asking because from what I can see, it looks like the objects only appear in the me3oth.txt when they are loaded into mmemory, not when they are actually activated by the game.

For example a game sequence (like e.g. the main sequence of a BioD pcc file) is loaded at some point with all its objects. These objects are then activated one after the other by the game when they are in memory. For me (analyzing game flow), it would be great to see when the objects are activated, rather than when they are loaded.

As a test, I "sabotaged" one particular object in a sequence. This way, the game freezes at this point but I knew which object was the last one activated. When checking the me3oth.txtx, I can find all the objects of the sequence but they are way up in the list (presumably at the point when they were first loaded into memory). Is there a way to output the objects only when they are activated by the game (so that if I sabotage one, this object (or the one before) would be the last one to show in the dump)?


I have to admit, this post includes a lot of conjecture on my part so let me know if I am way off base here. I am basically just looking for a way to reliably follow the game flow, object by object, so when the game crashes, I can track down the problem. Another way to do this would possibly using the Log objects (see here).
User avatar
The Fob
Modder
 
Posts: 702
Joined: 08 Oct 2012, 04:37
Has thanked: 242 time
Have thanks: 212 time

Re: ME3 On The Hook

Postby WarrantyVoider » 19 Apr 2013, 14:12

I guess you are talking about this:
void LogStuff()
{
FILE* L = NULL;
fopen_s ( &L, "Loot.txt", "a+" );
for ( int i = 0x0; i < GObjObjects->Num(); i++ )
{
if ( ! GObjObjects->Data[ i ] ) { continue; }
fprintf (L, "%6i : 0x%p %s\n", i, GObjObjects->Data[ i ], GObjObjects->Data[ i ]->GetFullName());
}
fclose ( L );
}


first GObjObjects is TArray, it has an adress in memory (that offset I have to find for each me3 version) and if you look that position up, you see 3 int32 : pointer to the table(data),num , count (which should be equal), looks like this (look into dump)
Image

now this table is the global "memory" of all unreal objects the game has access to. also all base objects (like the first 1000 objects f.e.) usually stay as others are created from them. anyway, you should get different dumps if you are in different levels, but you also need to know that unreal loads level files in advance! lets have a look a the table:

Image
here you see a big table of pointers to the actual objects. now you might also understand what if(object==NULL) means, it literally checks if the pointer is zero^^. anyway if you look up such an object you look actually at a vmt representation of that class, means all members from the parent class plus the new members of the current class are in another big table, the format for all these classes you can get by looking up the defenition in the sdk (you need to train a bit to find out how a c++ structure relates to a vmt in memroy, wikipedia has a good article about that)

so coming back to the code: GObjObjects->Num() or (GObjObjects->Count()) accesses that second(third) int32 in first pic, its the global count of objects. now GObjObjects->Data[ i ] is one of the these pointers in the table in the second picture. if you now take GObjObjects->Data[ i ]->GetFullName() and take it apart: you get a pointer to the table, a pointer to an object and a pointer to an member of the object at that position... its all alot "pointing to something" but once you get your head around its actually easy :D

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: ME3 On The Hook

Postby The Fob » 20 Apr 2013, 02:34

Thanks for the explanation!
This kind of confirms what I suspected in my post above though.
you should get different dumps if you are in different levels, but you also need to know that unreal loads level files in advance!


Thing is, I am trying to find a way to monitor game flow within a level. When a level is loaded, all the sequence objects are loaded as well. That always happens, even if I screw things up with my mod
Say I have a sequence that goes like this:
259 -> 178 -> 3913 -> 417

Now I mod something that stops object 3913 from working properly. Usually, if I do that, the game just freezes but often it is very hard to pinpoint where exactly the problem occurs (because often, I mod one object that influences another, etc.).
All the sequence objects will still be loaded at the beginning of the level so they will al show up in the dunp like this

178
(possible other objects in between, like textures, etc.)
259
(possible other objects in between, like textures, etc.)
417
(possible other objects in between, like textures, etc.)
3913
(possible other objects after this, like textures, etc.)

but the game will stop running when it actually tries to activate object 3913.
Right now, I often have to figure it out by multiple days of trial and error where exactly the game stops working. It would be extremely useful to have a monitoring program that logs the actual activation of sequence objects. For example, the output here would be:

259
(possible other objects in between, like textures, etc.)
178
(possible other objects in between, like textures, etc.)
3913

... and no other sequence object after this

Because I can see that 3913 is the last sequence object that was activated before the game froze/crashed, I can start my debugging procedure from there and that would make thing a lot faster and easier.
Of course, if it's just 4 objects like in this example it's not a big deal but often (and especially with my current project) I have 100+ candidates for problems in multiple files. That's when this would come in really handy.
My question was if you think that something like this is possible.

EDIT: Don't want to dilute the hooking thread with this rather specific issue though, so maybe we should take this discussion to another one.
User avatar
The Fob
Modder
 
Posts: 702
Joined: 08 Oct 2012, 04:37
Has thanked: 242 time
Have thanks: 212 time

Re: ME3 On The Hook

Postby WarrantyVoider » 20 Apr 2013, 07:19

nah its ok, all you want is "hooking" the log function (detouring it, dump messages, redirect), but the question is: which of all "Function" objects is the one that has to be hooked? BioHUD.PostRender() f.e. is a function thats called every frame and thats how I draw stuff...

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: ME3 On The Hook

Postby The Fob » 28 Apr 2013, 09:07

Ok, guys, I finally managed to make a script for the hook which allows you to monitor game flow rather well. It's a tad complicated to get going so bear with me.

The idea is to hook a sequence object class. For example, the SeqAct_Log class (while the game no longer really logs events in sequences, the objects are still around in many sequences).
So let's say you want to track whenever the game activates a SeqAct_Log event.

Just load the level where you want to track game flow and use the dump command to see which objects are loaded in this level. Now open the ME3OTH.txt and search for a SeqAct_Log occurrence (e.g., something like "SeqAct_Log PersistentLevel.Main_Sequence.SeqAct_Log"

Add this object to the hook. I am not sure if WV ever made a tutorial on how to add new objects to the hook but if not, I can make one at some point,

Once you added the new object to the hook, you need to make a function that can read out the object when it is activated in game and that gives you the precise information which file an which sequence it is from.
For this, I have written the function GetFullName2, which you'll have to add to Core_functions.h (best to add it right next to the old GetFullName function)
Here is the code for it:
char* UObject::GetFullName2() 
{
if ( this->Class && this->Outer )
{
static char cOutBuffer[ 512 ];
class UObject* OuterObj = this->Outer;

strcpy_s ( cOutBuffer, this->Class->GetName() );
strcat_s ( cOutBuffer, " " );
strcat_s ( cOutBuffer, this->Outer->GetName() );
strcat_s ( cOutBuffer, "." );

while ( OuterObj->Class && OuterObj->Outer )
{
strcat_s ( cOutBuffer, OuterObj->Outer->GetName() );
strcat_s ( cOutBuffer, "." );
OuterObj = OuterObj->Outer;
}

return cOutBuffer;
}

return "(null)";
}


Now, you also need to add a very simple function to the sdk_test.cpp that lets you write the full name and path into the ME3OTH.txt. Here it is:
void LogStuff3()
{
FILE* L = NULL;
fopen_s ( &L, "ME3OTH.txt", "a+" );
fprintf (L, "%s\n", pCallObject->GetFullName2());
fclose ( L );
}


As a last step, you need to call your LogStuff3 function, whenever the hook detects an appropriate object.
In the file sdk_test.cpp, find the unction ProcessEventHooked(). Inside that function, find the line
__asm pushad

right under that line, insert the following code:
if (strstr(pCallObject->GetFullName(),"Seq"))
{
LogStuff3();
}


That's it.. Build and inject but don't forget, you can only inject your monitoring dll AFTER you loaded your level of interest. Otherwise the game will crash.
Once you run the game, a list of all SeqAct_Log objects that are activated will be stored in the ME3OTH.txt.
Looks like this:
Spoiler:
Enter hook...
Found offset : 0x1ab5634
Success!!
Starting Inputcontroller...
SeqAct_Log EXP_Pack001_Orbits.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log EXP_Pack001_Orbits.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log Walking_Character.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log SecurityScan.Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log Security_Airlock.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor.
SeqAct_Log GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log NonLanding_Selection.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log DLC_Prompt.NonLanding_Selection.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log DLC_Prompt.NonLanding_Selection.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log GalaxyMap_VO_Alerts.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log GalaxyMap_VO_Alerts.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log GalaxyMap_VO_Alerts.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log EXP_Pack001_Orbits.GalaxyMap.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log Comm_Should_Bark.Comm_Officer.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log Comm_Should_Bark.Comm_Officer.Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log Main_Sequence.PersistentLevel.TheWorld.BioD_Nor_203CIC.
SeqAct_Log Mark_Mission_in_SM.Area_transition.Main_Sequence.PersistentLevel.TheWorld.BioP_Nor.
SeqAct_Log Mark_Mission_in_SM.Area_transition.Main_Sequence.PersistentLevel.TheWorld.BioP_Nor.
SeqAct_Log Main_Sequence.PersistentLevel.TheWorld.BioP_Global.
SeqAct_Log Main_Sequence.PersistentLevel.TheWorld.BioP_Global.


From there it is relatively easy to go back to the sequence editor and check what exactly happened when and - if e.g. your game crashed because of a mod - where exactly the problem occurred. I hope this is useful for some people for faster bug-fixing.
You can of course monitor any sequence object you like. Thec ode here is designed so that all you have to do is to change the object you add to the hook.

If WV is ok with it, I can upload the solution, so that people can easily modify and use it.

The Fob has been thanked by:
User avatar
The Fob
Modder
 
Posts: 702
Joined: 08 Oct 2012, 04:37
Has thanked: 242 time
Have thanks: 212 time

Re: ME3 On The Hook

Postby WarrantyVoider » 28 Apr 2013, 15:03

Great! Already included in my ME3OTH :P



a general hooking function:
Spoiler:
void HookAll()
{
int count = 0;
LogMessage(L"Start hooking...");
for ( int i = 0; i < GObjObjects->Num(); i++ )
{
if ( ! GObjObjects->Data[ i ] || ! GObjObjects->Data[ i ]->Class ) { continue; }
if (!strncmp(GObjObjects->Data[ i ]->Class->GetName(),"Sequence",8))
{
UObject * obj = static_cast<UObject *> (GObjObjects->Data[ i ]);
hook = new toolkit::VMTHook(obj);
hook->HookMethod(&ProcessEventHooked, 70);
count++;
if(count%10==0)
LogMessage(L".");
}
}
wchar_t* temp = (wchar_t *)malloc(sizeof(wchar_t) * 1024);
swprintf(temp,L"Hooked %i Sequence Objects",count);
LogMessage(temp);
}


greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section

WarrantyVoider has been thanked by:
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: ME3 On The Hook

Postby mnn » 17 May 2013, 13:40

WV: I have not checked whole ME3OTH, but does everything has to be done through in C++? Not that I don't know C++, but it's whole other beast than C# :)
mnn
User
 
Posts: 42
Joined: 22 Aug 2012, 17:10
Has thanked: 1 time
Have thanks: 15 time

Re: ME3 On The Hook

Postby WarrantyVoider » 17 May 2013, 17:23

sure, just port like 10.000 lines over to c# and use something like this: http://www.codingthewheel.com/archives/how-to-inject-a-managed-assembly-dll/ that will be a loooong process I guess, good luck

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: ME3 On The Hook

Postby mnn » 22 May 2013, 18:06

I'm thinking of starting a project, that will "expand" multiplayer in certain way and I would like to use your ME3OTH as basis for doing that. But writing everything in C++ would really slow down the development.

Could I, in theory, use C++/CLI (i.e. managed C++) and relay relevant information to C# side?
mnn
User
 
Posts: 42
Joined: 22 Aug 2012, 17:10
Has thanked: 1 time
Have thanks: 15 time

Re: ME3 On The Hook

Postby WarrantyVoider » 22 May 2013, 18:36

you dont seem to understand the concept of injected dlls. you write a program that runs *inside* another program, there is no other *side* (except you implement tcp connection like me). So yeah in theory you can inject entire .net frame work and base your c# dll on that. but then you cant use me3oth, because its c++ (pointers and stuff)... unless you compile it to a dll, write a c# wrapper for it (10.000+ lines of code, same as rewriting it from scratch) and include it into your c# dll. then c# doesnt know pointers actually (except with tricks), so its pretty useless...
you can always try process to process communication (network, pipes, files, shared memory, etc) but its always slower than beeing a thread inside the process. so even if that sdk was written in c# (which I wouldnt know how, as c# doesnt know pointers) there is no way to use it (afaik). you have to write a least a core c++ application that you remote control, but not in the sense of "read all objects and send me the list", bottleneck here is really p2p communication...

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

PreviousNext

Return to Technical Research

Who is online

Users browsing this forum: No registered users and 0 guests

suspicion-preferred