Welcome
Ladies and Gents:

These forums are now closed and registration disabled.

Please join us at our new forum on Proboards. Our hope is that these new forums are more stable, provide more and better features, and allow continuation of the project forums in a safer, more secure, long term environment.

me3explorer.proboards.com

--The ME3Explorer Team

Serialization in C++

Coder-centric area for programming advice and questions.

Serialization in C++

Postby WarrantyVoider » 08 Jun 2013, 12:15

I wanted to get a feeling for how unreal code works, if you try to fake its behaviour. To serialize something, you override its << and >> operator for every stuff you want to save/load, here an example (also for me to lookup :P)

Spoiler:
#include "stdafx.h"
#include "targetver.h"
#include <stdio.h>
#include <tchar.h>
#include <iostream>
#include <fstream>
using namespace std;

class MyInt
{
private:
int i;
public:
MyInt& MyInt::operator = (int other)
{
i = other;
return *this;
}

operator int()
{
return i;
}

friend fstream& operator << (fstream& out, MyInt obj)
{
out.write((char*)&obj.i, 4);
return out;
}
friend fstream& operator >> (fstream& in, MyInt& obj)
{
in.read((char*)&obj.i, 4);
return in;
}
};

class MyFloat
{
private:
float f;
public:
MyFloat& MyFloat::operator = (float other)
{
f = other;
return *this;
}
operator float()
{
return f;
}
friend fstream& operator << (fstream& out, MyFloat obj)
{
out.write((char*)&obj.f, 4);
return out;
}
friend fstream& operator >> (fstream& in, MyFloat& obj)
{
in.read((char*)&obj.f, 4);
return in;
}
};

class MyObjectClass
{
public:
MyInt a;
char b[100];
MyFloat c;
friend fstream& operator << (fstream& out, MyObjectClass obj)
{
out << obj.a;
out.write(obj.b, sizeof(obj.b));
out << obj.c;
return out;
}
friend fstream& operator >> (fstream& in, MyObjectClass& obj)
{
in >> obj.a;
in.read(obj.b, sizeof(obj.b));
in >> obj.c;
return in;
}
};

int _tmain(int argc, _TCHAR* argv[])
{
fstream MyFileOut("file.bin", ios::out | ios::binary);
MyObjectClass obj;
obj.a = 0x12345678;
memset(&obj.b, 0, 100);
sprintf(obj.b, "Hello World!\n");
obj.c = 3.1415f;
MyFileOut << obj;
MyFileOut.close();
fstream MyFileIn ("file.bin", ios::in | ios::binary);
MyObjectClass obj2;
MyFileIn >> obj2;
MyFileIn.close();
printf("obj2.a = %x\n",(int)obj2.a);
printf("obj2.b = %s", obj2.b);
printf("obj2.c = %f\n", (float)obj2.c);
while(1){}
return 0;
}


the file looks like this:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000 78 56 34 12 48 65 6C 6C 6F 20 57 6F 72 6C 64 21 xV4.Hello World!
00000010 0A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000060 00 00 00 00 00 00 00 00 56 0E 49 40 ........V.I@


greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Serialization in C++

Postby WarrantyVoider » 08 Jun 2013, 20:37

ok, here a closer solution: 1) no >> operator needed 2) I use a "archive" class that knows itself when to read and when to write so 3) same code for loading/saving!!!!

Spoiler:
#include "stdafx.h"
#include "targetver.h"
#include <stdio.h>
#include <tchar.h>
#include <iostream>
#include <fstream>
using namespace std;

class MyArchiveClass
{
private:
fstream* MyFile;
bool IsLoading;
public:
MyArchiveClass(fstream& File)
{
MyFile = &File;
IsLoading = true;
}
MyArchiveClass(char* Filename)
{
MyFile = new fstream(Filename, ios::out | ios::binary);
IsLoading = false;
}

void Close()
{
MyFile->close();
}

void SerializeBytewise(char* P, int len)
{
if(IsLoading)
{
MyFile->read(P, len);
}
else
{
MyFile->write(P, len);
}
}

friend MyArchiveClass& operator << (MyArchiveClass& Ar, int& i)
{
Ar.SerializeBytewise(reinterpret_cast<char*>(&i), sizeof(i));
return Ar;
}

friend MyArchiveClass& operator << (MyArchiveClass& Ar, float& f)
{
Ar.SerializeBytewise(reinterpret_cast<char*>(&f), sizeof(f));
return Ar;
}

MyArchiveClass& operator = (MyArchiveClass other)
{
MyFile = other.MyFile;
IsLoading = other.IsLoading;
return *this;
}
};

int _tmain(int argc, _TCHAR* argv[])
{
MyArchiveClass Ar("file.bin");
int i = 0x12345678; float f = 3.1415f;
Ar << i;
Ar << f;
Ar.Close();
i = 0; f = 0;
fstream MyPackage("file.bin",ios::in | ios::binary);
Ar = MyArchiveClass(MyPackage);
Ar << i;
Ar << f;
Ar.Close();
printf("i = %x\nf = %f", i, f);
while(1){}
return 0;
}

and the file:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000 78 56 34 12 56 0E 49 40 xV4.V.I@


greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Serialization in C++

Postby WarrantyVoider » 08 Jun 2013, 22:06

this makes fun, if I split it up in classes, its actually nice to read:

MainProgram.cpp
Spoiler:
#include "stdafx.h"
#include "Archive.h"
#include "TestObject.h"

int _tmain(int argc, _TCHAR* argv[])
{
Archive Ar("file.bin"); //Create Archive Object, Create File
TestObject obj = TestObject(0x12345678, 3.1415f, "Hello World"); //Create Complex Object
obj.Serialize(Ar); //write to file
Ar.Close(); //file close
obj.DebugPrint(); //check values
obj = TestObject(0, 0, ""); //overwrite with something different
obj.DebugPrint(); //check values
fstream MyPackage("file.bin", ios::in | ios::binary); //open filestream to just created file
Ar = Archive(MyPackage); //create new Archive Object from that file
obj.Serialize(Ar); //this time load from file
Ar.Close(); //file close
obj.DebugPrint(); //check values
while(1){}
return 0;
}


my TestObject class:
Spoiler:
#include "stdafx.h"

class TestObject
{
public:
int i;
float f;
char* c;
TestObject(){};
TestObject(int _i, float _f, char* _c)
{
i = _i;
f = _f;
c = _c;
};
void Serialize(Archive& Ar)
{
Ar << i << f << c;
}

void DebugPrint()
{
printf("i = %x\nf = %f\nc= %s\n\n", i, f, c);
}
};

and here the archive class:

Spoiler:
#include "stdafx.h"

class Archive
{
private:
fstream* MyFile;
bool IsLoading;
public:
Archive(fstream& File)
{
MyFile = &File;
IsLoading = true;
}
Archive(char* Filename)
{
MyFile = new fstream(Filename, ios::out | ios::binary);
IsLoading = false;
}

void Close()
{
MyFile->close();
}

void SerializeBytewise(char* P, int len)
{
if(IsLoading)
{
MyFile->read(P, len);
}
else
{
MyFile->write(P, len);
}
}

friend Archive& operator << (Archive& Ar, int& i)
{
Ar.SerializeBytewise(reinterpret_cast<char*>(&i), sizeof(i));
return Ar;
}

friend Archive& operator << (Archive& Ar, float& f)
{
Ar.SerializeBytewise(reinterpret_cast<char*>(&f), sizeof(f));
return Ar;
}

friend Archive& operator << (Archive& Ar, char*& s)
{
int len;
if(!Ar.IsLoading)
len = strlen(s) + 1;
Ar << len;
if(Ar.IsLoading)
s = new char[len];
Ar.SerializeBytewise(s, len);
return Ar;
}

Archive& operator = (Archive other)
{
MyFile = other.MyFile;
IsLoading = other.IsLoading;
return *this;
}
};

this is pretty universal usable... now I need to implement TArrays :D

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Serialization in C++

Postby WarrantyVoider » 09 Jun 2013, 15:16

I more and more like this way of coding, I wrote a minimal PackageEditor(!), that *could* save, but for now I stick to logging. Usage:
PackageEditor File.pcc > log.txt
and it will log all headerfields, all names and imports... that will look like this:
Opening file	: core.pcc
Magic : 0x9E2A83C1
PackageVersion : 0x02AC
LicenceeVersion : 0x00C2
HeaderSize : 0x0002162C
Foldername : None
PackageFlags : 0xA0A80008
Unknown01 : 0x00000000
NameCount : 0x00000314
NameOffset : 0x0000008E
ExportCount : 0x00000613
ExportOffset : 0x000060D4
ImportCount : 0x00000014
ImportOffset : 0x00005EA4
Unknown02 : 0x0001FDE0
Unknown03 : 0x0002162C
Unknown04 : 0x00000000
Unknown05 : 0x00000000
Unknown06 : 0x00000000
GUID : 0D183BD4-4C123A23-399140B3-E2D45B82
Generations : 1
0. ExportCount : 0x00000613
0. NameCount : 0x00000314
0. NetObjCount : 0x00000624
EngineVersion : 0x000018EF
CookerVersion : 0x0003006B
Unknown07 : 0x15330000
Unknown08 : 0x08AA0000
CompressionFlag : 0x00000001


Names : 788
00000000 : A
00000001 : A1
00000002 : A2
00000003 : Abs
00000004 : ABT_Cubic
00000005 : ABT_EaseIn
...
00000785 : YPlane
00000786 : Z
00000787 : ZPlane


Imports : 20
00000000 : ArrayProperty
00000001 : BioMask4Property
00000002 : BoolProperty
00000003 : ByteProperty
00000004 : Class
00000005 : ClassProperty
00000006 : ComponentProperty
00000007 : Const
00000008 : Enum
00000009 : FloatProperty
00000010 : Function
00000011 : InterfaceProperty
00000012 : IntProperty
00000013 : NameProperty
00000014 : ObjectProperty
...


also check this way of coding the ENTIRE loading AND saving function for the header... isnt that nifty?! I like it alot... :P
void Serialize(Archive& Ar)
{
Ar << Magic << PackageVersion << LicenceeVersion << HeaderSize << Foldername << PackageFlags << Unknown01 ;
Ar << NameCount << NameOffset << ExportCount << ExportOffset << ImportCount << ImportOffset;
Ar << Unknown02 << Unknown03 << Unknown04 << Unknown05 << Unknown06;
Guid->Serialize(Ar);
Gen->Serialize(Ar);
Ar << EngineVersion << CookerVersion << Unknown07 << Unknown08 << CompressionFlags;
}
...


greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Serialization in C++

Postby WarrantyVoider » 28 Jun 2013, 18:52

ok, I tried to do something similar in C# but its not so easy and I cant seem to use <<.

this code defines a container struct, that can either read from or write to value, depending if its set to loading or not
public struct SerializingContainer
{
public bool isLoading;
private MemoryStream Memory;
public SerializingContainer(MemoryStream m)
{
this.Memory = m;
this.isLoading = true;
}

public static int operator + (SerializingContainer Container, int i)
{
if (Container.isLoading)
{
byte[] buff = new byte[4];
Container.Memory.Read(buff, 0, 4);
i = BitConverter.ToInt32(buff, 0);
}
else
{
byte[] buff = BitConverter.GetBytes(i);
Container.Memory.Write(buff, 0, 4);
}
return i;
}

public static float operator + (SerializingContainer Container, float f)
{
if (Container.isLoading)
{
byte[] buff = new byte[4];
Container.Memory.Read(buff, 0, 4);
f = BitConverter.ToSingle(buff, 0);
}
else
{
byte[] buff = BitConverter.GetBytes(f);
Container.Memory.Write(buff, 0, 4);
}
return f;
}

public void Seek(int pos, SeekOrigin origin)
{
try
{
Memory.Seek(pos, origin);
}
catch(Exception)
{
}
}
}


so for what all this work? here an example usage:

public class SkeletalMesh
{
public struct Vector3
{
public float X;
public float Y;
public float Z;
}

public struct BoundingStruct
{
public Vector3 origin;
public Vector3 size;
public float r;
}

public BoundingStruct Bounding = new BoundingStruct();

public SkeletalMesh(SerializingContainer Container)
{
Container.isLoading = true;
Serialize(Container);
}

public void Serialize(SerializingContainer Container)
{
SerializeBoundings(Container);
}

private void SerializeBoundings(SerializingContainer Container)
{
Bounding.origin.X = Container + Bounding.origin.X;
Bounding.origin.Y = Container + Bounding.origin.Y;
Bounding.origin.Z = Container + Bounding.origin.Z;
Bounding.size.X = Container + Bounding.size.X;
Bounding.size.Y = Container + Bounding.size.Y;
Bounding.size.Z = Container + Bounding.size.Z;
Bounding.r = Container + Bounding.r;
}
}


when an SkeletalMesh object is created, it gets a SerializingContainer, which is just a wrapped memorystream with the overloads of + for int and float. it sets IsLoading true and calls the serializing method. while we are loading, values get written into boundingbox structure. If I now want to save, I just set IsLoading to false and call my serializing function again!

why does that work?

take a deeper look at this:
MyVar = Container + MyVar;

there are 2 things that can happen here:
1. when loading: the second operant after + is discarded, from the internal memorystream the value is read and returned
2. when saving: the second operant after + is written to internal memorystream and returned, so MyVar reassigns itself and stays as it is

the plus side is, that I only need to write code once for loading AND saving... again I love this way of working^^

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section

WarrantyVoider has been thanked by:
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time

Re: Serialization in C++

Postby Fog.Gene » 16 Feb 2014, 12:21

I'd like to implement this in C#, but I have a couple of doubts:

1. In the SerializingContainer struct, it seems that the read and write methods seem to only read/write the first 4 bytes of the stream. I'm guessing this is by design, and that's why you put the Seek method in there, right?

2. The internal memory stream in SerializingContainer declares itself in the constructor, but the argument stream looks as if it comes from nowhere. I see you initialize the struct later in the SkeletalMesh's class constructor; I assume that is enough to have it not throw a null exception?

I'm gonna play a bit with this. I'm thinking that I could write a NameTable test class the way you have SkeletalMesh, and see how it works first hand.

I like this a lot btw.

----------------------------------

Edit: I'm going to attempt to answer my own questions:

1) The stream is actually passed to the container when initializing SkeletalMesh class, as its constructor will require a container stream (e.g. a mesh file or whatever). That stream's position will be advancing with the read, so it is logical that the data is always a pos = 0.

2) See 1), the stream is passed with the SkeletalMesh's constructor. I'll have to have a stream declared and instanced before initializing the class anyways.


Good? No

----------------------------------

Edit2: I was wrong above :P SerializingContainer is initialized first, then the class; which makes sense looking at the constructors. Dunno what I was thinking above, probably nothing, that's what.

I copied your serializing struct and a similar class to SkeletalMesh but to read from my NameTable. And man this code is gorgeous! Only thing I added was a method for strings in SerializingContainer.

I still have to play a lot more with it, but it looks like this code and I are gonna be friends :)
Regere: Let the world know of its sovereign. I shall be the rule by which all things are measured.
Polyhistor: Then I am to be the slave, for all things shall be my master.
Fog.Gene
Emeritus
 
Posts: 259
Joined: 20 Feb 2013, 05:09
Has thanked: 119 time
Have thanks: 65 time

Re: Serialization in C++

Postby WarrantyVoider » 16 Feb 2014, 15:43

haha, great reading... :P you know, you could have asked me too^^

greetz WV
always backup your files!
mess with the best or die like the rest!
"I tried everything!" - "mkay, please list that..." ; please dont pm me for help, we have a help section
User avatar
WarrantyVoider
Emeritus
 
Posts: 2270
Joined: 22 Aug 2012, 11:33
Has thanked: 480 time
Have thanks: 626 time


Return to Coders' Help

Who is online

Users browsing this forum: No registered users and 0 guests

suspicion-preferred