[Tool] Union - A Mod for Gothic 1/2 2023-11-06

System Requirements:

* Official game Gothic 1, Gothic 2 or Gothic 2 Night of The Raven
* Windows XP Sp3 or higher
* Visual C++ 2008, 2010
* DirectX7, DirectX8
* NET Framework 4.0
* Administrator rights

Installation:

1. You should make sure that the game has all the latest patches. To do this, you can use the installation package PlayerKit + Patch. It contains:

Gothic.exe - an officially published executable file available for
download. This file will be installed if a user has
one of the many official game versions.

* The x64 bit version of the patch in Gothic.exe uses the LAA (Large Address Aware) hack to allow the use more than 2GB of memory for x32-bit applications. This patch does not affect the engine bytecode, but only replaces the PE header bytes responsible for activating this flag.

Description: https://docs.microsoft.com/en-us/windows/win32/memory/4-gigabyte-tuning

Patch: https://www.worldofgothic.de/dl/download_552.htm

Resources used:

Engine crc32 crc32 (with LAA) Source
Gothic1 1.08k 225BA11E A3496B67 https://www.worldofgothic.de/dl/download_6.htm
Gothic2 1.30 A2EE682C FFB8118B https://www.worldofgothic.de/dl/download_278.htm
Gothic2 2.6 rev2 2BCD7E30 00D32D21 https://www.worldofgothic.de/dl/download_278.htm

Differences between normal executables and with LAA ones:

Address Gothic1 1.08k Gothic1 1.08k (LAA)
0x0000014E 0E 2E
0x00000190 00 F2
0x00000191 00 46
0x00000192 00 82

Address Gothic2 1.30 Gothic2 1.30 (LAA)
0x0000016E 0E 2E
0x000001B0 BC DC

Address Gothic2 2.6 rev2 Gothic2 2.6 rev2 (LAA)
0x00000156 0E 2E
0x00000198 89 A9

GothicStarter.exe - normal + modified manager for launching mods.

Source: https://github.com/GothicFixTeam/GothicFix/tree/master/GothicStarter

XT_DayStd.sgt - music theme file with a loud click corrected.
Xardas_Tower.sty - music theme file with a loud click corrected.
Font_15_White.ftn - replaces the damaged (crc32: C61A14B6) font file for Gothic 1.
OrcAttack.bik - replaces damaged (crc32: any) video for Gothic 2.
Extro_PAL.bik - replaces damaged (crc32: 5E995A57) video for Gothic 2.

Additionally:

- Renames the invalid directory `_Work\Data\Textures\_Complied\` to `_Work\Data\Textures\_Compiled\`
- Replaces the file extension `_Work\Data\Scripts\Content\Cutscene\Ou` `.dat` to `.bin`
- Replaces the file extension `_Work\Data\Scripts\Content\Cutscene\Ou` `.lsc` to `.csl`
- Renames the file `_Work\Data\Scripts\_Copliled\Game.dat` to `_Work\Data\Scripts\_Copliled\Gothic.dat`

2. After that you can install the Union itself. The installation package includes the following files:

Shw32.dll - Union library itself
Vdfs32g.dll - alternative virtual file system
Union.patch - engine's fixes file
SystemPack.ini - settings file
Bink32w.dll - adapted library for playing BIK video Gothic2 -> Gothic1.

The external symbol _BinkSetVolume@12 -> _BinkSetVolume@8 is patched in this library.
The Gothic 2 library supports more formats than the Gothic 1 version.

Union.vdf
* .m3d - files of sound drivers. They are applied in the game settings.
After selecting the driver, the game must be restarted.

For a complete list see Appendix [2]
zActivateCtrl.dll - A library that sets the UnionActivated constant scripts and in Gothic.ini to true if Union is running.
zBinkFix.dll - is a library that extends videos to full screen.
zMouseFix.dll - is a library that brings mouse control to DirectX8.
This mouse is compatible with the Steam overlay!
zMouseFix.patch - auxiliary script for the correct activation of the DirectX8 mouse.
Invalid_Source_File.* - files that will be displayed instead of nonexistent ones if the SystemPack.ini option is set -> UseErrorVisuals = true.

For a complete list see Appendix [3]
Humans_Skeleton-T_Spawn.man - Corrected skeleton spawn animation file.

3. Then you can install Ninja and other patches of your choice.

4. Launch the game in any convenient way.
- SystemPack.ini will be contained with necessary strings on the first launch
- Additionally, the current used ini will be fixed too(Gothic.ini, GothicGame.ini, ..., <mod>.ini)

* [INTERNAL] extendedMenu = 1
* [VIDEO] zVidResFullscreenBPP = 32
* [ENGINE] zTexCacheSizeMaxBytes = 100000000
* [INTERNAL] menuShowVersion = 1
* [INTERNAL] gameStartFailed = 0
* [INTERNAL] gameAbnormalExit = 0

- The System/Autorun folder will be created
- A directory of intermediate files will be created Data/$Templates$
* it looks like a hidden folder when SystemPack.ini -> ShowHiddenFiles = true.
- An entry will be added to the registry
* Path - HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers\
* Parameter - Path of the current executable file
* Value - "DisableNXShowUI HighDpiAware"

Virtual file system:

Union includes an alternative file system. It is faster than the original one.
- It doesn't require dump files. The list of objects is loaded in real time.
- Data is accessed through a faster hash table.
- When reading, file data is cached, which significantly reduces the number of requests to the hard drive.
This significantly increases the loading speed of the game.
- OGG file support is implemented at the file system level. Audio track is decoded
at the moment of reading the file and is transferred to the engine in a form it can work with.
- Ability to quickly finish the process by holding down the CTRL + ALT + Q key combination.
- Ability to display missing files of models, textures and sounds as warning objects.
For this, the SystemPack.ini parameter -> UseErrorVisuals = true must be set. For a complete list of objects, see Appendix [3]

Union loading steps:

1. The first step of Union is to check if DEP and Interface Scaling are disabled. Disabling DEP is necessary so that mods under the control of Ikarus can work seamlessly with the memory of the process. Interface scaling allows the operating system to stretch windows and enlarge text. If you do not disable this
parameter, the user playing in windowed mode will have the window frame increased significantly.
As a result, the game window can be larger than the screen size. These parameters are individual for each EXE file. Turning them off occurs by adding special keys to the registry (DisableNXShowUI and HighDpiAware). If Union writes
to the registry, then the process immediately goes to restart for the changes to take effect. Since this happens in the very first place, the user will not even notice this procedure.

2. At this step, Union includes the first type plugins. The first type plugins are plugins that are loaded BEFORE the virtual filesystem and can only be loaded from physical directories. You can set the list of plugins in SystemPack.ini -> PluginList. At the end of the plugin name you should specify
two asterisks (e.g. Ninja.dll**).

3. Union loads the virtual file system. The loading process identifies all files with the .Patch extension and the contents of the Autorun folders.

4. Union loads the second type plugins. The second type plugins are plugins that are loaded immediately after the virtual file system. Such plugins can be loaded from VDF and MOD volumes. Connection is equivalent to step 2, but one asterisk should be added at the end of the plugin name.

5. In this step, Union loads and executes all .patch files found in the physical and virtual directories of the game (step 3).
.patch files are scripts, the syntax and purpose of which is aimed at making changes to the memory of a process or conducting preparatory measures at the time of launching the engine. An example of such a patch is the main patch of the package - Union.patch, which contains all fixes for Gothic I, Gothic Sequel, Gothic II and Gothic II NoTR. You can read more in detail here: https://worldofplayers.ru/threads/42178/.
Additional patches which are applied from Shw32.dll directly:
* Patch for RTTI Dynamic_cast, for native conversion of Engine Classes and Union Gothic API.
* A patch for the DirectSound3D sound driver, due to which some sound effects disappeared, because the system
thought the sound source was too far away.
* A patch for QueryPerCounter that resets the system uptime, due to which the game periodically did not start.
* Patch for redirecting IO instructions for the ability to read BIK files from VDF or MOD volumes.
* Patch for low FPS and window frame in full screen mode in ddraw.dll for Windows 7 and higher systems.

6. Union waits for the call to the Engine Entry Point. When WinMain is called, Union will load third type of plugins. The third type plugins are plugins that are loaded in parallel to the engine and can be launched from VDF or MOD volumes.
The fundamental difference between this type is that it is possible to define global instances of Gothic classes in them.

Run such plugins in two ways:

I. By specifying their names in SystemPack.ini -> PluginList, you do not need to add asterisks at the end.
II. Putting the libraries in a folder named System/Autorun in a physical or virtual directory.

7. The next step is to determine your Operating System and Graphics Card. Based on this, the optimal launch parameters are selected.
For owners of Intel Graphic or AMD + Win10, the windowed mode will be set, this should compensate for the poor FPS of the fullscreen regime. For NVidia owners, the G-Sync values will be controlled by the game so that the game doesn't crash on new generations of graphics cards.

8. Launching the game. Further, Union will not interfere with the operation of the engine, if plugins are not connected to it, which have a standard a set of callback functions. The standard set of callback functions is a list of functions for which Union will
intercept engine events. For a list of such functions, see Appendix [1]. It should be noted that Union produces hooks
only for those functions for which there is at least one non-Null callback function that is not empty.
To intercept, patch hooks are used that change the addresses of the calling instructions. For more details, see the section `Interception of functions`.

N. The process terminates with the preliminary finishing of the virtual file system. After that VDF and MOD volume again
will become available for changes.

Patch files:
These are scripts that are used to change memory values or committing some actions when starting the engine. Files
patches are launched from any directories monitored by the file system (System, Saves, _Work and any others at the request of the engine).
The launch is performed both from physical directories and from VDF & MOD. If physical and virtual directories contain
identical patches with identical paths, physical copies will have read priority.
* An additional [patch name] .MAP file can be created. This happens when the patch file has at least one patch marked
'static' (see the description of the patch format for more details). This file allows you to immediately download all previously made changes to
process memory, skipping re-parsing of static blocks. By default, the file cannot be seen through
standard files explorer. To make the file hidden, but visible, set the value SystemPack.ini -> ShowHiddenFiles = true.
Description of the patch format: https://worldofplayers.ru/threads/42178/

Plugins:
These are libraries dynamically loaded via Union. Plugins loading can be done both Physically and from VDF or MOD volumes.
There are three options for loading libraries:

1. The classic way is to specify the list of library names in SystemPack.ini -> PluginList separated by commas. Also through this parameter you can control the priority of launching plugins:

I. If you specify two asterisks (plugin.dll**) at the end of the library name, then it will be loaded earlier than the Virtual
file system. But in this case it cannot be loaded from VDF or MOD volume.
II. If you specify one asterisk (plugin.dll*) at the end of the library name, then it will be launched immediately after the Virtual file system. at this stage the library can be loaded from VDF or MOD volume.
III. If the name of the library is specified without changes (plugin.dll), then it will be loaded simultaneously with the game. At this stage libraries have the ability to use global instances of Gothic classes.

2. Autorun folder. If the library is located in the Physical Directory `System/Autorun` or in the Virtual Directory `*/Autorun`, then the library will be loaded along with the engine, as in step 1.III. In current versions of Union, the order of loading plugins from this folder is determined by the dependencies of the libraries on each other. This means that if one plugin imports symbols from another, then the Exporting will be loaded first, and then the Importing.
3. The patch file. They start automatically. This means that plugins can also run patches along with them. There are two
script functions: LoadLibrary ("plugin.dll") and LoadPlugins ("plugin1.dll", ..., "pluginN.dll").

If plugins are written using the Union SDK, then each plugin can have a Standard set of callback functions.
These are functions that called during key game events (Appendix [1]). To call these functions, Union intercepts
related procedures through Hookie-patches (see Interception of functions). Interception of each individual procedure is performed only if there is at least one non-Null callback function and a non-empty function. Union determines whether a function is empty by its bytecode. If the function begins with a RET instruction, then it is considered as empty.

Gothic API:
Plugins containing the Gothic API from Union can take full advantage of the engine's class capabilities. Gothic API offers
classes in which:

1. Implementation for four engines (4 Gothic versions) at once.
2. The sizes of the fields are correct. This means that the programmer can access all the properties of the classes without fear of accessing invalid memory location.
3. The methods work directly with the Engine. Class functions have a simple statement that forces your code to execute
directly from the engine. This is somewhat similar to how the import table works.
4. Correct virtual table. Virtual methods are implemented in such a way that their offsets are the same as offsets engine. To do this, Union takes into account:

I. Virtual methods are arranged in the same order as in the engine table.
II. The compiler swaps the overloaded virtual methods, so in the Gothic API they appear in the reverse order of the table.
III. No non-virtual function name should be the same as the virtual function name, since such an overload also affects the order of virtual methods.
IV. Multiple inheritance order also forms the order of the virtual table. Therefore, you need to choose the right which class should be inherited first.
V. Determining the names of purely virtual methods required the analysis of heirs for their correct placement in the class.

5. The programmer can independently create or delete instances of the engine classes. For this, Union has implemented a common the whole process memory allocation. In plugins on the Union template, the new/delete operators are overloaded containing function calls allocating and freeing memory used by the Game Engine, Virtual File System and Union itself.

6. Dynamic conversion (dynamic_cast) of polymorphic types is possible due to the patched Runtime function and correct virtual table. The patch strips off the extra namespace from the class name, so that the engine perceives API classes as native. At the same time, the performance of the program does not change.

7. Convenient hierarchy of class contents, as well as completely public access to all its members. At the beginning of the class are located enumerators and structures, followed by class fields, methods, virtual methods, static functions, and static properties.

Interception of functions:

Union uses two types of hooks to intercept functions.
1. MS Detours - a solution from Microsoft, which substitutes a transition to another procedure at the beginning of a function.
2. Patches - an alternative solution from Union. The program defines all instructions in which the function takes part, and
then replaces the addresses and offsets with another routine. This method has the best effect on compatibility with
other modifications. Despite better compatibility, this option has a drawback. Some function calls do not exist explicitly, then only Detours can intercept them. For example, plugins have a set of Standard functions callback (see plugins description). Hook-patches are used in their implementation. Therefore, they should not influence the work or other projects.

* SystemPack.ini has a UseAdaptiveHooks parameter. If you set this parameter to true, then Union will try to replace all Detours on Patches. If the user has problems with the compatibility of Mods and Plugins, then this option will allow you to noticeably
increase their compatibility with each other. This option is an emergency measure. It is necessary to report the developers about the found bugs to fix the problems.

Resource manager:
This is a program that allows you to get up-to-date resource assemblies and information related to developments within the Union project, so that the user can get files and information from one place. The most up-to-date assemblies are always uploaded in the manager products that the user can download, update, install, uninstall, convert to an offline installer, and
distribute it however you like. To prevent problems with the installation, the manager from version 3.2 by default asks
administrator rights. The program uses `[System Drive]\ProgramData\Union\` to store executable files (1),
repositories of downloaded resources (2) and for the content of the Union SDK (3).

1. The manager stores system files in the System folder:
* Ucore32.dll - the core of the installer.
* Updater.exe - the program used for updating manager components. Called directly by the manager.
* Locale.cs - the file containing data with translation of the user interface.
* Root.cs - the file containing the resource hierarchy from the last successful session.
* Errors_Manager.log - the file containing errors that occurred during the execution of the Manager.
* Errors_Updater.log - the file containing errors that occurred during the execution of Updates.
* history_d.cs - the file containing the history of installation directories for displaying hints for the next installations.
* history_f.cs - the file containing the history of installed files to display hints for the next installations.
* Language.cs - the file containing information about the current language
* Theme.cs - the file containing information about the current theme

2. The Storage folder is used to store downloaded resources.

3. SDK folder contains developer tools files, if installed.

The manager is updated automatically if an update is available. In case there is no internet connection, the files you previously downloaded will be available for use. Descriptions and links will be taken from the previous session. The program interface is a collection of directories, resources, descriptions and links. The resource status is determined by colors:

- Green - the object has been downloaded and can be executed.
- Blue - the object is available for download.
- Blue with a red border - the object has recently appeared and can be downloaded.
- Orange - the object can be updated.
- Gray - the object is inaccessible due to the lack of a network.

By using Union, you agree to be bound by the license agreement below:

This license uses the Steam Community conduct[1] and the GMDK[2] principles. You have the right to distribute the end-product of Union without notifying the authors[4]. However, you cannot change the resources related to the Union
yourself. You may not redistribute third party packages that obstruct for a normal operation of Union and additional content to it without a reason[3]. You have the right to file a complaint for the developers[4] of Union if you think that your rights have been violated. The authors of Union and its representatives[5] can: advise you on any questions regarding the Union package and the current licenses; impose sanctions on products if they think may violate the terms of this license and be unacceptable for public distribution. Union does not collect any type of information of the user's PC. Union does not change system parameters that were not described on the main Union page. The license terms are subject to change at any time without prior notice.


[1] https://store.steampowered.com/online_conduct

[2] https://www.worldofgothic.de/modification/?go=modlizenz

[3] Obstruction of normal operation includes any operation related with interference in:

Any Union components without Union authors permission; additional content (plugins, patches, etc.) without coordinating those with convent developers. For example: memory patching; blocking execution; implicit redefinition of user settings, etc. An exception can be considered specific cases, such as: package incompatibility; violation of the performance
of the modification; complete duplication of functionality, etc. Any exceptions must be previously agreed with the authors of the affected content.

[4] Union developers are: @Gratt(gratt.union@gmail.com), @killer-m.

[5] Union representatives are: @Auronen, @Siemekk

Appendixes:

1. The list of Standard callback functions:
a. Registration of external functions;
b. Initialization of the game session;
c. Loading a new game;
d. Loading a save;
e. Level changing;
f. Level changing by a trigger;
g. Saving the game;
h. Pause setting;
i. Pause release;
j. Level rendering;
k. Render the menu.

2. List of sound drivers:
a. MSSA3D.M3D
b. MSSDS3D.M3D
c. MSSDS3DH.M3D
d. MSSDX7.M3D
e. MSSEAX.M3D
f. MSSRSX.M3D
g. MSSSOFT.M3D

3. List of files displayed instead of missing ones:
a. INVALID_SOURCE_FILE.MDL
b. INVALID_SOURCE_FILE.MDM
c. INVALID_SOURCE_FILE.MMB
d. INVALID_SOURCE_FILE.MRM
e. INVALID_SOURCE_FILE.OGG
f. INVALID_SOURCE_FILE.WAV
g. INVALID_SOURCE_FILE-C.TEX

Community Rules:
- do not appropriate other people's products, and do not distribute them without the permission of the authors (if this is not the direct task of the product);
- do not interfere with other users or developers for no apparent reason;
- do not create conflict situations or tension in the community;
- do not create virus programs.
- When distributing the product, comply with the laws of the country.

Each member of the community has equal rights and can freely create and distribute their own products for Gothic and Gothic II games using the Union and Union SDK tools, if the products do not contradict the community rules. In case of violation of the rules, the product may be blocked at the Union core level.

Links:
Executable files Gothic1 1.08k: https://www.worldofgothic.de/dl/download_6.htm
Executable files Gothic2 1.30: https://www.worldofgothic.de/dl/download_278.htm
Executable files Gothic2 2.6 rev2: https://www.worldofgothic.de/dl/download_278.htm
LAA (Large Address Aware) Hack: https://www.worldofgothic.de/dl/download_552.htm
Large Address Aware (description): https://docs.microsoft.com/en-us/windows/win32/memory/4-gigabyte-tuning
Description of the patch file format: https://worldofplayers.ru/threads/42178/
Original resource Union on WOP.RU: https://worldofplayers.ru/threads/40376/
Union resource on WOP.DE: https://forum.worldofplayers.de/forum/threads/1571097-Release-Gothic-½-Union-Ressourcen-Manager-(DEU-ENG-RUS)