Small team workflow for a source build of Unreal, aka “homemade UGS”

For large teams using a source build of Unreal, UnrealGameSync is a common method for distributing editor builds to non-coders, but it’s not a great fit for small indie teams. So what’s the alternative?

I’m working on a project at the moment for a client with a small team. We’re using a source build of Unreal, with Git for source control. This immediately raises a question about how we share builds with non-coders on the team: if we were using a binary build of the engine we could just shove the Binaries folder into LFS, but that’s not enough on a source build. On larger teams I’ve used UnrealGameSync for this, but it’s not very appealing to small indie teams:

• it requires perforce, which needs paid licenses and a server
• it needs MySQL and IIS for metadata
• it’s Windows only

If you’re a small team that’s come to Unreal from Unity after The Great Riccitiello Foot-shoot, I think expecting you to get set up with perforce, MySQL and IIS is serious overkill – especially because this isn’t an issue at all when using Unity. Unity has a C# compiler builtin, so your artists make their own builds without even realising they’re doing it. You could try installing Visual Studio on their machines and expect them to use it but this is likely to result in a coloured pencil in the eye.

UGS is great for larger teams because it keeps source control and binary builds in sync, and allows devs to easily roll back to earlier versions if the build breaks. That’s important for large teams because it takes time to even work out what broke the build, never mind how to fix it – and if your many artists have to down tools in the meantime that’s a big waste of money. But none of this applies to small teams, so in that sense they don’t need most of what UGS has to offer.

So if we aren’t going to use UGS, what’s the alternative? Also, UGS is just responsible for distributing the builds, how are they created in the first place? We’ll get back to this, but first let’s talk about engine associations.

Engine Association

The Engine Association is the model train group my Dad goes to on Sundays. It’s also a field in the .uproject file. When you right-click a .uproject and select ‘switch engine version’, this is what’s updated. The EngineAssociation field can contain 3 types of values:

1. A release version number, e.g. “5.3” referring to an ‘official’ binary distribution of the engine, downloaded from the launcher
2. A GUID referring to a source version of the engine. These GUIDs are generated randomly by the Setup script you run when you first sync the engine from git
3. Blank (“”). If a .uproject has a blank EngineAssociation, the loader (UnrealEditorServices) will run the first engine it finds, working upwards through the directory tree from the folder containing the .uproject

If you’re using a source build of the engine, you’re using method 2 – a GUID. The problem with this is because the GUIDs are randomly generated, they don’t match across different coders’ machines. The GUIDs are stored in the registry on Windows, and in a file on other platforms, so forcing them to be the same on all machines is slightly fiddly.

A blank EngineAssociation (method 3) is the most straightforward way of forcing the association between a project and a source build of the engine, but it does require the slightly strange setup of placing the project folder inside the engine source folder, as in:

c:\MyUnrealEngine - contains the custom build of the engine
c:\MyUnrealEngine\MyCoolGame - contains the MyCoolGame project

If you structure the project this way and set the EngineAssociation blank, you can share the tree with other coders and it’ll build fine. So that fixes it for coders, but we still can’t share the build with artists.

BuildEditorAndTools.xml

There’s a BuildGraph script in Engine/Build/Graph/Examples called BuildEditorAndTools.xml. This is what’s used to create builds of the editor and upload them to perforce for UGS. Thankfully it’s split into subtasks, so we can use it to do just the first part. Note the script assumes it’s running on Windows, so if you’re on another platform you’ll need to look out for the following:

• it has a command line option to change the target platform, but it’s mostly ignored. You could implement all the missing bits, or just replace all instances of ‘Win64’ with e.g., ‘Mac’
• it tries to include an Incredibuild-enabled version of ShaderCompileWorker as a dependency, which only exists on Windows. You can work around this by commenting out the line mentioning XGEControlWorker.exe

So if we had an engine and project as described in the last section, this is how we might run it from a terminal in the c:\MyUnrealEngine folder:

Engine\Build\BatchFiles\RunUAT.bat BuildGraph -Script=Engine\Build\Graph\Examples\BuildEditorAndTools.xml -Target=Copy to Staging Directory -set:EditorTarget=MyCoolGameEditor

This will create a folder called c:\MyUnrealEngine\LocalBuilds\MyCoolGameEditorBinaries. Inside that folder will be two more, Engine and MyCoolGame. These contain all the binary dependencies that make up the editor and the project. So if we just zip that folder, chuck it on a shared drive, and write a script that downloads it and unzips it into the right places that’ll get everyone up and running, right? Nearly… but no: we need to talk about build IDs.

Build IDs

The editor has a build ID (in UnrealEditor.version), and each module has one too (in its respective UnrealEditor.modules). If a module’s ID doesn’t match the editor’s, the editor will ask if you want to rebuild that module, and refuse to load if you don’t.

The build ID is randomly generated and assigned to the editor (in the UnrealEditor.version file) every time you build. The same ID is assigned to every rebuilt module’s UnrealEditor.modules file too. The fun part is if you don’t do a full clean build of all modules, the modules you didn’t need to rebuild will be left with the old build ID, which guarantees an editor that won’t load. We can handle this by writing our own build ID to the editor and all modules.

Turns out there’s a better way of handling the build ID: if you add a BuildId value to Engine/Build/Build.version, during the build (even a partial build) it will be propagated to all modules. Thanks to Mark for the tip!

We Have UGS At Home

Originally I’d intended for the build script to be a batch file/shell script, when I thought it just needed to build the editor and zip the results. Once I realised we needed to open files and change contents I switched to Python (since Unreal comes with a version of Python, so everyone on the team has it available).

The python script I wrote doesn’t run exactly the BuildGraph command I mentioned above. That example uses the “Copy to Staging Directory” task, which makes the build and copies it to the LocalBuilds folder. I found on Windows the 30gb build contained 29.5gb of PDBs, enough that it took a notable amount of time just to copy to the staging folder. Debug information will be useful much later on, but right now it’s just a huge waste of time and space. So instead I run the “Tag Output Files” task, which builds the editor and writes a manifest to Engine/Saved/BuildGraph/Tag Output Files/Tag-OutputFiles.xml. The “Copy to Staging Directory” task parses this manifest and copies every file listed to LocalBuilds. My script does this manually, ignoring the PDBs. Then it zips the tree, naming the zip after the current git commit ID, uploads it to the cloud, and posts an update to discord. The download script gets the latest zip from the cloud and unzips it into the right place, with some housekeeping to prevent unnecessary downloads.

Note: if you’re making zips on non-Windows platforms, make sure your archiver preserves file mode bits, or your executables won’t be executable when they’re unzipped

The scripts provide a fraction of what UGS offers, but they also took a fraction of the time and effort to setup. Right now they do everything we need them to, but it’s simple enough to build in more features later if we need them. And we didn’t have to switch to perforce!