Use Deep Learning Tool inside Unreal

In this post I would like to introduce how to setup Unreal as a deep reinforcement learning environment. Before going on please get your deep learning environment ready and install TensorFlow or Theano. Our environment is based on the amazing Unreal Python Engine created by 20tab:
https://github.com/20tab/UnrealEnginePython

I've successfully installed on Ubuntu, Windows and Mac. Installing on Windows & Mac are relatively easy so let's start with Ubuntu first.

Ubuntu

1. Installing Unreal

First you need to register an account on EpicGame website. For linux we need to compile from source code.  Make sure you get granted for downloading from github then execute the commands below:

# Get authorized on github by EpicGames first. 
# Can use -b to specify UE4 version. Ex: git clone -b 4.16
git clone https://github.com/EpicGames/UnrealEngine.git
cd UnrealEngine
./Setup.sh
./GenerateProjectFiles.sh
make
#....wait for 30+ mins....
# Run UE4 editor
./Engine/Binaries/Linux/UE4Editor

1.1 Unreal Terminology

There are some must-know terminologies of Unreal:

LEVEL: A gaming region, saved as (.umap) so it's also called Map	ACTOR: Any movable object in a level
PAWN: Subclass of Actor controlled by player or AI (NPC)	BLUEPRINT: Visual game scripting system

More terminologies can be found here.

1.2 Learning Unreal Basics

To create your own test environment you need to learn how to manipulate objects (actors) and write scripts. There are lots of tutorials on Unreal website and I suggest to read the following first:

a. Level Designer Quick Start

b. Unreal Engine 4 Tutorial for Beginners: Getting Started

c. Blueprints Quick Start Guide


NOTE: Unfortunately there is no Epic Games launcher for Linux, so we need to download free examples using Windows or Mac and then upload to Linux.

2. Installing UnrealPythonEngine

Now we need to install the UnrealPythonEngine plugin from
https://github.com/20tab/UnrealEnginePython
By default Unreal will load and compile any plugs in /Plugins folder under the project directory. Note we need to create C++ project, not Blueprint:

The installation steps are listed as below:

• Create a new Unreal C++ project and close the editor once the project is fully started
• Go to the just created project directory and create the Plugins folder
• move to the Plugins folder and clone the UnrealPythonEngine plugin:

git clone https://github.com/20tab/UnrealEnginePython

Before we re-open Unreal project and compile the plugin, we need to setup Python engine first.

2.1 Choose Python 2.7 or Python 3.5+

The default Python engine is 3.6. To change it we need to edit the build file:

Plugins/UnrealEnginePython/Source/UnrealEnginePython/UnrealEnginePython.Build.cs

And add your Python installation folder into pythonHome as below:

public class UnrealEnginePython : ModuleRules
{ 
  
    // leave this string as empty for triggering auto-discovery of python installations...
    private string pythonHome = "/usr/include/python2.7;/usr/lib/x86_64-linux-gnu/libpython2.7.so";
    ......

Reopen the project and start compiling the plugin, a "not compatible" warning message may be shown:

Ignore the warning and click "No", and you will see the next dialog

Click "Yes"

*Compiling error for UnrealPythonEngine

If you are using clang++-3.9 and older Unreal version (like 4.15) to compile UnrealPythonEngine, you will encounter the following error:

ThirdParty/FBX/2016.1.1/include/fbxsdk/core/fbxproperty.h:1242:70: error: binding dereferenced null pointer to reference has undefined behavior [-Werror,-Wnull-dereference] return StaticInit(pObject, pName, FbxGetDataTypeFromEnum(FbxTypeOf(*((FbxReference*)0))), pValue, pForceSet, pFlags); 

To fix it, change your clang++ version to 3.8 or lower:

sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.8 100
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.8 100

2.2 Open Python Console

Now we are ready to run some Python scripts. Make sure your deep learning tool (TensorFlow or Theano) is accessiable for the UE4Editor. If VirtualEnv is used, activate the environment before running UE4, for example:

/data/UnrealEngine-4.17$ source ~/theano/bin/activate
(theano) /data/UnrealEngine-4.17$ ./Engine/Binaries/Linux/UE4Editor

Open the Unreal project and select Window -> Developer Tool -> Python Console

Import your Deep Learning framework, then you are ready to go!

Windows & Mac

For Windows and Mac, just download the binary releases from Github of UnrealPythonEngine and unzip it in your project's Plugins folder. Make sure you download the right Python version.