Continuous Integration and Unit Tests in Unity

Continuous Integration, and especially Unit Tests have been a subject of major confusion in the Unity community. For our current project, we've managed to set up a Continuous Integration environment with automated Unit Testing. How did we do that? What hurdles did we face? What are the practicalities of doing this? Read on!

The Problems

Limitations

Unit Testing in the Unity environment is limited at its core as most features of a game require user intervention. As there are some solutions, even one by unity, that can work from pipeline for every build but they work from inside Unity itself and require setting up scenes and such. Generally, with a generic Unit Testing environment, you can't test the Unity-specific functionality. But assuming you did good on encapsulation and separation of functionality from the UI, your actual code should be disconnected from the Unity stuff.

Dependencies

The other problem is that you can't really run anything which is found in the UnityEngine namespace, including Debug.Log. Those lines will probably be present in your code and you have to have a way to test them, but this is solvable.

The Solutions

Visual Studio

First of all, we've created a new Test Project in Visual Studio Express (which is free to use!). To this solution we added the project "Assembly-CSharp-vs.csproj" and connected it by dependency (reference) to the main test project. Now we can test our code as it's already referenced to UnityEngine and the actual csproj will be automatically updated when we edit it. We also used the built in testing tools of Visual Studio for simplicity.

Perforce

We've set up a Perforce system on Assembla (This is a bit tricky, i'll write more about it in the next post.) and set up a Jenkins server to monitor the perforce depot (it's like a repository for SVN) and run the tests in our project. You can do it with any combination of Source Control, hosting service (or your own repository) and Continuous Integration server.

Workarounds

This is the interesting part. As this whole setting can't work on its own flawlessly, we had to implement some simple workarounds to keep it running.

Setting Source Control on the build machine

Usually removed from the Source Control environment, this time we had to include the visual studio csproj files in the Source Control system. These files are automatically recreated by unity every time you add or remove a file in the editor. This needs to be passed on to the tester so it updates the project data with the relevant files.

Hiding Unity functionality

If you try running the tests you'll probably receive this nonsensical error:

Test method ############## threw exception: 

System.Security.SecurityException: ECall methods must be packaged into a system module.

This actually mean that you tried running something from UnityEngine in VisualStudio. Usually by following the trace you'll be able to get to the exact method it occurs in. Also, it seems to mostly come from rogue Debug.Log calls (although some of calls seem to originate from PlayerPrefs or WWW).s

One options is to remove them, but this can be hazardous if you want to keep logs at runtime. Our solution had 2 parts to it:

1) Replace Debug.Log with a modified log4net module. The only modification we added was to print Debug.Log whenever you log anything with the log4net module (also Debug.LogWarning and Debug.LogError when applicable). This means we're using log.Debug(string) instead of Debug.Log(string). This also helps us to send the logs somewhere safe or keep them in a file when necessary and not just print them in the editor.

2) Now that all the Debug.Log lines are moved into one file, we can hide them. The ideal solution would be to find a precompiled define statement which will only be compiled in unity (the editor or any other build platform, but not in Visual Studio). So we did just that. We added a #IN_UNITY precompiler header which only unity can read so it's false by default. In order to make Unity know about it we put two files in the Asset folder (the file names and their location is important) and called them smcs.rsp and gmcs.rsp. For more information about what are those files, look here (Ctrl+F for smcs). Then we put this line into both files: 

-define:IN_UNITY

And that's it! Now all those line will work from unity but not from Visual Studio.

Conclusion

The next post will include some information on how to make Jenkins work with Perforce in Assemble and why we couldn't use TeamCity. Hope you can make use of all this information. Please comment on this post if you have any further inquiries!