If you don’t know already, Tundra is an open-source build system I’ve written and have been maintaining for a couple of years now. The source is available on github (https://github.com/deplinenoise/tundra).
In maintaining the tool, I’ve learned some lessons and rethought some of my initial design decisions. This post is about how I want to evolve the tool with those lessons in mind.
What has worked well
Tundra is up to version 1.2 now and is relatively stable. It’s definitely not all bad. The current tool is fast and portable. Its used in a lot of projects by friends and other random people on Twitter (you know who you are!) Specifically, I think the following design decisions have worked out well:
Multi-threaded build engine in C
The build engine itself has proven to be robust and relatively bug-free since early on. It does parallel dependency and header scanning, so incremental builds run really quickly. It uses simple memory management techniques (just flat arrays mostly) which has turned out to work really well.
Using Lua for configuration
Lua is remarkably fast and flexible. I didn’t expect the front-end to run as quickly as it does, given that it does a ton of string processing and data collection before the actual build engine runs. It will commonly dish out a large DAG for the backend in under 100 ms, which is a great achievement for a interpreted script language that has to hit disk to find file names and other things.
Hard separation between configuration and building
The current design emphasizes the two distinct phases in the tool: configuration and building. Once the Lua front-end has produced a DAG, it is no longer part of the runtime. This makes the build faster, as the C side can run unimpeded by script language performance and multi-threading issues.
Header scanning over implicit tracking
Some new-wave build tools (tup, ninja) depend on tool-specific options or OS/file system-level hooking to automatically track implicit dependencies. This can be convenient when it is supported, but significantly limits the build tool in what platforms it can run on and what tools you can use with it. Tundra has successfully been able to accommodate quirky Amiga toolsets, cross-compilation scenarios (Tundra itself is cross-compiled!) and other non-mainstream scenarios such as custom assemblers by scanning the file system for #include information and keeping a cache of such discovered dependencies.
What could be improved
There are some things I’d really like to fix with version 2.0:
Cleaning up stale build products in the file system
Because the DAG is generated fresh every time, the front-end doesn’t generate a complete DAG including all possible configurations and variants. This means a debug build will know nothing of the release build targets. If the build engine knew about all the possible outputs, it could safely deduce that certain output files are now stale and can be deleted. This keeps the file system nice and tidy.
Targeting the build engine with other types of configuration input
The build engine doesn’t really care about the Lua configuration, but because the tool is a monolithic executable with a single front-end, it’s awkward to try to use it to build something completely unrelated to code. For example, if you’d like to use the build engine to build game assets, it requires a significant amount of Lua scripting within the existing framework (which is code-oriented) to try to express those build rules. If the build engine could be configured in more ways, it would be more useful.
There is some integration with Visual Studio and Xcode in the current tool, but it is relatively limited. It would be cool if the current front-end could be made to generate project files that integrate with these tools more easily.
Here’s how I’ve been thinking about fixing some of this, while keeping the best features around.
Completely divorce configuration and build engine
It would be cool to split the tool in three pieces, analogous to how the gcc compiler driver is really multiple executables:
- One driver executable (the one you invoke – tundra.exe). This is a light-weight front-end binary that first launches a configuration executable (if needed, see below) and then kicks the build engine.
- One configuration executable (tundra-luafrontend.exe). This is a program that encapsulates the task of reading configuration data and producing a DAG as output.
- One build engine executable (tundra-buildengine.exe). This program just reads DAG data from a file and executes the build as fast as possible.
This design enables some nice benefits:
- The front-end binary only needs to run when needed (the build files have changed, glob queries would change results, that sort of thing.) Most builds use an identical input DAG. Therefore the top-level build can just skip running the front-end program entirely and use the cached DAG from the previous run. This will shave of precious milliseconds from every incremental build.
- The build engine can be targeted directly using a custom front-end tool. This means you can plug in some other DAG generator to build your game assets or run other build rules that are not easily expressible in the Lua front-end. Maybe you already have existing build system data (Visual Studio projects?) that you want to run on the Tundra back-end.
- Maintenance becomes simpler, as changes to the front-end can be tested in isolation without involving the back-end, and vice versa.
Produce complete DAG data – Clean up file system
If the build-engine sees complete DAG data (all configurations, variants, platforms and so on) it can clean up the file system state before it starts building. The reason this isn’t done today is performance; we don’t want the Lua front-end to generate 8x as many DAG nodes if we’re only going to build the debug config anyway. With the mandatory caching outlined above, this problem goes away–the data will only be regenerated when you change the build files anyway.
What would you like to see in Tundra 2.0? Drop me a line and let me know!