Book Review: The Game Asset Pipeline

The Game Asset Pipeline by Ben Carter is interesting in that it is probably one of the only books out there on asset pipelines for games. The books is broken into too major parts: high level asset management, and low level processing details. Overall, the book handles these issues effectively and with some interesting insight into pipeline work in general.

Carter suggests a number of useful tips for those working on a asset pipeline. The output of all systems should be a custom intermediate file format. This will allow the intermediate files to be processed and transformed into final assets without the need to re-export all of the assets from the 3D applications (or other tools). By having them as simple text files, it will also allow any additional last minute processes, without the huge overhead of having to re-export from multiple tools (especially since the end format is likely to change). Since ideally the format should be as close as possible to what will actually be contained in memory while the game is running, the intermediate format can allow for processing into the final assets, without having to constantly re-export the same assets everytime the end file format changes (and it likely will a lot, especially at the end). Also the tools should do as much preprocessing as possible to ensure speed at game runtime.

The eternal debate of text vs binary files is also brought up in the book. It is recommended that text files are used (although binary can be brought in later if necessary), but more so than just using text files, is the idea of text files with an implicit structure. If a text file has no structure such as:

Fred
Solider
Aggressive
1000

It becomes very difficult to add things in and the parser will likely be confusing since it is expecting things in a certain order. A much better approach is to have the structure built into the file such as:

CharacterName = Fred
CharacterType = Soldier
AIType = Aggresive
HitPoints = 1000

With that kind of approach it is much easier to see what the structure is, and with a parser of this type, things can be added, reordered or removed without issue (such as if the AIType does not have to be set or whatever). A still better approach to the plain text file is some sort of structured file format like XML or JSON. While I like JSON for most things, as it closely resembles data structures, it is important to note that once you start getting into very large files, it becomes trickier to navigate all the closing brackets in JSON, whereas XML gives all the end tags which is somewhat easier to navigate. Both are entirely capable of getting the job done, but it is best to look at what languages you are using and what kind of support they have for parsing (C# and XML go hand in hand, as do more web languages like Javascript with JSON).

Carter continues with some of the high level asset management and that it is different than source control (although there are some basic similarities, the size of the assets, the number of binary files, etc.) makes for different challenges in an asset management system. The most important aspect of an asset management system are turnaround time, metadata about the assets, versioning and handling broken data. The book looks at a number of subversion systems including SVN, Perforce, CVS and Alienbrain. The use of databases (like MySQL or Oracle) are not usually the best options unless the database is setup to allow for huge file sizes.

The concepts of a client/server architecture for the asset management system is essential, and should support locking and unlocking files, versioning of files, preview functionality, showing modifications and some sort of caching system to improve performance.

The rest of the book is on more low level details such as texture, geometry, environment processing, asset dependency analysis and final data output. It is interesting, but is more platform specific than the high level concepts on asset management in the earlier chapters. In some ways, I would have preferred more of an expansion on the asset pipeline aspects and less on the low-level details (perhaps splitting up the two major topics into two separate books and get everyone to buy both :P)

Overall a good book on many of the concepts of the asset pipeline for those not familar, and is a good resource on a topic with not a lot of literature on it.

Bye for now,
Michael Hubbard
http://michaelhubbard.ca