Book Review: The Game Asset Pipeline

The Game Asset Pipeline by Ben Carter is interesting in that it is probably one of the only books out there on asset pipelines for games. The books is broken into too major parts: high level asset management, and low level processing details. Overall, the book handles these issues effectively and with some interesting insight into pipeline work in general.

Carter suggests a number of useful tips for those working on a asset pipeline. The output of all systems should be a custom intermediate file format. This will allow the intermediate files to be processed and transformed into final assets without the need to re-export all of the assets from the 3D applications (or other tools). By having them as simple text files, it will also allow any additional last minute processes, without the huge overhead of having to re-export from multiple tools (especially since the end format is likely to change). Since ideally the format should be as close as possible to what will actually be contained in memory while the game is running, the intermediate format can allow for processing into the final assets, without having to constantly re-export the same assets everytime the end file format changes (and it likely will a lot, especially at the end). Also the tools should do as much preprocessing as possible to ensure speed at game runtime.

The eternal debate of text vs binary files is also brought up in the book. It is recommended that text files are used (although binary can be brought in later if necessary), but more so than just using text files, is the idea of text files with an implicit structure. If a text file has no structure such as:

Fred
Solider
Aggressive
1000

It becomes very difficult to add things in and the parser will likely be confusing since it is expecting things in a certain order. A much better approach is to have the structure built into the file such as:

CharacterName = Fred
CharacterType = Soldier
AIType = Aggresive
HitPoints = 1000

With that kind of approach it is much easier to see what the structure is, and with a parser of this type, things can be added, reordered or removed without issue (such as if the AIType does not have to be set or whatever). A still better approach to the plain text file is some sort of structured file format like XML or JSON. While I like JSON for most things, as it closely resembles data structures, it is important to note that once you start getting into very large files, it becomes trickier to navigate all the closing brackets in JSON, whereas XML gives all the end tags which is somewhat easier to navigate. Both are entirely capable of getting the job done, but it is best to look at what languages you are using and what kind of support they have for parsing (C# and XML go hand in hand, as do more web languages like Javascript with JSON).

Carter continues with some of the high level asset management and that it is different than source control (although there are some basic similarities, the size of the assets, the number of binary files, etc.) makes for different challenges in an asset management system. The most important aspect of an asset management system are turnaround time, metadata about the assets, versioning and handling broken data. The book looks at a number of subversion systems including SVN, Perforce, CVS and Alienbrain. The use of databases (like MySQL or Oracle) are not usually the best options unless the database is setup to allow for huge file sizes.

The concepts of a client/server architecture for the asset management system is essential, and should support locking and unlocking files, versioning of files, preview functionality, showing modifications and some sort of caching system to improve performance.

The rest of the book is on more low level details such as texture, geometry, environment processing, asset dependency analysis and final data output. It is interesting, but is more platform specific than the high level concepts on asset management in the earlier chapters. In some ways, I would have preferred more of an expansion on the asset pipeline aspects and less on the low-level details (perhaps splitting up the two major topics into two separate books and get everyone to buy both :P)

Overall a good book on many of the concepts of the asset pipeline for those not familar, and is a good resource on a topic with not a lot of literature on it.

Bye for now,
Michael Hubbard
http://michaelhubbard.ca

Book Review: Game Engine Toolset Development

Not so much a programming book as much as it is a look into the entire world of tools development, Game Engine Toolset Development by Graham Wihlidal is an interesting read for those wanting to improve the production pipeline or tool creation. Now that I my current job is focused more on the technical artist/programmer instead of a team lead game programmer, I figure I should take a bit more time on the technical artist and tools building aspects (which are for me back to basics). The book focuses on a wide range of topics, and focuses more so on implementation details than tool design.

The book begin largely on the politics of tools and maintenance and different stakeholders in the tools, as well as some examples of different environments in which tools are created and maintained. There are a few examples of professional studios, who focus on elaborate GUI systems, and WYSISYG (what you see is what you get) world editors.

From there, the book has a chapter on why to create tools with C# with .Net (which is the language and framework used throughout the book). While I agree that .Net can be a great and quick framework for developing solid tools quickly, I would recommend sticking with the language that is closest to the game engine or 3D application whenever possible (since it is often the case that some parts (or the entire tool) can be merged into the game code, and it is better to not have too many languages in any production environment (if possible)).

There is a lot of focus on coding standards, design and documentation which is extremely important for tools, since they are often written specifically for one project, and can be very difficult to port to other projects if the design and standards are poor. Plus, the fact that many tools are written once and then the programmer moves on to something else, it is easier to forget why the code was written a certain way when come back to it in six months. Having a good design and (up-to-date) documentation is very important if no one is going to be looking at the code until the next project, or something breaks. There is some discussion of code metrics, UI design, using NUnit for unit testing, Microsoft coding conventions and using FxCop to enforce coding policies (I also like StyleCop for coding standards). The coding documentation is with NDoc, which I am not as familiar with, but will have to check out sometime (I currently like Doxygen since it is useful for a lot of languages).

All in all, a good book that deals a lot more with the entire job of tools development, which is somewhat unique in terms of computer books. I realize that for some people (and companies) the need for tools is not quite so apparent. The best analogy I can think of is that tools are the "scaffolding" in which you can build up any project. They solve the simple problems, help tackle the larger problems, glue together what needs to be connected, and improve the productivity of the entire department. Without scaffolding tools everyone would be trying to add another floor to a building by hanging out a window, and if that is the case, you are not going to be able to build quite so high...

Keep reaching for the top,
Michael Hubbard
http://michaelhubbard.ca