Pages

Tuesday, September 28, 2010

Game performance and optimization

The question of performance and what can benefit from additional work optimizing code (or art) is a very complex problem. To grossly oversimplify the questions of performance, it is worth looking at a few of the terms used to describe what involved in the actual measure of performance. Usually when talking with people about a game's performance in simplistic terms they will focus on fps (frames-per-second) but there are many other considerations memory usage, texture memory usage, cpu usage, gpu usage, network usage (if applicable) or things like load time, amount of loads, and more. To measure any of these it is worth focusing on testing different types of components, and the platform and/or engine and getting minimum specs for what you need (size of art assets, number of triangles, animations, size of textures, number of lights, number of postprocess fx, complexity of shaders, etc.), the list goes on and on.

One thing that is often talked about CPU bound vs GPU bound, meaning that one of these systems is a bottleneck for the other.

CPU bound could be for a lot of processing, physics calculations (on the CPU) or a lot of other complex calculations (searching or sorting large amounts of data or complex AI calculations like pathfinding).

GPU bound if a lot of polygons, high resolution textures, or dealing with a lot complex shaders (often with multiple lights and shadows).

The best approach is to find the first largest bottleneck and try to minimize it, then the next largest and so on. While it is sometimes worthwhile to look at the code and try and optimize it (things like replacing divides, multiplications ect. with bitshifts and attempting to precalculate as much as possible before actually hitting runtime), this won't solve the problem if the game falls over from huge art assets that the hardware does not easily support. It is often considered that premature optimization is a huge mistake, since it is often difficult to know if you are optimizing the right part of the code (and often the code becomes significantly more complex and more difficult to maintain if it is optimized). It is still worthwhile to make as efficient code as possible while writing it (considering order of magnitude algorthims (big O notation)), but if you don't actually test with real data, you may be wasting huge amounts of time optomizing the wrong thing.

In terms of performance and optimizations, I almost always come back to an example that a colleague of mine gave me that he saw on a previous project. The project was on a 3D handheld and the team wanted to add a depth of field postprocess render fx to the game, but were worried that the performance hit would be too expensive. They decided that the look was so cool, they had to try it and see what it would cost. What they found was that by adding this fx they actually got better performance! On this particular 3D handheld both the CPU and GPU shared the same bus, so by adding this postprocess fx the bus was freed up to handle a lot of CPU processing, while the GPU was dealing with this more complex rendering. Not an easy thing to estimate or plan for :P

There are a few useful tools in helping debug performance, PerfStudio for AMD http://developer.amd.com/gpu/PerfStudio/Pages/default.aspx and PerfHUD for NVIDIA http://developer.nvidia.com/object/nvperfhud_home.html are great tools for seeing the low level API calls to the graphics card as well as debugging tools for texture visualization and overrides which are great for shader debugging, checking mipmap level etc.

One of the things that really impressed me in the UDK was the use of performance and memory built right into the editor, as well as things like mipmap levels, shader complexity, that are great ways of helping artists debug scenes and texture sizes. Unity3D offers a few tools (like the stats window and profiler) to help see where some of the memory is being used, but I haven't seen it at the level of UDK anywhere (and will still need to look at XNA, but without an editor, it may be difficult to provide any of these kinds of tools).

In the end, it all comes down to trade offs, and what you can live with for performance for your game, usually at the expense of something else, and hopefully not too much of the visual look. It is worthwhile to develop some tests that check for significant changes in memory usage, video memory usage, frame rate, etc. on every commit (or at least everyday) to help catch these memory issues as soon as possible (especially on a large project).

Best of luck optimizing,
Michael Hubbard
http://michaelhubbard.ca

Book Review: Lifehacker 88 Tech Tricks to Turbocharge Your Day

With all the books on agile methodologies and planning and managing time I thought I would look at trying to improve some of the regular aspects of my day-to-day activities to try and find "more time". There are still 24 hours in a day, but it is helpful to realize how and where you are spending time on things, and how to improve your use of that time. I enjoy the website http://lifehacker.com/ and finished up reading Lifehacker 88 Tech Tricks to Turbocharge Your Day by Gina Trapani. Overall, the tips are quite interesting and range from the simple (like focusing your attention and emailing yourself things to do) to the more complex (such as setting up a personal wiki server).

I am a big fan of automating things, and love a really slick and complete pipeline. But sometimes the trick isn't to know how to automate something, it is what to automate. It would be easy to try and say "everything", but you really need to start somewhere.

The Lifehacker book really got me thinking about examining my day and thinking what can be improved, and what could be automated. This almost requires an outsider perspective to really examine the details of how tasks are accomplished and where time is spent in a given day. It can be from the complex (like how to push the game & server to another environment), to the trivial (every time you start your computer, do you always open the same three programs). While the Lifehacker book and website provide details on good tips (like carrying around a flash drive of your applications, "firewall" your attention to prevent distractions through a few sets of tasks, or tips for tuning your computer or search habits) the real benefit to this kind of book is taking some time to reflect on your own habits and think about improvements.

I will be keeping track of where I spend my time "in the life of a game programmer/technical artist/team lead and will look into where my time goes in a follow up blog post and try to examine how my time can be improved to increase overall productivity (and possibly sanity :P).

Until then,
Michael Hubbard
http://michaelhubbard.ca

Sunday, September 26, 2010

Book Review: C++ In Action Industrial Strength Programming Techniques

With all the posts on Agile methodologies I decided to take a break and follow up on more programming aspects. C++ In Action Industrial-Strength Programming by Bartosz Milewski is as the preface states "... is not a language reference or a collection of clever tricks and patterns. This book is about programming." The book offers some useful insights into those programming coming from a C or Java background, but ramps up with some important theory of programming, with useful examples for function pointer routers, techniques for data encapsulation with templates and the windows API.

One of the ideas that struck a chord with me was the case against "defensive programming". Often programmers want to be as careful as possible, but by writing very defensive code, they end up writing nice little places for nasty bugs to hide. Milewski recommends using asserts on parameter input instead of attempting to handle invalid input at all, which in turn prevents bugs from "covering their tracks". He goes on to recommend that programmers do not try and write code that works no matter what. By attempting to write such defensive programming the code becomes more obscure and prone to errors. Instead, programmers should take an approach of validating parameters (and using asserts) and not attempting to handle invalid data by passing on more invalid/garbage data.

I really can get behind this concept, and often will try and write code that does a lot of error checking with quick returns. I find that defensive programming often also goes along with lots of nested if statements and tree logic that ends up complicating the code. The programmer wants to be defensive but does not check the parameters ahead of time, and instead complicates the overall structure of the code. Here are two simple examples of stuff I see all the time. The "Foo" example (below) is not uncommon to see, even though the "Bar" is easier to read and does the exact same thing. By focusing on validating parameters you end up with shorter, cleaner code which is much easier to understand and debug.

// Foo function (with valid inputs do the foo).
int Foo(Object obj, int param1, int param2, bool param3)
{
if(obj != NULL)
{
if(param1 < param2)
{
if(param3)
{
// Do the foo.
return 0;
}
else
{
return -1;
}
}
else
{
return -1;
}
}
else
{
return -1;
}
}

// Bar function (with valid inputs do the foo).
int Bar(Object obj, int param1, int param2, bool param3)
{
if(obj == NULL || param1 > param2 || !param3)
{
return -1;
}

// Do the foo.
return 0;
}

The book is also available at http://www.relisoft.com/book/index.htm and while it does deal with a lot of different elements (like windows specifics, team work and porting code) the end result is an interesting read with some good looks on program transformation and improved structure.

I hope to get more real code examples up soon and will hopefully have some interesting content of my own. But until then I remain...
Michael Hubbard
http://michaelhubbard.ca

Thursday, September 16, 2010

Book Review: Agile Game Development With Scrum

Staying on this Agile game development interest, I was at the local book store the other day and ran across the book Agile Game Development With Scrum by Clinton Keith and thought, "how appropriate". Needless to say I was intrigued about the aspects of applying Agile directly to game development and immediately started reading it.

One of the elements that Keith keeps repeating throughout the book is that there are "no silver bullets" and adopting Agile won't save a team already on a death march schedule, nor will it allow a team to handle an impossible schedule. What Keith proposes instead, is that Agile will give a team a framework to learn about scheduling issues, current velocity and burn down charts as well as processes to detect potential issues with the schedule or design, far earlier in the project.

The main point that I found crucial to success, was the idea of cross domain teams. A couple artists, level designers, game programmers, server programmers etc. would form a single team responsible for part of the game, and would handle all aspects of that feature, while iterating over the different stages every two week sprint. Adopting this team structure, I feel, would be a huge boon to any game dev work place as the slowest part of any game is always (always!) cross team integration. Here are two examples of what could happen in a non-cross domain team:

1. An artist creates an asset and has no developer support to test it. Without integration they go on to make multiple assets the same way spending weeks (or months) of effort, only to find out later they need to modify all their assets now that dev has begun testing them.

2. Server team creates APIs for client, but no client team is available to test them. Server developers go off and create new APIs. Once the client begins testing the APIs they find they need to add additional parameters and/or bug fixes. This requires days of rewrites from a different developer, since the original author is working on another critical section and is not available to update the code.

I see these kinds of issues happening over and over again in large scale isolated teams, where individuals begin to make decisions that should require cross domain knowledge, but instead move forward without that knowledge, and often wastes significant time when having to fix their mistakes. This amount of rework can also result in a crunch time later to try and catch up on the previous wasted effort.

One of the other important things that Keith mentions is the issue of burnout during a crunch, and estimates that only two to three weeks actually show any improved velocity. Once a month (or more) of crunch has been put into effect the overall velocity is actually worse than that of a non-crunch velocity! People will get tired, frustrated and upset with extended overtime, and Keith points to the http://www.igda.org/quality-life-white-paper-info that measures the quality of life of game developers across multiple industries, which is a good read I will probably comment more on in a later post.

The book also goes on to describe a number of key aspects regarding Agile practices (running a scrum, user stories, minimizing up front documentation, continuous integration and working builds every sprint) but also concepts for implementing and introducing Agile and scrum to the workplace, shifting emphasis to a team responsibilities for setting schedules and working with Agile for Art, QA and Producers.

Overall the book offered some valuable insights and interesting experiences. I would love try and implement more Agile processes at my current workplace, but we shall see how strong the culture is resistant to change. There are a number of jokes about trying to change culture like Angry Monkeys if you are not familiar with the story (the link points to RTP Scrolls which gives a good example). It is a shame really, since I do feel that Agile is the way to go for any game, and that all management styles eventually move closer to Agile anyway, since it allows the most flexibility and creative changes to make the game more fun. I guess it is like that saying, that the only thing harder than learning something new, is forgetting something old.

But when it all comes down to it, game development should be fun too...
Bye for now,
Michael Hubbard
http://michaelhubbard.ca

Saturday, September 11, 2010

Extreme Programming

For those of you following, I have been delving into more & more practices and thought I would look into Extreme Programming (XP), created by Kent Beck (one of the original authors of the agile manifesto). Some of you (like me) may feel that the term "extreme" may be a bit overused (do our developers have to do barrel-rolls while coding at their desks?) All kidding aside, my interest in Agile development (especially Agile game development) is currently peaked and I will continue to delve into more an more aspects of it.

I decided to pick up Sams Teach Yourself Extreme Programming in 24 Hours by Stewart Baird as I had never really picked up a book that mentioned learning something so quickly (I remember the old programming post Teach Yourself Programming in 10 years by Peter Norvig :P). Nevertheless, I always feel it is important to start somewhere, and a book like Baird's gives a distilled view and framework to graft a suitable framework for extreme programming in a very concise and organized manner.

Baird's book, breaks down the XP aspects into twenty-four, "one-hour" chapters, each dealing with a different aspect of XP. There were a few chapters that were either Java or .NET based (especially related to Unit testing), so the relevance on those may vary depending on your interest, but the underlying principles are still important. But, of course the question "what makes it "EXTREME" Programming"? XP like many Agile methodologies focuses on "simplicity, communication, feedback and courage" (I feel like "planning" would also be important to add in the list, but XP focuses more on the developer than the leader/manager). XP has a lot of similarities to other agile requirements, but handles them in a slightly different manner, the most obvious being the focus on pair-programming.

The argument for pair-programming is the focus on quality, by having two sets of eyes on code as it is being written, the developers will end up with constant feedback, improved code quality and less bugs and design flaws. In Hour 11, Baird mentions the time lost upfront will be improved later as less bugs will be found and have to be fixed, thus actually saving development/bug fixing time. The trade-off being 15% slower dev times (to use pairs), but 15% less defects. Baird gives an example of how this will save time:

50,000 lines of code takes 1 individual 1000 hours (at 50 lines per hour).
50,000 lines of code takes a pair 1150 hours (at 50 lines per hour). 15% more time.

Baird uses Watts Humphrey's A Discipline for Software Engineering that 100 defects occur for every 1000 lines of code, but estimates that 30% could still remain after some software development "rigor" has been put forward, meaning 1500 defects in 50,000 lines of code. Baird mentions the total time could be 10 hours per defect to find and fix each one of them.

15,000 hours spent finding the 1500 defects from an individual developer.
12,750 hours spent finding the 1275 defects from a pair of developers. (15% less defects)
= A saving of 2,250 hours.

While I encourage the devs on my team to work together as much as possible, I am not as sure about implementing pair-programming. In my opinion, the most useful situations for pair programming would be when there is a lack of design, research is required in solving the problem, a lack of motivation from one of more developers or a real need for information sharing on a certain section of code. If the design is already in place and knows the path to take I would be less inclined to use pair-programming in those cases.

I will keep some of the extreme programming ideas in my back pocket and will think about them. I prefer the scrum approach for organizing, but do like some aspects of the book (including some benefits of pair-programming). For me simplicity is one of the best focuses, and in a professional work environment it is important to focus on YAGNI (You Ain't Gonna Need It) to prevent over-engineering (although I would encourage those interested in this type of engineering to experiment on their own home projects). I also like the idea of collective ownership of the code and of course the 40 hour work week (which is especially hard in games, as there are a lot more iterations than there are in other requirements (like instrumentation logic, or security systems, and I imagine some applications and systems too)).

I will continue investigating Agile methodologies as well as mention some of the "extreme" ideas to see if any of the other leads are interested in exploring these ideas. I would be curious to see how they work with the company, but think I may shelf them for the immediate future.

Until next time, I remain,
Michael Hubbard
http://michaelhubbard.ca

P.S. I am following up on the Extreme Programming RoadMap and will see if I can find much in the way of Extreme Game Dev (although I still think it may be hard to top Keith's Agile Game Development with Scrum (previous post)).

Art Books

Something a little different today, I'm having a look at more art related books. I feel like I always want to use "both sides" of my brain and enjoy the artistic aspects and results of the tools and games I create. For me, it was always important to get technical, so that when I had a great idea for something, I could cover all aspects of it without having to worry about technical or art abilities really getting in the way. That being said, I should focus a bit more on the artistic side (hard with so much programming these days) but, I got a few books from the library and had a good read.

Webcomics: Tools and Techniques for Digital Cartooning
by Steven Withrow & John Barber is good at describing different artists workflows of a number of cartoonist. I was interested in how some of the artists used of Poser, and while I have played with some demos, I realize this is still quite a useful tool for rapid prototyping many different characters, clothes, poses etc. From what I remember it was one of those programs that may add up with add ons, but could be useful as virtual artist dummy for things like lighting on a character, proportions, foreshortening and composition. One of the other interesting aspects of the book is the focus on the business aspects, such as page rates from publishers in print, subscription services, micro payments and merchandise. There is still some Flash work, and I feel that while there are a lot of competitors (HTML 5, or some more of the 3D programs in the browser like Unity) it is still going to be a while before something really competes with Flash on the web.

Drawing and Painting Fantasy Worlds by Finlay Cowan was a good book to inspire young artists or those who need a bit more motivation. Some of the quotes from other disciplines such as Thomas Mann's "Order and simplification are the first steps towards mastery of any subject" are sprinkled throughout the book (sidenote: for those that read Mann's Death in Venice I also liked the quote: "It is most certainly a good thing that the world knows only the beautiful opus but not its origins, not the conditions of its creation", and that is also a good warning quote for those looking at Death in Venice). Anyway, Cowan focuses on things like research and inspiration, using the internet, library, books, museums, mentors and other artists. I liked his idea of a focus on a "sacred space" that keeps you in the mindset for work and minimizes distractions (like the internet, or blog posts :P) There is a focus on figure drawing and life drawing (which I am a big fan of (here for some of my stuff)) and also work with 3D aspects using Poser. There are also more technical aspects such as digital painting starting in greys and and layers, importing different images to create depth in the work and collecting different things with unusual patterns like sun dollars, tree bark, poppyseed muffins etc. and incoprotating them in the work. There was also some of the necessary elements of one, two and three point perspectives and using grids for cities. I also really enjoyed the words "THERE ARE NO EXCUSES" (pg 118) adapt to whatever situation you have and try and be productive wherever you are. Be creative whenever and wherever you can.

Digital Manga Workshop: An Artists Guide to Creating Manga Illustrations on Your Computer by Jared Hodges and Lindsay Cibos deals with a number of elements in Photoshop and Corel Painter, with a few nods to Paint Shop Pro, openCanvas and GIMP. The tech for the book largely revolves around a wacom tablet, and has a few good tips here for some of the work. Choosing 300 DPI, dealing with transparent line art, digital inking for photoshop with a round hard brush, (5 pixel diameter, 0 angle, spacing < 25%, roundness and hardness 100, minimum diameter 1%, other options set to 0 and off, mode color burn, size control pen pressure, count 1). Also some tips like using hue and saturation to change hair color or shadow and highlights, is probably overlooked by those who use the hue/saturation for just initial color correction. Hodges and Cibos recommend using 4 different colors (for cel shading) a base, shadow, dark shadow, and highlight for hard materials and only two tone shading for soft color. The book also goes into airbrush color blending more colors or painting style (traditional or watercolor) and mentions coloring some lines (not all black) to give a softer look.

It was nice stepping away from the programming books, and while sometimes with art books the best part is looking at the artwork and getting ideas, it is still nice when some of the technique is mentioned on how to accomplish some of these visual results.

Until next time,
Michael Hubbard
http://michaelhubbard.ca

Agile Retrospectives (inspect and adapt)

Well, I said I would try and minimize the leadership aspects of this blog, but have been focusing on trying to help my team more with deadlines looming. One thing I will be doing with the team is trying to get them to help me help them :P Also, I suspect that I will be focusing on more and more Agile workflow models as I feel there are still a lot of hints of waterfall in the current system. Thus, my venture into trying to understand and apply these new ideas. I have some understanding of agile from my previous work at Leap In Entertainment where management there did a good job of applying Agile methodologies (without having to label everyone pigs or chickens).

I, however need a bit more information about agile as a whole (and will delve into other books on managing extreme programming etc.) and have begun absorbing the knowledge from online resources and library books. One that caught my attention was Ester Derby and Diana Larsen book Agile Retrospectives: Making Good Teams Great after I stumbled upon their video presentation Agile Retrospectives Talk. Of course "making good teams great" is exactly what I want to accomplish...

The need for retrospectives is to "inspect and adapt" the project and processes before the project is finished (i.e. while there is still time to improve the processes). Derby and Larsen recommend a commitment to doing retrospectives every iteration (in my case every two weeks) so that we can fix things before waiting until the end (and it being too late). Some of the things to investigate are the patterns that show how we work together, and determining what works and what doesn't, bringing up problems to solve and finding and reinforcing team strengths.

With this is mind I will try and focus on both the productivity and satisfaction of the team, and experiment, experiment, experiment. With each iteration hopefully we can see what worked from the previous retrospective and what did not. By finding out what does not work and recording it we will at least be less likely to keep repeating the same mistakes.

The breakdown that the book (and video suggest) are in five stages (page 19):

Set the stage
Gather Data
Generate insight
Decide what to do
Close the retrospective

The book also recommends allocating some shuffle time as well for very long iterations.

My focus for our next sprint/retrospective will be for trying to examine and explain what works and what does not, how well are we applying coding standards, how well are we refactoring, how well are working as a team and meeting our deadlines.

Wish me luck,
Michael
http://michaelhubbard.ca

P.S. The book is also on Scribd: Agile Retrospectives on Scribd

Monday, September 6, 2010

Herding Cats... as a Team Lead

Been quite busy, herding cats :P I won't be updating Game Programmer/Technical Artist to Team Lead/Game Programmer/Technical Artist, but will be mentioning a few new aspects as a new lead in a few blog posts. My main focus is still art and programming, but have been focusing on trying to run a good team as well. My favorite saying for leading programmers is that leading programmers is like herding cats... "You can get a cat to go anywhere you want, as long as the cat wants to go there too". This can be a lot trickier than it sounds, and in a lot of ways code can be easier than people :) I really enjoy the team I am with and have some really good guys, but dealing with people (especially with the often conflicting priorities across teams) can be tricky. I try and stay as close to the code as possible, and still try and write as much as I can, but find I need to delegate more of the smaller details as I have to be more responsible for more of the overall project and structure of the code base.

In order to help with my new responsibilities I have been reading up on a few management books, and am hoping to try a few of the management techniques to hopefully improve what I can. I finished reading Herding Cats: A Primer for Programmers who Lead Programmers by J. Hank Rainwater, which holds some interesting ideas about the challenges of programmer team management. Rainwater's main focus is "thinking and adapting" and constantly reading to learn new techniques and keep your passion for the craft (both the programming and management side). I really enjoy reading (so that is not a problem) but will extend my focus to management (agile, extreme programming etc.) to try and stay informed of some of these aspects as well.

One of the things that Rainwater mentions is "Don't let your email inbox govern your day" he recommends an email thread that is longer than three emails should be talked about on the phone or in person, and that feature creep should be caught as it can initially appear as email clarifications. He also mentions however, that it is important to only hold meetings that are useful, meetings that do not have any action items could usually be better accomplished through an email or wiki page update.

The book goes on to focus on some hiring practices, working with the different levels of an organization and looking in the mirror and evaluating yourself as a leader. I expect that I will re-read chapters of this book as it is filled with a lot of good ideas and common sense. When I was taking Business Management and Organization courses in university (oh so long ago) I often felt like compared with the technical (programming) books they perhaps did not offer as much new material as they did common sense. I think I was a little unfair in that assessment as I now feel like it is important to put those common sense theories into practice, which are not without their own challenges.

I have started reading more agile books (we are trying to get away from waterfall development as much as I can) and have begun implementing daily quick 10 minute stand up meetings in the morning to try and increase communication and sort out action items for the day (the big three questions, what did you do, what are you going to do today and what (if anything) is blocking you), and find that there are often more blocks than is easily anticipated. I have mentioned to the team a few times not to wait until the morning if they discover blocks, and always have something else for them to work on in the meantime if they have a block that requires other teams to work on first (often server or art). Hopefully the meetings are useful for these guys, I always get something out of it, and feel a few of them do too, but either way try and keep it quick.

Lately I have been working a lot of overtime (leading being my day job and programming being my night job), but after the last few months of leading I am (hopefully) getting a better hang of it and will try and put some more of these ideas (and hopefully some from the new books on my desk) into practice and get more coding/leading done in the day.

Until next time,
Michael
http://michaelhubbard.ca