Pages

Showing posts with label team lead. Show all posts
Showing posts with label team lead. Show all posts

Wednesday, March 9, 2011

Book Review: Death March: The Complete Software Developer's Guide to Surviving 'Mission Impossible' Projects

A great book by Edward Yourdon, Death March: The Complete Software Developer's Guide to Surviving 'Mission Impossible' Projects. Being part of a crunch turned to death march projects, this book speaks volumes on the subject and is a worthwhile read (hopefully before you start on a project). For a death march project to occur, one or more project parameters exceed the norm by more than, 50%, time or resources needed that are not available results in a death march. Yourdon continues with the disturbing news that "Death projects are the norm, not the exception", even with the data that "the average project is likely to be 6 to 12 months behind schedule and 50 to 100 percent over budget."

How do death marches occur? The book describes a number of possible catalysts: company political reasons, naive promises made by marketing, senior executives or naive project managers, naive optimisim of youth (build any system over the weekend), "marine corps" mentality (no sleep necessary for "real programmers"), intense competetion, startup mentality, or unexpected crises. Yourdon stresses the political aspect as the most likely candidate of a death march. The politics can boder on the completely irrational: "Hysterial optimism, which is when everyone in the organization desperately wants to believe that a complex project, which has never been attempted before, but which has been realistically estimated to require three years of effort, can somehow be finished in nine months" (pg 10).

Why do people participate? High risk, high reward, naivete or optimism of youth, buzz of working closely together, Mt. Everest syndrome, alternative is unemployment, or revenge. The results are sacrifice personal health, mental health and personal relationships. I also talked about some of this in my post: http://gameprogrammertechnicalartist.blogspot.com/2011/01/why-do-people-crunch.html

If management expects time and budget to be padded they are expecting the estimation of schedules to be some "negotiating game", but is also likely to be some degree of naivete and lack of understanding of what really goes on. The owner of a death march is often a much higher-level manager than would be normal for the software project, usually with a long hierarchy of shareholders, stakeholders, project managers and customers in between and around. A long hierarchy often gets distorted on the way down to the project manager, where (on the way down) additional elements are tacked on to the message, bloating the project scope and timeline, with additional requirements, constraints and processes.

The major cause of death march projects is politics, often related to the negotiating games. Management always wants the enormous luxury of a promise of what something will cost, how long it will take and not have to worry about these factors, and also gives them a scapegoat for blame if the promise is broken. Senior management needs to share the burden of uncertainty and allow for changes at all levels, not bringing in someone for an on the spot estimate and building their entire project around something that has not been careful worked through. Always defer instant estimates, and always give +- estimates like 3 to 6 months, or 6 months +-50%

The book talks about four different categories for death marches:

High moral & High chance of success = Mission Impossible (a lot of hard work, but exciting)
Low moral & High chance of success = Ugly (marine corp mentality, whips cracking).
High moral & Low chance of success = Kamikaze (try and go out with one last hurrah)
Low moral & Low chance of success = Suicide (no alternative)

The interesting thing is that these values are all relative to the individual programmer. Some programmers may feel they are on a Suicide project, while others feel like the same project is a Mission Impossible one. People's commitment changes over time, almost always in a worse way.

Work has to be mutually beneficial, and if the threat of being fired or bypassed for a raise or promotion are common, then the strongest bargaining chip you have is displaying that you are ready to walk away from the relationship, if the results aren't mutually acceptable. Two specific issues that also have a significant impact on motivation, and which are usually under the manager's direct control: rewards and overtime. Bonuses are tricky and can backfire, at a startup with part of stock option available they may be a good modivation (like tens of thousands, or millions of dollars), support for the whole family, taxi service, grocery service, and helping to provide medical attention to the whole family. This costs money, but usually a very small amount. Give extended vacation (they will need it) not just a few days but extended period of time. Consider a paid sabbatical (6 months to just work on whatever they want it to be), and loaning project equipment.

The main focus of the dealing with death marches is "triage", where each task is separated into "must-do," "should-do," and "could-do" categories. After all the tasks have been separated, all work is done on the must-dos, then the should-dos and then the could-dos (if there is time). The idea behind this is if "80-20" rule holds true, the project team might be able to deliver 80 percent of the "benefit" of the system by implementing 20 percent of the requirements (if they implement the right 20 percent). Stephen Covey puts it in First Things First [I], "the main thing is to make sure that the main thing is the main thing."

Yourdon offers some other project manager tips as well.

- Project leaders should put in as many hours as possible to lead by example.
- Target 60-80 hours a week, but know your limits, everyone reaches a point of diminishing returns.
- Need to be trueful with the team, disillusioning to find out crucial information later.
- Make sure people are compatible.
- Show work in increments, but avoid prototypes which only perpetuate an illusion of completed work.
- Modifications to baseline requirements should be made public for all to see.
- Don't introduce new processes to a death march project.

At some point the death march is likely to reach an "ugly crisis" where it is shown that the project will not be completed on time, and the project manager is fired and replaced with a new project manager who is supposed to "get it out on time". The work-in-progress created by the project team before the "ugly crisis" occurs usually ends with the sad result of being thrown away: "The real reason why all of this partially-completed work ends up being wasted inventory is that no one will ever have time to come back to it. Assuming that the project team members (now under the control of a new manager, whom they may or may not respect) is able to deliver the "bare minimum" of critical functionality, they're usually so exhausted that half of them quit. And the users are so disgusted with the project that they never bother asking for the rest of the unfinished functionality; or conversely, they're so satisfied with the minimal functionality that they never bother asking for the rest of the system".

Even though everyone understands the issues intellectually, the political battles surrounding death march projects makes it almost impossible to reach a consensus on a reasonable triage. It's only when the "ugly crisis" occurs that the various parties finally agree on something that they should have agreed upon when the project began.

Try to focus on what works, not the process or methodology, or buracracy. Not to say not to have a process, since that can be a significant process, but be reflective of what works and what doesn't. Minimize number of tools needed, like a mountain climber, just bring the essentials.

Finally, Yourdon recommends quitting or leaving if it is a viable option, learning something new for less pay will make you a happier person. If any of these succeed, what is to say that they won't try it again? Some companies need to fail to see what they are capable of, hopefully it doesn't ruin the company. It is important to not measure the success of a project by the number of divorces, broken relationships and ulcers.

I have been part of a few death marches, and while a few can be bearable for a little while, in the long run you will almost always regret it. Avoid these types of projects if at all possible, everyone needs balance in their lives. If you find yourself on this type of project, the most important thing to remember is this: Life > Job

Best of luck,
Michael Hubbard
http://michaelhubbard.ca

P.S. Or if you prefer Life > Work (Life is greater than Work, for my non-programming friends), I should make it a t-shirt :P

Sunday, February 20, 2011

Day in the Life of a Game Programmer/ Team Lead (in Crunch)

Well, the day still starts at 7:00 AM, but the timeline is a bit different in a crunch. See my original post for comparison: http://gameprogrammertechnicalartist.blogspot.com/2010/10/day-in-life-of-game-programmer-team.html

07:00-07:14 (00:14) Wake up to an alarm, keep hitting snooze.
07:14-07:32 (00:18) Shower, (skip breakfast) and leave.
07:32-07:51 (00:19)
Leave apartment walk to bus stop.
07:51-08:30 (00:39) Get on bus, rest my eyes, but do not sleep.
08:30-08:33 (00:03) Walk to office, drop stuff off at my desk.
08:33-08:37 (00:04) Grab one of the many coffees of the day.
08:37-08:44 (00:07) Update code, check email, some admin stuff.
08:44-08:54 (00:10) Talk with one of the dev, answer questions.
08:54-09:15 (00:21) Update and print out bug list from QA.
09:15-09:30 (00:15)
Stand up meeting for half the dev team.
09:30-09:45 (00:15)
Stand up meeting for the other half of team.
09:45-10:00 (00:15) Meeting with server leads, about bug list.
10:00-10:06 (00:06) Grab another coffee, talk about a bug.
10:06-10:36 (00:30) Code.
10:36-10:42 (00:06) Clarify some code standard.
10:42-10:50 (00:08) Give current status.
10:50-10:54 (00:04) Answer some questions about specific feature.
10:54-11:38 (00:44) Code.
11:38-12:18 (00:40) Test and submit some bug fixes.
12:18-12:24 (00:06) Walk to get some lunch, call girlfriend.
12:24-12:36 (00:12) Buy some lunch (Subway), take it to my desk.
12:36-01:07 (00:31) Update repository, eat, and continue to code.
01:07-01:10 (00:03) Give status.
01:10-01:14 (00:04) Code.
01:14-01:21 (00:07) Answer important emails for clarification.
01:21-01:26 (00:05) Code.
01:26-01:29 (00:03) Help answer a developer question.
01:29-01:35 (00:06) Code.
01:35-01:43 (00:08) Grab another coffee, and coffee related break.
01:43-01:48 (00:05) Answer some questions in email.
01:48-01:51 (00:03) Code.
01:51-02:33 (00:42) Help developer debug an issue.
02:33-02:35 (00:02) Grab another coffee.
02:35-02:57 (00:22) Code.
02:57-03:01 (00:04) Answer email.
03:01-03:36 (00:35) Merge code and look over changes.
03:36-03:46 (00:10) Answer questions.
03:46-04:11 (00:25) Code.
04:11-04:20 (00:09) Give status.
04:20-04:24 (00:04) Code.
04:24-04:47 (00:23) Grab coffee with another dev.
04:47-05:59 (01:12) Code.
05:59-06:14 (00:15) Discuss implementation issue with other dev.
06:14-06:27 (00:13) Pizza arrives, grab a few slices and a pop.
06:27-07:17 (00:50) Code.
07:17-07:36 (00:19) Talk with developer, write email for tomorrow.
07:36-09:58 (02:22) Code.
09:58-10:09 (00:11) Add notes and tasks for tomorrow.
10:09-10:43 (00:34) Talk with other devs about tasks, status.
10:43-10:58 (00:15) Leave and walk/wait for bus.
10:58-11:36 (00:38) Rest on bus, phone and reading.
11:36-11:46 (00:10) Walk home.
11:46-12:02 (00:16) Talk with girlfriend, get ready for bed.
12:02-12:36 (00:34) Personal email, brief web surf.
12:36-07:00 (06:24) Sleep, so the whole thing can start again.


Breakdown:

Work:
Administration (email, delegating tasks, status): 2 hour 8 minutes.
Breaks (lunch, dinner, coffee): 1 hour and 14 minutes.
Coding: 8 hours and 33 minutes.
Helping Others: 1 hour and 30 minutes.
Meetings: 45 minutes.

Home:
Computing (web surfing, project coding): 34 minutes.
Exercise: 0 minutes.
Personal Maintenance (food, shower, etc.): 48 minutes.
Reading (including bus travel time): 38 minutes.
Sleep: 6 hours 24 minutes.
Travel (not including bus reading time): 1 hour 26 minutes.

Other Facts:
Read 48 emails.
Sent 13 emails.
No code review.

If you compare this with the previous day in the life entry there are a few substantial differences. Even though I am spending approximately 16 hours and 14 minutes away from home, the coding time still only amounts to half of my day. All the other aspects including meetings, admin and help have gone up, and nothing has gone down (on the work side).

The home side is where all the time goes. Sleep takes a hit (about half and hour less), exercise is non-existent, and coupled with fast food for lunch and dinner is going to make me gain weight and have less energy overall. This plays into attempting to sleep on the bus, but really turns into wasted time, since I want to sleep in the morning (and am too tired to read), but can't really get to sleep on the bus with all the noise and crowded seats in the morning. I also tend to miss making breakfast, but will try and grab something like a banana, apple or bagel on my way out the door to eat on the way to the bus.

There are still a lot of interruptions, a lot more status updates and emails. Organizing the crunch is tricky since there are always "fires" that have to be put out, and as things move faster, more coordination is needed to make everything work smoothly.

With so much time at work, there is very little personal time for any time with loved ones, projects, additional reading, or relaxation. I try and phone when I have free time (lunch and on the bus), but this is not the same as being around. If you are at this pace (or worse) for a prolonged period of time, that means you are on a death march. If this lasts longer than a few weeks (longer than three weeks maximum) it will start to take a toll. This is not the worse pace I have been on (I slept under my desk multiple consecutive nights earlier in my career), but it is a difficult one to maintain.

With so many tired devs, so much extra coordination, many interruptions, minimal code reviews, extra status updates, it becomes a tricky situation to balance. Throw in low energy, running on coffee and a lot of other tired developers and co-workers, it can quickly turn ugly if spirits are not high. Everyone wants to see the project out, but there is still a lot of work to do. I should probably take my vitamins.

Wish me luck,
Michael Hubbard
http://michaelhubbard.ca

Sunday, January 30, 2011

Why do people crunch?

I was asked by a friend an colleague: "why do people crunch"? This is not an easy one to answer. I suppose for everyone it is a little different, and is often based on the person themself, their attitude, personality and history. For me, I usually enjoy the work, and like the challenge and the feeling of accomplishment when all is said and done (to a point). There are others who will feel it is their duty and responsibility to complete a task to the best of their ability. Others still will fear the consequences if they do not get their work done, and for many it is likely a different combination of all of these responses (and more).

What a crunch can be, is a way of finally pushing the efforts of your labor out into the real world. It can be a symbol of pride to get your work out there, and to finally see it stand by itself. It is hopefully something to be proud of, to feel a sense of accomplishment and satisfaction of what was achieved (often the more difficult projects are the most rewarding in the end). Hopefully along the way you have gelled with the other developers and created a great and fun team.

The opposite end of the spectrum is a situation where whips crack, and people yell. Where all of your blood, sweat and tears is just going towards making someone else richer, and everyone on the team feels that way. While some overtime is almost always expected with any sort of programming job, a ridiculous work schedule should not be accepted for any job. Do what you can, and make sure that whatever you do, you work as a team. Making games should be fun, if it is not, something is going seriously wrong.

Best of luck,
Michael Hubbard,
http://michaelhubbard.ca

Sunday, December 12, 2010

Book Review: Peopleware

The book Peopleware:Productive Projects and Teams (2nd edition) by Tom Demarco and Timothy Lister should be required reading for anyone in charge of or working with software projects. So much of it is such good sense, and yet so much of it will make you shake your head and say "why are we still doing it like this?" If you feel like you are not being able to do work between 9-5, read this book.

Two of the major concepts in the book are teams "jelling" and programmer "flow".

A team "jelling" or coming together is so important and yet nearly impossible to plan for. Demarco and Lister mention that you can create a scenario for a team to jel, but it all depends on the people involved in whether or not the team comes together. There are however elements that can be "teamicide" or kill a team (defensive management, bureaucracy, physical separation, fragmentation of time, quality reduction of the product, phony deadlines, and clique control).

Some recommendations are to group people together. Groups usually go quiet at the same time, you want them to interact and trust each other. If they are not sitting together, they are more likely they will interact with people on other projects and become friends with them instead of the people on their team. Trust your people. Get the right people and turn them loose, hiring the right people is the most important thing. Develop a feeling of permanence, train people to go through the ranks, make it a place people are expected to stay, not just passing through.The authors use the imagery of a team like a choir (or visualize an orchestra), with everyone playing their part in a complex symphony of sound and harmony, which differs from the traditional example of a team like a sports team, which often only focuses on the star players. The book also talks about having "free electrons", people whose judgement and work ethic make the decisions they make better than those mangaging them. Which again comes down to trusting your team. If you are a bad manager (and aware of your shortcomings), the best thing you can do is hire the best people you can find and get out of their way, and try and avoid changing too much.

You want your team to work together, to not compete against each other and to be happy and productive... how that happens is always complex, because it depends on the individuals in your team, but making it work is your job as a leader.

The other element programmer "flow" and how work is like sleep, it is best done in long periods of silence with minimal interruptions. Try to minimize interrupting people, programmers (and all workers) need quiet time to think and concentrate. Even with music or white noise the programmer is not using their full mental capacity, which, while they are able to perform some logical function, still might not be able to perform or think as creatively. The book also focuses on noise, how lots of people packed into cubicles make a lot of noise (the smaller the space, the more noise they make) and that a PA system is almost insulting to the occupants in disrupting the masses in order to find one person.

The book talks a bit about the Capability Maturity Model (CMM) and how so many companies are in the lowest levels (often with no sign of improvement).

Another part of the book that struck a cord with me is the huge cost of turnover. Even if the chair stays warm with a new body coming in, they are less then useless the first day, and take other peoples time to get up to speed. If they are lucky it will be less than 2 years before they replaced a senior person, but there is still no guarantee they will "jell with the team, be as reliable as the previous person, or will even stay long enough to get up to speed that the previous employer was able to manage. The cost can be enormous for replacing people, if it takes six months to get up to speed an be productive coming into a project that has seven months left, you have paid someone seven months worth of pay for one month of good work. A quote from the book suggests it can be even higher than that: "One of our clients, a builder of network protocol analyzers and packet sniffers, estimates that it takes more than two years to bring a new worker up to speed (pg 207)."

The book also has the idea that there is a "slumbering giant" in a corporation, against sillyness, sharing the load of responsibility against problems and focusing on the never ending battle of how to improve the workplace. Hopefully he awakens soon.

"The ultimate management sin is wasting people's time (pg 215)."

Great book, get it for your loved (or not so loved) managers for Christmas :P
Michael Hubbard
http://michaelhubbard.ca

Monday, October 18, 2010

Coding Style and Stylecop

How important is coding style? This debate is something that really draws out the "artist" and critic in every programmer. In many ways the style of writing code is similar to writing poetry, and what makes the subject so hotly debated is that the beauty of the code, sentence or poem is all in the eye of the beholder. Just like those critics of of prose and poetry, the serious critiques of coding style come from understanding the limits and history of the form. Practice reading and writing lots of code, in different languages if possible, and acknowledging how many ways there are to write the same functionality.

In the same way that experimentation in poetry and prose makes for a better understanding of the overall form, understanding how different coding standards can make for better (or worse) code. With every coding standard the code will be limited but also potentially clarified in a new way.

Organization of code one way can make code reviews easier, but potentially making maintanability more difficult, code may be separated into different sections that are ordered in a certain way that causes significant scrolling, opening of multiple files and jumping through layers of code, that structured a different way may minimize bouncing around.

Coding standards for teams is especially important, and should be strictly adhered to as best as possible, for the overall benefit of the team. Consistency is very important as being able to quickly jump into a new class, module or section of the code "should" be as easy as jumping into a section that you wrote yourself. Coding standards are an agreement between programmers to create a cohesive work, much like any collaborative effort, the final product should be a uniform work of art.

The most diplomatic way to handle this is often through adopting a tool that lets catches errors quickly and consistently. This also frees up code reviews to be on more important aspects like architecture, re-usability, error checking, design patterns etc. as non-adherence to coding standards often ends up becoming the main focus if the code strays too far from the standards outlined. These tools should also be part of the commit process, so that the code is rejected if it does not pass the coding standards (in certain cases some of these tools may have to have slight adjustments, as desired, but with a focus on consistency).

For those C# Unity or XNA projects some guys really like Resharper (30 day trial at http://www.jetbrains.com/resharper/) For a free style specific checker I like StyleCop: http://stylecop.codeplex.com/ there is a good amount of customization and easily embeds into Visual Studio. The information that is provided from the code checks make it easy to know the issue and change the code to follow the coding standards that have been setup.

For python there is pep8 http://pypi.python.org/pypi/pep8 and pychecker http://pychecker.sourceforge.net/ as well as a few more.

In general the style and syntax checkers are becoming quite popular (especially with scripting languages that don't compile) but regardless a good "style cop" will improve code quality, readability of all sections of the code, and allows everyone to focus on writing great code, instead of what it looks like in the end.

Until next time, StyleCop: that is all.
Michael Hubbard
http://michaelhubbard.ca

Saturday, October 16, 2010

Book Review: Agile Software Engineering

Agile Software Engineering by Orit Hazzan and Yael Dubinsky works largely as an undergraduate textbook, dealing a lot with how to teach and educate Agile methodologies in a learning environment. The book is broken up into progressively more involved steps with each chapter ending with a summary and reflective questions. There are also breakdowns of running an agile workshop and dealing with planning, time estimation, reflection. As a whole, the book is quite useful as a beginner-intermediate approach to agile, and would likely work well for teachers & students, as a lot of the structure of the book is related to setting up and teaching agile in a classroom.

One of the more unique parts of the book is how the book references different aspects of agile using the HOT (Human, Organizational and Technological) perspectives. I found that looking at different agile concepts through these filters a useful way to see how agile works on many different levels. The book is also good at giving examples regarding globalization and how organizational culture can be dealt with in projects that deal with outsourced or contracted work, and how agile attempts to address these issues by providing additional transparency and interactions. The HOT perspectives I found were especially pertinent in those areas.

My favorite example, however, was on the the chapter on Trust using elements of game theory, and where the Prisoners Dilemma is applied to a work environment. This was applied specifically to where developers choose to cooperate or compete with one another and results in three possible scenarios:

1. If both developers cooperate they will be more likely to succeed.
2. If only one developer attempts to cooperate (by learning the others code, helping the other etc.) there may be a success of the project, but potentially the one developer that choose to cooperate may not receive as much recognition (since time was spent cooperating instead of competing).
3. If both developers compete (do not help each other, do not learn each others code) there will be no integration of efforts and the project will likely fail.

Obviously the ideal scenario is where both developers cooperate with one another and are awarded equally. As a lead or manager it is important to recognize when and where this type of scenario presents itself, and attempt to foster cooperation over competition.

The book also points to a link to code of ethics at the ACM (Association for Computing Machinery) and while I was not able to find the original link provided in the book at http://info.acm.org/server/se/code.htm I was able to find a similar document on the same site at http://www.acm.org/about/code-of-ethics

Please allow me to delve into a small tangent here: while I personally prefer to not have to mention ethics directly to my team (in hopes that everyone plays nice), there is often a significant amount of emotion that developers have for their work, and some developers can be very defensive about there work, to the point of being difficult to deal with. This attitude is unfortunate as every piece of code can be improved, every single piece of code! There is no such thing as perfect code, just as there is no such thing as a perfect poem, or a perfect painting or anything else that requires design. Things like structure, patterns, organization of the code, variable names, comments, formatting, optimization and more, are all subject to interpretation of what is good and thus can vary greatly from developer to developer. As a team lead, I need the developers to cooperate on the coding standards, but even then there can be a significant amount of hubris in some developers which can cause internal clashes. I have been lucky in my attempts to be like "lukewarm water" between "fire" and "ice", but through my experience feel like a code of ethics may also be a necessity, especially in larger teams.

Overall, the book definitely fits the niche well of being a good learning tool for those interested in Agile methodologies, and is probably best suited in a group environment as many of the reflective questions would likely garner some very interesting responses.

Until next time, I remain,
Michael Hubbard
http://michaelhubbard.ca

Thursday, October 14, 2010

Creating a Great Team

Comparison time: Great Hockey Team and Great Programmer Team.

While I am living in Toronto, I will instead focus on the Vancouver Canucks as a great team example. What do they have?

Star Players: Henrik and Daniel Sedin, Roberto Luongo.
Solid Reliable Players: Ryan Kesler, Alex Burrows, Christian Ehrhoff.
Good to Decent Players: the rest of the Canucks
Up and Coming Rookies: Cody Hodson, Cory Schneider
Parting ways with those that don't fit: Kyle Wellwood
Decent managment: Mike Gillis, Alain Vigneault

Some explanations:

Star Players: These are your guys leading the charge. They will have the most experience, most skill and most dedication to the project. Having a few of these guys will inspire the rest of the team to work harder and better, and will likely be the ones you have to rely upon in times of need.

Solid Reliable Players
: While the stars are working their magic, these are the workhorses of the team, dependable, reliable, and dedicated. While they might not have the skill and experience of the stars, it is unlikely you can get stars in every position. The better skilled these lines are, the better chance you will have of completing the project successfully.

Good to Decent Players:
These are the rest, and they should be of varying skill level, but all capable in some way, notice I didn't include any bad players. To be in the NHL, you have to have some talent, it should be exactly the same for your company. A player that has no talent will not survive in the NHL or on any good team, and your team should be no different. Give people a chance, give them training, but if they don't have the capability (or the heart), it will forever be a weak link that could be filled by someone else that would help the rest of the team more.

Up and Coming Rookies
: These are your juniors and new team members, there is always potential that these could develop into your next stars, or alternatively be sent back down to the minors. A sports team will often give a person an extended tryout, but that doesn't mean they have to make the team, if they aren't getting the job done.

Parting ways with those that don't fit: Sometimes a player can have some good skills, but just not be the right fit for the position they are looking for. Sadly at these times, it is important to do what is best for the team, and that often means parting ways with those individuals who could be useful (maybe with another project at the company) but not on the team as it is now.

Decent management: Just decent management? Sure, while I like the management in Vancouver, it is really the players that are the stars, not the management team. You need smart people at the top helping make the decisions, but unless they are helping score goals (or commit bug fixes) often it is better for them to just let the players do their job.

So what does this mean? Basically the best thing you can hope to do, is hire the best people you can at every position and turn them loose. If you build a team of just rookies and a couple of fourth liners, don't expect them to be able to compete in the big leagues. In the end, you always get way you pay for.

I keep hoping that Toronto will turn it around, but can't help but wonder where their super star players are?

Go Canucks, Go Leafs!
Michael Hubbard
http://michaelhubbard.ca

Monday, October 4, 2010

Day in the Life of a Game Programmer/ Team Lead

In order to improve my time (and hack my life) I have to know what it has in it. Here is my breakdown of an average (quiet) day, although it is still 9:00-6:30. We have been pretty good about doing 9:00-5:00 hours and that is what most of our guys do, and I would prefer to keep it that way. For me, every day is slightly different, some days I have five hours worth of meetings. Other days I do not leave the office till 8:00pm or come in the next day at 7:00am. It is still better than the hours I have spent at other companies, and have been a member of the 100 hours a week club more times than I would like. But this is a pretty good indication of what I would consider average for now, and with the hope I can get back to more 9:00-5:00 days once the major deadlines have passed.

The day starts for me (usually) at 7:00 AM, and I also included the time it takes for each task in parenthesis.

07:00-07:09 (00:09) Wake up, and then really wake up.
07:09-07:32 (00:23) Morning routine, get dressed.
07:32-07:53 (00:21) Make and eat breakfast while checking email.
07:53-07:59 (00:06) Cleanup, brush teeth, put shoes on.
07:59-08:17 (00:18) Leave apartment start walking to bus stop.
08:17-08:54 (00:37) Get on bus to work, start reading a book.
08:54-08:58 (00:04) Get off bus walk to office and my desk.
08:58-09:29 (00:31) Update code, check email, handle admin duties.
09:29-09:45 (00:16) Stand up meeting for half the dev team.
09:45-10:00 (00:15) Stand up meeting for other half of the devs.
10:00-10:30 (00:30) Assist devs requiring clarification on tasks.
10:30-10:36 (00:06) Write emails to other departments on tasks.
10:36-10:39 (00:03) Grab a glass of water.
10:39-10:45 (00:06) Help a developer find a bug they are fixing.
10:45-11:01 (00:16) Read source code and documentation.
11:01-11:35 (00:34) Code.
11:35-11:38 (00:03) Answer a few questions on a feature.
11:38-12:05 (00:27) Code.
12:05-01:00 (00:55) Lunch.
01:00-01:57 (00:57) Code and test, submit some updated APIs.
01:57-02:10 (00:13) Discuss bug fixes and design strategy.
02:10-02:23 (00:13) Code.
02:23-02:33 (00:10) Work with developer finding bugs.
02:33-02:40 (00:07) Code.
02:40-03:01 (00:21) Work with developer solving issue.
03:01-03:09 (00:08) Code.
03:09-03:21 (00:12) Help developer fix bugs.
03:21-03:27 (00:06) Get some tea.
03:27-03:37 (00:10) Code.
03:37-03:41 (00:04) Answer questions about code base.
03:41-04:00 (00:19) Reply to email and write additional email.
04:00-04:12 (00:12) Work on bug fixes with developer.
04:12-04:28 (00:16) Code.
04:28-05:01 (00:33) Read code review from large feature drop.
05:01-05:06 (00:05) Discuss some updates to a part of the code.
05:06-05:31 (00:25) Finish code review and add comments.
05:31-05:55 (00:24) Read updates to code base from other devs.
05:55-06:24 (00:29) Code and submit.
06:24-06:33 (00:09) Shut down, start walking to bus stop.
06:33-07:15 (00:42) Get on bus, read book.
07:15-07:24 (00:09) Get off bus, start walking home.
07:24-07:31 (00:07) Freshen up.
07:31-07:53 (00:22) Check mail and email.
07:53-08:36 (00:43) Make dinner, eat and cleanup.
08:36-09:56 (01:20) Personal code, while listening to music or tv.
09:56-10:41 (00:45) Get changed for gym, pump iron and shower.
10:41-11:45 (01:04) Play around on computer, code, art or games.
11:45-11:51 (00:06) Get ready for bed.
11:51-12:05 (00:14) Read a bit in bed.
12:05-06:59 (06:55) Sleep (perchance to dream).


Breakdown:

Work:
Administration (email, delegating tasks): 1 hour 53 minutes.
Breaks (lunch, coffee): 1 hour and 4 minutes.
Coding: 4 hours and 2 minutes.
Helping Others: 1 hour and 56 minutes.
Meetings: 31 minutes.

Home:
Computing (web surfing, project coding): 3 hours 7 minutes.
Exercise: 45 minutes.
Personal Maintenance (food, shower, etc.): 1 hour 34 minutes.
Reading (including bus travel time): 1 hour 33 minutes.
Sleep: 6 hours 55 minutes.
Travel (not including bus reading time): 40 minutes.

Other Facts:
Read 39 emails.
Sent 5 emails.
Added 37 comments to code review.

A few noteworthy points to consider about the data. The travel time is actually 1 hour and 59 minutes total for me every day, but I only counted the parts I do not do anything else (i.e. not the reading time). Looking over the numbers, it is interesting to see how much time is spent on helping others and administrative tasks. When I first joined the company as a game programmer, I was able to spend almost my full-time on coding and sent very few emails. However as time went on and more people were hired, I spent more time helping the other programmers too. Still, it is interesting to see how the day breaks down like this, and on further investigation where I can improve my use of time (like maybe try exercising in the morning as part of the regular morning routine)? Or take 30 minute lunches instead of the full hour and leave at 6:00 PM? Or try and come in earlier instead of leaving later?

Another thing to consider is how long it takes a game developer to get "in the zone". Some people at work estimate it can take up to ten minutes to get in the zone, but only one second to get out of it. While it would be great to minimize interruptions, it is far better for me to help remove the other developers roadblocks whenever possible.

Anyway, I will think over the data and may update any changes I made to my day, in a future post.

Until then,
Michael Hubbard
http://michaelhubbard.ca

Thursday, September 16, 2010

Book Review: Agile Game Development With Scrum

Staying on this Agile game development interest, I was at the local book store the other day and ran across the book Agile Game Development With Scrum by Clinton Keith and thought, "how appropriate". Needless to say I was intrigued about the aspects of applying Agile directly to game development and immediately started reading it.

One of the elements that Keith keeps repeating throughout the book is that there are "no silver bullets" and adopting Agile won't save a team already on a death march schedule, nor will it allow a team to handle an impossible schedule. What Keith proposes instead, is that Agile will give a team a framework to learn about scheduling issues, current velocity and burn down charts as well as processes to detect potential issues with the schedule or design, far earlier in the project.

The main point that I found crucial to success, was the idea of cross domain teams. A couple artists, level designers, game programmers, server programmers etc. would form a single team responsible for part of the game, and would handle all aspects of that feature, while iterating over the different stages every two week sprint. Adopting this team structure, I feel, would be a huge boon to any game dev work place as the slowest part of any game is always (always!) cross team integration. Here are two examples of what could happen in a non-cross domain team:

1. An artist creates an asset and has no developer support to test it. Without integration they go on to make multiple assets the same way spending weeks (or months) of effort, only to find out later they need to modify all their assets now that dev has begun testing them.

2. Server team creates APIs for client, but no client team is available to test them. Server developers go off and create new APIs. Once the client begins testing the APIs they find they need to add additional parameters and/or bug fixes. This requires days of rewrites from a different developer, since the original author is working on another critical section and is not available to update the code.

I see these kinds of issues happening over and over again in large scale isolated teams, where individuals begin to make decisions that should require cross domain knowledge, but instead move forward without that knowledge, and often wastes significant time when having to fix their mistakes. This amount of rework can also result in a crunch time later to try and catch up on the previous wasted effort.

One of the other important things that Keith mentions is the issue of burnout during a crunch, and estimates that only two to three weeks actually show any improved velocity. Once a month (or more) of crunch has been put into effect the overall velocity is actually worse than that of a non-crunch velocity! People will get tired, frustrated and upset with extended overtime, and Keith points to the http://www.igda.org/quality-life-white-paper-info that measures the quality of life of game developers across multiple industries, which is a good read I will probably comment more on in a later post.

The book also goes on to describe a number of key aspects regarding Agile practices (running a scrum, user stories, minimizing up front documentation, continuous integration and working builds every sprint) but also concepts for implementing and introducing Agile and scrum to the workplace, shifting emphasis to a team responsibilities for setting schedules and working with Agile for Art, QA and Producers.

Overall the book offered some valuable insights and interesting experiences. I would love try and implement more Agile processes at my current workplace, but we shall see how strong the culture is resistant to change. There are a number of jokes about trying to change culture like Angry Monkeys if you are not familiar with the story (the link points to RTP Scrolls which gives a good example). It is a shame really, since I do feel that Agile is the way to go for any game, and that all management styles eventually move closer to Agile anyway, since it allows the most flexibility and creative changes to make the game more fun. I guess it is like that saying, that the only thing harder than learning something new, is forgetting something old.

But when it all comes down to it, game development should be fun too...
Bye for now,
Michael Hubbard
http://michaelhubbard.ca

Saturday, September 11, 2010

Extreme Programming

For those of you following, I have been delving into more & more practices and thought I would look into Extreme Programming (XP), created by Kent Beck (one of the original authors of the agile manifesto). Some of you (like me) may feel that the term "extreme" may be a bit overused (do our developers have to do barrel-rolls while coding at their desks?) All kidding aside, my interest in Agile development (especially Agile game development) is currently peaked and I will continue to delve into more an more aspects of it.

I decided to pick up Sams Teach Yourself Extreme Programming in 24 Hours by Stewart Baird as I had never really picked up a book that mentioned learning something so quickly (I remember the old programming post Teach Yourself Programming in 10 years by Peter Norvig :P). Nevertheless, I always feel it is important to start somewhere, and a book like Baird's gives a distilled view and framework to graft a suitable framework for extreme programming in a very concise and organized manner.

Baird's book, breaks down the XP aspects into twenty-four, "one-hour" chapters, each dealing with a different aspect of XP. There were a few chapters that were either Java or .NET based (especially related to Unit testing), so the relevance on those may vary depending on your interest, but the underlying principles are still important. But, of course the question "what makes it "EXTREME" Programming"? XP like many Agile methodologies focuses on "simplicity, communication, feedback and courage" (I feel like "planning" would also be important to add in the list, but XP focuses more on the developer than the leader/manager). XP has a lot of similarities to other agile requirements, but handles them in a slightly different manner, the most obvious being the focus on pair-programming.

The argument for pair-programming is the focus on quality, by having two sets of eyes on code as it is being written, the developers will end up with constant feedback, improved code quality and less bugs and design flaws. In Hour 11, Baird mentions the time lost upfront will be improved later as less bugs will be found and have to be fixed, thus actually saving development/bug fixing time. The trade-off being 15% slower dev times (to use pairs), but 15% less defects. Baird gives an example of how this will save time:

50,000 lines of code takes 1 individual 1000 hours (at 50 lines per hour).
50,000 lines of code takes a pair 1150 hours (at 50 lines per hour). 15% more time.

Baird uses Watts Humphrey's A Discipline for Software Engineering that 100 defects occur for every 1000 lines of code, but estimates that 30% could still remain after some software development "rigor" has been put forward, meaning 1500 defects in 50,000 lines of code. Baird mentions the total time could be 10 hours per defect to find and fix each one of them.

15,000 hours spent finding the 1500 defects from an individual developer.
12,750 hours spent finding the 1275 defects from a pair of developers. (15% less defects)
= A saving of 2,250 hours.

While I encourage the devs on my team to work together as much as possible, I am not as sure about implementing pair-programming. In my opinion, the most useful situations for pair programming would be when there is a lack of design, research is required in solving the problem, a lack of motivation from one of more developers or a real need for information sharing on a certain section of code. If the design is already in place and knows the path to take I would be less inclined to use pair-programming in those cases.

I will keep some of the extreme programming ideas in my back pocket and will think about them. I prefer the scrum approach for organizing, but do like some aspects of the book (including some benefits of pair-programming). For me simplicity is one of the best focuses, and in a professional work environment it is important to focus on YAGNI (You Ain't Gonna Need It) to prevent over-engineering (although I would encourage those interested in this type of engineering to experiment on their own home projects). I also like the idea of collective ownership of the code and of course the 40 hour work week (which is especially hard in games, as there are a lot more iterations than there are in other requirements (like instrumentation logic, or security systems, and I imagine some applications and systems too)).

I will continue investigating Agile methodologies as well as mention some of the "extreme" ideas to see if any of the other leads are interested in exploring these ideas. I would be curious to see how they work with the company, but think I may shelf them for the immediate future.

Until next time, I remain,
Michael Hubbard
http://michaelhubbard.ca

P.S. I am following up on the Extreme Programming RoadMap and will see if I can find much in the way of Extreme Game Dev (although I still think it may be hard to top Keith's Agile Game Development with Scrum (previous post)).

Agile Retrospectives (inspect and adapt)

Well, I said I would try and minimize the leadership aspects of this blog, but have been focusing on trying to help my team more with deadlines looming. One thing I will be doing with the team is trying to get them to help me help them :P Also, I suspect that I will be focusing on more and more Agile workflow models as I feel there are still a lot of hints of waterfall in the current system. Thus, my venture into trying to understand and apply these new ideas. I have some understanding of agile from my previous work at Leap In Entertainment where management there did a good job of applying Agile methodologies (without having to label everyone pigs or chickens).

I, however need a bit more information about agile as a whole (and will delve into other books on managing extreme programming etc.) and have begun absorbing the knowledge from online resources and library books. One that caught my attention was Ester Derby and Diana Larsen book Agile Retrospectives: Making Good Teams Great after I stumbled upon their video presentation Agile Retrospectives Talk. Of course "making good teams great" is exactly what I want to accomplish...

The need for retrospectives is to "inspect and adapt" the project and processes before the project is finished (i.e. while there is still time to improve the processes). Derby and Larsen recommend a commitment to doing retrospectives every iteration (in my case every two weeks) so that we can fix things before waiting until the end (and it being too late). Some of the things to investigate are the patterns that show how we work together, and determining what works and what doesn't, bringing up problems to solve and finding and reinforcing team strengths.

With this is mind I will try and focus on both the productivity and satisfaction of the team, and experiment, experiment, experiment. With each iteration hopefully we can see what worked from the previous retrospective and what did not. By finding out what does not work and recording it we will at least be less likely to keep repeating the same mistakes.

The breakdown that the book (and video suggest) are in five stages (page 19):

Set the stage
Gather Data
Generate insight
Decide what to do
Close the retrospective

The book also recommends allocating some shuffle time as well for very long iterations.

My focus for our next sprint/retrospective will be for trying to examine and explain what works and what does not, how well are we applying coding standards, how well are we refactoring, how well are working as a team and meeting our deadlines.

Wish me luck,
Michael
http://michaelhubbard.ca

P.S. The book is also on Scribd: Agile Retrospectives on Scribd

Monday, September 6, 2010

Herding Cats... as a Team Lead

Been quite busy, herding cats :P I won't be updating Game Programmer/Technical Artist to Team Lead/Game Programmer/Technical Artist, but will be mentioning a few new aspects as a new lead in a few blog posts. My main focus is still art and programming, but have been focusing on trying to run a good team as well. My favorite saying for leading programmers is that leading programmers is like herding cats... "You can get a cat to go anywhere you want, as long as the cat wants to go there too". This can be a lot trickier than it sounds, and in a lot of ways code can be easier than people :) I really enjoy the team I am with and have some really good guys, but dealing with people (especially with the often conflicting priorities across teams) can be tricky. I try and stay as close to the code as possible, and still try and write as much as I can, but find I need to delegate more of the smaller details as I have to be more responsible for more of the overall project and structure of the code base.

In order to help with my new responsibilities I have been reading up on a few management books, and am hoping to try a few of the management techniques to hopefully improve what I can. I finished reading Herding Cats: A Primer for Programmers who Lead Programmers by J. Hank Rainwater, which holds some interesting ideas about the challenges of programmer team management. Rainwater's main focus is "thinking and adapting" and constantly reading to learn new techniques and keep your passion for the craft (both the programming and management side). I really enjoy reading (so that is not a problem) but will extend my focus to management (agile, extreme programming etc.) to try and stay informed of some of these aspects as well.

One of the things that Rainwater mentions is "Don't let your email inbox govern your day" he recommends an email thread that is longer than three emails should be talked about on the phone or in person, and that feature creep should be caught as it can initially appear as email clarifications. He also mentions however, that it is important to only hold meetings that are useful, meetings that do not have any action items could usually be better accomplished through an email or wiki page update.

The book goes on to focus on some hiring practices, working with the different levels of an organization and looking in the mirror and evaluating yourself as a leader. I expect that I will re-read chapters of this book as it is filled with a lot of good ideas and common sense. When I was taking Business Management and Organization courses in university (oh so long ago) I often felt like compared with the technical (programming) books they perhaps did not offer as much new material as they did common sense. I think I was a little unfair in that assessment as I now feel like it is important to put those common sense theories into practice, which are not without their own challenges.

I have started reading more agile books (we are trying to get away from waterfall development as much as I can) and have begun implementing daily quick 10 minute stand up meetings in the morning to try and increase communication and sort out action items for the day (the big three questions, what did you do, what are you going to do today and what (if anything) is blocking you), and find that there are often more blocks than is easily anticipated. I have mentioned to the team a few times not to wait until the morning if they discover blocks, and always have something else for them to work on in the meantime if they have a block that requires other teams to work on first (often server or art). Hopefully the meetings are useful for these guys, I always get something out of it, and feel a few of them do too, but either way try and keep it quick.

Lately I have been working a lot of overtime (leading being my day job and programming being my night job), but after the last few months of leading I am (hopefully) getting a better hang of it and will try and put some more of these ideas (and hopefully some from the new books on my desk) into practice and get more coding/leading done in the day.

Until next time,
Michael
http://michaelhubbard.ca