Time Estimation Sucks

From the perspective of myself, a software developer, time estimation always sucks. I believe I’m working in an environment which has a better view of time estimation than those I’ve worked in before, but we still do it and I think that it’s sucking is inherent. I’ll explain why in a minute.

However, I want to say that time estimation is probably essential for project managers, product managers, executives, sales, and the like. For them, perhaps they can provide a barometer giving them a good indication of where things will be in the future. However, I don’t speak for any of them, I’m speaking from the selfish position in which I stand (as we all do and would admit when we’re being honest).

Time Estimates are Guesses

A time estimate is, by definition, going to be inaccurate. Until someone finds a prophet willing to use his gift for mundane predictions like, “How long will this new feature take to implement?” Or, “What’s the timeline on this fix?” The answer will be wrong.

To a person without the training it may seem like a software developer works magic and comes up with solutions, but that’s not the case. She’s merely working through developing an explicit set of directions, logic, and math to allow the computer to do something. However, the complexity of a problem is rarely well-known until you try to work through it. Thus, you don’t know what problems you are going to run into.

This is further complicated by the fact that you have to work within some framework including programming language, available libraries, code already in built up, customer expectations, code maintainability, solution flexibility, etc. A software developer often has to make decisions and compromises along the way and it’s not always clear what the ramifications of each of these decisions will have.

Implied Margin of Error

All time estimates include an implied, but rarely recorded or even knowable, margin of error. When I estimate the time it will take to do something, I will often say something like, “Well, since I just did a very similar process in importing that fail for client X, it will take about 2/3rd’s as much time to do it for client Y since I’m familiar with the code and added some pieces there to make doing this task easier.” I’m implying a narrow margin of error.

On the other hand, I might say, “Well, the code in that part of the application was not well future-proofed and this will build on there. I may have to do some refactorings, so I’d say we’re looking at 2 days, plus or minus a week.” I’m even stating it here, the large margin error.

Yet, what if the first task turns out to take 5 times as long as I said, does that mean my margin of error was wrong or does that mean I was operating on incomplete information?

When a meteorologist says, “50%” chance of rain, is he wrong when it rains on your house? Or when it doesn’t? No, it was just an estimate, which is a statistical entity that cannot be analyzed on those terms as a data point. Only after you have a collection of data points that shows that when the meteorologist suggested 50% chance of rain, it actually rained 60% of the time. Then, he’s been wrong (and even in this analysis, you have to be careful about data bias and all the other statistical issues your analysis may contain).

Large Estimates

Getting into the actual business case, we find that making a large time estimate on something will get you into trouble when you make the estimate.

I recall a case when I suggested I was starting on something during the last week of December after Christmas. I broke the task down into all the steps I would have to do to complete the task and concluded that an reasonable estimate of time would be the end of the second week of February. I told this to the executive in charge of the product and he said, “We’d really like to be able to tell the clients the start of the third week in January.” I said, “No. I don’t think that’s reasonable.” He said, “We’d really like to be able to tell the clients the start of the third week in January.” In other words, “I don’t care what your estimate is, this is the estimate we’ll use.” My response, “I will give you something at that time, but it won’t be finished.”

In this case, it actually took all the way through April to get it done, largely because the project ended up having twice as many features as originally spec’d.

As a software developer, I am strongly motivated to report short estimates to avoid pressure from above. Sometimes, however, a long estimate is the only one that will work, but I always expect to be criticized if I give a long estimate on something.

Short Estimates

If I end up giving an estimate for a project that actually takes me 2 to 5 times as long to complete, then I’m in big trouble. These are the times when executives yell at people. Directors and project leads ask you what lessons you’ve learned, and if someone really gets upset, you have weekly phone calls with someone in HR listening in.

So here’s where making a long estimate for a short task seems like a good idea. But then, what happened in the last section? There’s no win in this game.

Accurate Estimates

So what happens when you get it dead on? Do you get a big promotion, cigars and cognac with the executives, and a fat bonus? Nope. You just plod on. You might get such if you made a long estimate, got away with it, and then delivered way early, but that might just mean you’re a good con-artist, not necessarily good at estimating your time.

What’s the Answer?

I don’t have one for managers. That’s not my problem, but managing my managers is. So, here’s the solution (at least what seems to work best for me at this point):

1. Communicate frequently with the bosses. Anyone you know to be a stakeholder in your project should get something from you often enough to keep them in the loop, but not so often they get annoyed. Hard to balance, but in generally you ought to be talking to your supervisor at least once every day, your project/product managers should get information at least twice a week and as much as once every day, and executives should get direct feedback from you once a week if they have a stake.

   And if you find out something is really critical to someone, update them as often as your supervisor. If it’s an emergency or something critical to a sales pitch being made tomorrow, send out an update at every stopping point.

   Regular status reports help managers feel in control and you want them to feel in control. If they feel out of control, expect them to blame you for not telling them what was going on.

2. Make your guesses as good as you can. On complex tasks, break the task down as far as you can ahead of time and estimate each piece. Give yourself an explicit margin of error and percentage of accuracy you’d place on that margin. Sum up the time estimates and margins and use that to present you estimate. Take some time to do this well, don’t rush and spend more time considering the harder to estimate parts.

   When you give your estimate, you might share some or all of this with your supervisor (depending on how well you think he’ll be able to cope with the information). Don’t give this to anyone else, but use this information to determine how to talk about it when reporting on your estimate. Use vague, fuzzy language for parts you aren’t sure about and use concise, direct language on the parts you are pretty sure about.

3. Stand by your estimate. If your boss says, “Unacceptable,” or doubts you, listen to them and explain more about your estimate. Especially if you’ve worked through the time it will take in details, wavering in a moment of doubt is probably not going to serve you or anyone.

4. Be flexible. If the bosses need a shorter estimate for some reason or even expected a longer one, use that to your advantage. If you’ve done your homework and got it with you, perhaps you can negotiate parts of the spec away that aren’t as critical in order to cut down on time. Or perhaps there are things you can iron out a little better and get working really well in the time you have. Or perhaps you can use that opportunity at the tail end to get better testing and quality in place.

5. Meekness wins when all else fails. Never ever burn your bridges. When it hits the fan because you prove to be the woeful prophet you are, stand by your mistake, but stay calm. Be humble. You might have a really good idea what went wrong, but your bosses are pointing at something else. Point that out, but don’t beat them with it if they disagree.

Other than that, be you. I work really hard to never blame someone for a mistake, even if they made it. I do not tell on fellow employees when they aren’t doing their job. However, I try to make it a point to point out excellence in my fellows whenever I can. I don’t feel like it’s my job to do management (which, in my mind, means correcting and rebuking employees when they fail at something), but pointing out excellence and success is everyone’s business. If I really appreciate something a fellow developer has done or someone on the business side writing a really great spec, I try to make it a point to say so in conversation. Unfortunately, being the loner that I am, I don’t think I notice these things as often as I could.

Time estimation sucks, but it’s pretty much inevitable. The key is understanding why it sucks and how to deal with that.

Cheers.