Wednesday, February 08, 2006

Estimation: What We Don't Know

I was listening to a talk about prediction given by Nassim Nicholas Taleb, and I felt that it has a strong correlation to project estimation. He talks about how we focus on what we know when making a prediction, and not what we don't know. This makes sense, as you can only estimate based on past experience. Poor estimation on the other hand is often due to things that we didn't know or expect.
"You can not forecast unless you know the error rate"
- Nassim Nicolas Taleb
To be able to better estimate what we don't know, we need to know at know how badly we have estimated in the past. To do this we need to be diligent in providing estimations with copious notes prior to starting a project. During the project we need to update this estimate as we start to learn those things we didn't know. Following the project we need to review the estimates against the actual time, and try to make some conclusions from this. This still won't make us great estimators, but it will give us insight into the rate of error, which can be tacked on to an estimate for the next project. This rate of error is to cover things that we don't know, and is unpredictable.
"Most software estimates are performed at the beginning of the life cycle. This makes sense until we realize that estimates are obtained before the requirements are defined and thus before the problem is understood. Estimation, therefore, usually occurs at the wrong time."
- Robert L. Glass, Facts and Fallacies of Software Engineering
Part of the problem, as Robert Glass points out in his book, is that we estimate at the wrong time, and it is often by the wrong people. The job of estimation should never lie with management, as it is unlikely that they will understand the true difficulty of the problem being solved. Management is influenced by timing to market, and other factors that have nothing to do with the real time involved to solve the problem.
"Awful suspicion: Projects that set out to achieve "aggressive" schedules probably take longer to complete than they would have if started with more reasonable schedules"
- Tom DeMarco, The Deadline
Even when we provide good estimates, we may be forced to follow an aggressive schedule, which can actually increase the time to deliver a product. It is up to management to provide an atmosphere that is productive and sustainable. If this isn't the case, it is likely the estimation is wrong no matter how well it had been assessed. This is one of those things that we don't know when estimating a project, so all we can do is rely on the error rate from past experience.
"...refrain from hiring implementers until the specifications are complete. This is what is done when a building is constructed."
- Frederick P. Brooks Jr. , The Mythical Man-Month
Nearly 35 years ago Frederick Brooks had a team of 10 architects and 150 implementers, and 7 months to complete the system specification. The architect's manager said that it would take 10 months, not 7 to compete the specification. The implementer's manager said that his team could complete the task on time. In an effort to save time and keep people busy, Frederick Brooks put the implementers on the task. It took the 150 implementers 10 months to complete the specification, not 7 as promised, and it was poor quality.

The lesson here is that dumping unqualified people on a project at the wrong time does not make it happen faster, in fact it costs more and hurts the quality of the project. It is important that when estimating a project it is not adequate to just throw out a number of man-hours (or days/months/years) to completion. We must take into account timing and required skills into our estimates.

I recall a certain staffing session where a project estimated to be around 450 man-days was being discussed. I don't think it is fair to say that people were assigned to the project with disregard for their skill level, but they were being given specific tasks that they didn't have a high level of knowledge in, thus requiring additional training time. The issue here is that management wanted to be able to assign a given task to any programmer and expected the estimate to remain the same.

I don't think that this scenario is uncommon, and given the nature of the management of a largish project, I think it would have been very difficult to handle differently. After all, if you only have a certain number of programmers available, you don't have a lot of choice. Perhaps the estimate would have been better if it had been detailed as to the specific knowledge required and the timing for adding programmers to the project. Perhaps then it would be realized that the estimate is based on the ideal staffing scenario, and would require adjustment if actual staffing differs.

In the end, there is no way to estimate a project with any precision. You need to estimate based on what you know, trusting your gut. Add to this your error rate based on past performance. Then estimate the project, and detail the staffing that it relies on. Make notes on why you estimated what you did, and make additional notes as the project proceeds, especially when you face the unexpected. When the project is over, compare your initial estimate with the actual time to completion, and make a note of the difference for inclusion into your error rate.

Good luck.

1 comment:

John Reynolds said...

Being a terrible estimator, i agree with a lot of this. The only way you can improve future estimates is to literally analyze the estimates vs. implementation times you did in the past. Because I usually underestimate, I usually now just take my estimate and double it. Managers don't like that too much though.

Also, a future post should be about "implmentation". Not exactly what you do, but why does it seem that it takes as much time to do 95% of the task as it does the last 5%?