Measuring and analyzing estimation error are the basis of estimation learning related activities, such as deciding whether or not an organization has an estimation problem, identifying risk factors related to project performance in software development, and, evaluating and improving estimation and uncertainty assessment methods and tools.

The most commonly used measure of estimation error is the Magnitude of Relative Error (MRE). MRE is identical to the measure called FASE in other branches of forecasting research. The mean MRE (MMRE) is often used to average estimation error for multiple observations. It is not unproblematic to use MMRE as a measure of estimation accuracy, and several other measures, such as PRED and MER, is sometimes used. However, all estimation error measures have shortcomings. Hence, the measure that should be used in any given case depends on the context

Many factors can affect the measured estimation error. Measuring estimation error without a clear understanding of which factors contributed most to the estimation error, e.g., without an understanding of whether a high estimation error is caused by the factor “low estimation ability” or “high estimation complexity”, is rarely meaningful.

Papers that report estimation surveys. (Links not currently supported.)

Papers on estimation error measures and analysis. (Links not currently supported.)

See also Evaluation Methods in Principles of Forecasting: A Handbook for Researchers and Practitioners.

There are two different sources that cause uncertainty regarding effort usage in software development projects: (i) The inherent uncertainty regarding cost usage in a software project, and (ii) The uncertainty caused by lack of knowledge about the project. However, uncertainty can be reduced by flexibility of outcome and software development process.

Accurate assessment of the uncertainty of software development effort estimates is an important part of software estimation, e.g., when deciding whether or not to embark up a project, to support the bidding process, to support decisions about how large the project’s contingency budget should be.

Clearly, the use of cost uncertainty assessments that reflect the underlying uncertainty will improve the budgeting process. Unfortunately uncertainty assessments are usually over-confident about the accuracy of cost estimates.

Papers on uncertainty assessments. (Link not currently supported.)

See also Assessing Uncertainty in Principles of Forecasting: A Handbook for Researchers and Practitioners.

A plethora of estimation methods exists and there are several schemas for classifying them. Here we classify estimation methods in "expert estimation" and "formal estimation models".

- "expert estimation" is used as a label for estimation methods in which a significant part of the estimation process (particularly the final step, i.e., the “quantification step”) is based on intuition. Expert judgment-based estimation is not a single estimation method, but a spectrum of judgment-based processes. Even within the same software development organization, processes of expert judgment-based estimation may vary from those based on pure intuition to highly structured processes that use relevant historical data.

- "formal estimation models" is used as a label for estimation methods where a substantial part of the estimation (and particularly the "quantification step") is based on the use of mechanical and analytical processes, e.g., the use of a formula derived from historical data using regression analysis.

Software effort estimation research is inconclusive regarding which estimation approach is better, e.g. a recent review of studies comparing models and experts in software development effort estimation concludes that experts typically performs no worse than the models. One reason for this may be that it seems to be difficult to develop meaningful estimation models that do not require a high degree of expert judgment as input to the models in the first place.

Papers on estimation by estimation method (Links not currently supported):

 

See also the Methodology Tree in Principles of Forecasting: A Handbook for Researchers and Practitioners.