Adapted from pages 725-727 of Principles of Forecasting: A Handbook for Researchers and Practitioners (Armstrong 2001).
Managers should agree on an auditing process well in advance of the forecast review. They should then support the process. For example, they could provide forecasters with a check-list for standards and practices.
The Forecasting Standards Checklist at end the of this paper can help forecasters and decision makers examine the forecasting process. The checklist includes 16 areas with 140 principles. As a result, using it is no trivial matter. It includes a column (NA) to be checked for principles that are not applicable to the given situation. For example, some guidelines apply only to judgmental procedures. The checklist also includes a column labeled â€œ?â€ to indicate principles that seem applicable, but for which information is lacking.
The checklist is intended to provide ideas on ways to improve the forecasting process. Forecasters should examine the checklist prior to developing forecasting procedures. To insure objectivity, it may be sensible to have outsiders conduct an audit of the forecasting procedure. They could be forecasting experts from a different department in the same organization, or they could be specialists from outside of the organization.
To facilitate audits, forecasters should keep good records either in a notebook or in a computer log. People often argue about whether a forecasting method is reasonable, so a forecaster who keeps no record of the process of choosing a method might be accused of bias. A forecaster's notebook can protect the forecaster and the organization. Ideally, these records would be provided on a secure website.
The primary purposes of â€œStandards and Practicesâ€ are to help forecasters improve forecast accuracy, to better assess uncertainty, and to help decision makers to use the forecasts properly. They could also protect the forecaster. To our knowledge, no one has successfully sued a forecaster for making an incorrect forecast. However, some have successfully sued forecasters by showing that they did not adhere to best practice.
Beecham vs. Yankelovich illustrates some of the legal issues in forecasting. Beecham alleged that an inaccurate forecast for a new cold water detergent resulted in a $24 million loss. They claimed that Yankelovich used incorrect inputs to the forecasting models. In response, Yankelovich replied that Beecham failed to follow the marketing plan on which the forecasts were based because of changes in the advertising claims and reduced promotional expenses. This suit, which was settled out-of-court in September 1988, created much concern among research firms (Adweekâ€™s Marketing Week, December 7, 1987, pp. 1,4; Business Week, August 10, 1987, pp. 28).
An erroneous forecast of a severe drought in Yakima Valley in Washington caused farmers to undertake expensive actions with the cattle. The farmers took legal action against the U.S. Bureau of Reclamation to recover their losses. The government admitted making a mistake when they made an ill-advised subjective adjustment. However, it had been under no contractual agreement to provide a forecast. As a result, the court ruled against the farmers (Schinmann vs. U.S., 618 F. Supp. 1030, September 18, 1985), which was upheld by the appellate court (Schinmann vs. U.S., unpublished opinion, U.S. Court of Appeals for the Ninth Circuit).
In another case, four Massachusetts fishermen were lost at sea on November 21, 1980, because, their families claimed, of an incorrect weather forecast. Three families brought suit and won an initial judgment on the ground that the National Weather Service was negligent in failing to repair a weather buoy that could have provided useful data. The decision was overturned by the First U.S. Circuit Court of Appeals, and the U.S. Supreme Court refused to take the case (Brown vs. U.S., 599 F. Supp. 877, 1984, F.2d 199; 1st Cir., 1986). The issue was not the inaccurate forecast, but that the National Oceanic and Atmospheric Administration (NOAA) had failed to take reasonable steps to obtain accurate data. In addition, when it failed to obtain key information, NOAA did not notify users of this deficiency. The court ruled that there was no contractual requirement for the government to report on the process it used to make the forecast. Their ruling also implied that it is reasonable for forecasters to make tradeoffs between the cost of the forecast and its benefits. The reverse side of this is, if there is a contractual relationship, the forecaster should reveal the process.
In a British case, Esso Petroleum vs. Mardon (London, 1966 E. no. 2571), Mardon contracted with Esso to own and operate a gas station. A critical part of the negotiations was Essoâ€™s forecast that the station would sell 200,000 gallons of gas annually by the third year. Actual sales fell well short of the forecasted figure, and Mardon went out of business. Esso sued Mardon for unpaid bills. Mardon then countersued on the basis that the forecast misrepresented the situation. Esso had originally forecast the 200,000 gallon figure under the assumption that the gas pumps would face the road. After a zoning hearing, it had to change the stationâ€™s design so that the pumps were not visible from the road. Despite this unfavorable change, Esso used the original 200,000 gallon forecast in drawing up its contract with Mardon. Mardon won; the court concluded that Esso misrepresented the facts in this situation.
These cases imply that if you do not have a contract to provide forecasts, you are unlikely to be held liable. Furthermore, the courts recognize that forecasts involve uncertainty; making reasonable attempts to balance costs and benefits should provide forecasters with protection against lawsuits. Finally, forecasters can be held liable if it can be shown that they did not use reasonable practices to obtain forecasts, or if they intentionally used poor practices so as to bias the forecasts.
In addition to their use in legal cases, agreements on good standards of forecasting can be useful in auditing public projects. For example, does the government use adequate procedures to forecast the outcome of various projects for mass transportation, nuclear power plants, synthetic fuels, convention centers, and sports stadiums?
Forecasting Standards Checklist
The audit allows you to identify weak areas of your forecasting system. It asks you to first provide a brief description of the forecasting problem. You might have more than one type of forecasting problem: for example, your organization may need to forecast for inventory control, to forecast your competitor's actions, or to forecast the effects of a change in marketing variables such as price or advertising. You can conduct an audit for each type of problem. The system provides for security, although we provide no guarantee on security.
See the Introduction to the Forecasting Audit.
An alternative to using the Forecasting Audit Software, if you have a copy of Principles of Forecasting handy or are familiar with its content, is to use the Principles Checklist.
Forecasters, managers, and users of forecasts should be aware of the Legal Aspects of forecasting.
Please log in to the Forecasting Audit Software:
- If you do not have a username click here to create new account.
- If you do not remember a password click here to reset your password.