Gene Rowe, Institute of Food Research & George Wright, University of Strathclyde


Expert opinion is often necessary in forecasting tasks because of a lack of appropriate or available information for using statistical procedures. But how does one get the best forecast from experts?

One solution is to use a structured group technique, such as Delphi, for eliciting and combining expert judgments. In using the Delphi technique, one controls the exchange of information between anonymous panelists over a number of rounds (iterations), taking the average of the estimates on the final round as the group judgment.

A number of principles are developed here to indicate how to conduct structured groups to obtain good expert judgments. These principles, applied to the conduct of Delphi groups, indicate how many and what type of experts to use (five to 20 experts with disparate domain knowledge); how many rounds to use (generally two or three); what type of feedback to employ (average estimates plus justifications from each expert); how to summarize the final forecast (weight all experts’ estimates equally); how to word questions (in a balanced way with succinct definitions free of emotive terms and irrelevant information); and what response modes to use (frequencies rather than probabilities or odds, with coherence checks when feasible).

Delphi groups are substantially more accurate than individual experts and traditional groups, and somewhat more accurate than statistical groups (which are made up of non-interacting individuals whose judgments are aggregated).

Studies support the advantage of Delphi groups over traditional groups by five to one with one tie, and their advantage over statistical groups by 12 to two with two ties. We anticipate that by following these principles, forecasters may be able to use structured groups to harness effectively expert opinion.

Abstract of "Combining forecasts," J. Scott Armstrong - Full Text



To improve forecasting accuracy, combine forecasts derived from methods that differ substantially and draw from different sources of information. When feasible, use five or more methods. Use formal procedures to combine forecasts: An equal-weights rule offers a reasonable starting point, and a trimmed mean is desirable if you combine forecasts resulting from five or more methods. Use different weights if you have good domain knowledge or information on which method should be most accurate. Combining forecasts is especially useful when you are uncertain about the situation, uncertain about which method is most accurate, and when you want to avoid large errors. Compared with errors of the typical individual forecast, combining reduces errors. In 30 empirical comparisons, the reduction in ex ante errors for equally weighted combined forecasts averaged about 12.5% and ranged from 3 to 24 percent. Under ideal conditions, combined forecasts were sometimes more accurate than their most accurate components.

Keywords: Consensus, domain knowledge, earnings forecasts, equal weights, group discussion, rule-based forecasting, uncertainty.

Return to Practitioner's Page

Health Forecasting Page Master.

We are seeking a page master for this special interest group. If you are interested and have lots of ideas for the pages, please send your expression of interest to This email address is being protected from spambots. You need JavaScript enabled to view it..

Related Software

  • Modgen is the software used to create continuous time, microsimulation model similar to POHEM used by Statistics Canada. It is available free-of-charge. A comprehensive list of proprietary and free statistical software packages is available through The Econometrics Journal On-line.

Relevant Journals

Health Forecasting Page Master.

We are seeking a page master for this special interest group. If you are interested and have lots of ideas for the pages, please send your expression of interest to This email address is being protected from spambots. You need JavaScript enabled to view it..

Although health forecasting is an emerging field, many important articles have been published in the research literature.

Population and Demographic Forecasts

  • Numerous researchers have published extensively regarding methods of population forecasting. The International Journal of Forecasting dedicated an entire issue to population forecasting theory and practice. (Vol. 8, No. 3, November 1992)
  • R. D. Lee and L. R. Carter (1992), "Modeling and Forecasting U.S. Mortality," Journal of the American Statistical Association, 87 (419), 659-671.A recent evaluation of the performance of this forecasting model by one of the authors is available on the web.
  • Significant research has been done to assess the accuracy of official population forecasts and the underlying forecasting techniques in the United States and internationally. Several interesting articles are listed below.
    • J. M. Alho and B. D. Spencer (1985) , "Uncertain Population Forecasting," Journal of American Statistical Association, 80 (390), 306-314.
    • J. M. Alho and B. D. Spencer (1990), " Errors Models for Official Mortality Forecasts. Journal of American Statistical Association, 85 (411), 609-616.
    • H. Booth, J. Maindonald, and L. Smith (2002), "Applying Lee-Carter under conditions of variable mortality decline," Population Studies, 56, 325-336.

Infectious Diseases

  • A 2001 report published by the American Academy of Microbiology summarizes the state-of-the-field with regard to modeling of the relationship between climate and human health through changes in vector-born and infectious diseases.

Healthcare Workforce

  • The National Center for Health Workforce Analysis July 2002 report, Projected Supply, Demand and Shortages of Registered Nurses: 2000-2020 is an excellent example of health resource forecasting.
  • L. Greenberg and J. M. Cultice (1997), "Forecasting the Need for Physicians in the United States: The Health Resources and Services Administration's Physician Requirements Model," Health Services Research, 31 (6), 723-737.

Health Behaviors and Population Health

  • The Center for Disease Control and Prevention’s Smoking-Attributable Mortality, Morbidity, and Economic Costs (SAMMEC) software is a computational program used to estimate the number of annual deaths, years of potential life lost (YPLL), excess medical expenditures, and lost future productivity caused by smoking for the nation and states.
  • The Transdisciplinary Tobacco Use Research Center (TTURC) at the University of California, Irvineis developing a microsimulation model to estimate the impact of policy changes on smoking patterns.
  • This email address is being protected from spambots. You need JavaScript enabled to view it., of the Institute for Social Medicine at the University of Amsterdamdeveloped the Prevent microsimulation model in the late 1980’s that attempted to model the relationship between health behaviors such as smoking and long-term mortality rates.
  • Researchers at Statistics Canada have developed a continuous-time model, POHEM to assess the impact of different policy interventions or technologies on the health of the Canadian population.
  • M. Wolfson (1994), "POHEM – A framework for understanding and modeling the health of human populations," World Health Statistics Quarterly, 47, 157-175.
  • B. P. Will et al. (2001), " First do no harm: Extending the debate on the provision of preventive tamoxifen," British Journal of Cancer, 85 (9), 1280-1288.
  • Milton Weinstein at Harvard developed the Coronary Heart Disease Policy Model to model the impact of policy and technological advances on the incidence, prevalence, mortality from coronary heart disease, and changes in the cost of health care.
  • M. C. Weinstein et al. (1987), "Forecasting Coronary Heart Disease Incidence, Mortality, and Cost: The Coronary Heart Disease Policy Model," American Journal of Public Health, 77, 1417-1426 .
  • L. Goldman et al. (2001), "The Effect of Risk Factor Reductions Between 1981 and 1990 on Coronary Heart Disease Incidence, Prevalence, Mortality and Cost," Journal of theAmerican College of Cardiology, 38 (4),1012-1017.
  • D. Ruwaard et al. (1993), "Forecasting the Number of Diabetic Patients in the Netherlands in 2005," American Journal of Public Health, 83, 989-995.
  • Kenneth Manton at Duke University ’s Center for Demographic Studies has developed two health forecasting models to:
    • analyze discrete state health changes using population and vital statistics data
    • describe both discrete and continuous changes using data from longitudinally-followed community populations.
Both models can be modified based on expert judgment to deal with simulations of a multiplicity of possible interventions.

Health Care Spending

  • There are numerous examples of forecasting methods for health care expenditures in the United States and internationally. The first two articles critiques existing methods used by government actuaries to project future health expenditures and presents a probabilistic or stochastic models to assess uncertainty.
  • R. Lee and T. Miller (2002), "An Approach to Forecasting Health Expenditures, with Applications to the US Medicare System," Health Services Research 37 (5), 1365-1386.
  • R. Lee and S. Tuljapurkar (1997), "Death and Taxes: Longer Life, Consumption, and Social Security," Demography: 67-81.
  • T. E. Getzen and J. P. Poullier (1992), "International Health Spending Forecasts: Concepts and Evaluation," Social Science and Medicine 34 (9), 1057-1068. In projecting international health expenditures, the authors found that a combined method using econometric methods, exponential smoothing, ARIMA, and moving averages was more accurate than any single method according to mean absolute error (MAE).

 

Health Forecasting Page Master.

We are seeking a page master for this special interest group. If you are interested and have lots of ideas for the pages, please send your expression of interest to This email address is being protected from spambots. You need JavaScript enabled to view it..

The following forecasting methods are commonly used by practitioners to project future health and health care outcomes:

Judgmental

Statistical

Combining Forecasting Methods

Unaided Judgment Models

Although many health and health care forecasts use statistical methods to generate projections, most health forecasts are made by experts using unaided judgment. This technique is used when there is a low emphasis on accuracy of the forecast or very limited resources are available.

Structured Judgment Models

The Delphi technique is used often in the health sector when there is insufficient or unreliable data to conduct a statistical forecast. Projections developed by Delphi panels are believed to be more accurate than forecasts based on unaided judgment. There is limited direct evidence of the accuracy of forecasts using the Delphi method.

Extrapolation Methods

Population and demographic forecasts regularly analyze vital statistics data to project future population growth and demographic shifts by age, gender, and race/ethnicity using time series methods, but current practice is not uniformly applied or described. Demographers also forecast changes in a given population based on changes in birth rates, death rates, migration rates, and other factors using rules-based forecasts.

Multivariate Forecasting Models

When considerable data are available and causal relations between the determinants of health and health outcomes are reasonably well understood, multivariate models can be used to project future health outcomes. These requirements are rarely met in the health sector. One area where both requirements are met is infectious disease modeling.

Econometric Forecasting Models

The most common statistical health care forecasts use econometric models because important data needed for multivariate models are missing. Practitioners must rely on economic theory to describe the relations between important variables. Econometric models have been used in health to project:

  • Effects of changes in policy on hospital profits, cash flows and balance sheets
  • Distribution of health insurance coverage, health care spending, and health care utilization under Federal policy changes
  • Demographic composition, geographic distribution, and the distribution of health professionals by specialty resources. (Example)
  • Other researchers have attempted to project future health impacts of changes in behavioral risk factors that have been associated with disease. (Example)
  • Microsimulation models have been used to project the impact of policy and behavioral changes on population health for more than two decades using econometric techniques. (Example)

Combining Forecasting Methods

For most health conditions, long-term forecasts are needed to model associations between risk factors and final outcomes. For example, the health benefits of reductions in smoking initiation rates may only be seen 20 years later. The complexity of the factors that are known to influence health outcomes and interest in distributional effects argue for multivariate approaches to forecasting. Limitations in available data sets, however, suggest that econometric models or judgmental bootstrapping may produce more accurate forecasts. Given these considerations, it is likely that projections of future population health will based on a combination of forecasting techniques including time series models, multivariate analyses, econometric models, expert judgments such as the Delphi method. Results from the different approaches will be compared and combined. Combining forecasts using different methods has been shown to improve forecasting accuracy. (Example)