What's new

Legal aspects of forecasting

Forecasters have in the past been sucessfully sued for failure to follow best practice. This site provides guidance on best forecasting practice in the form of conditional evidence-based principles. To assess whether your, or someone else's, forecasting procedures are consistent with best practice, consider using the Forecasting Audit Software from here, or from the button near the top of this page. The Software can be used to help design forecasting procedures or to evaluate procedures that have been used before.

Is closing down coal mining good policy?

In Australia the Greens political party have advocated closing down the coal mining industry. This policy is based on implicit forecasts that the policy would help to avoid dangerous manmade global warming and that it would cause little or no harm to the economy. We have addressed the un-scientific nature of global warming forecasts elsewhere, and have shown that a no-change forecast of global mean temperatures is sufficiently accurate for policy making. Sinclair Davidson and Ashton De Silva show in their paper "Costing of The Greens’ Economic Policies: Mining", that the Greens' forecasts of the economic effect of closing down coal mining are not supported by proper analysis.

Carbon credits policy forecasting failure

How do anti-carbon dioxide climate policies work in practice. Here is the account of one policy implementation, in Indonesia, that was targeted at an area of tropical rain forest. We don't think those responsible for the policy used scientific forecasting methods to predict the outcomes.

Talks on climate policy at the Prague Symposium on Forecasting

Kesten Green will be talking about findings to date from the Global Warming Analogies Forecasting Project and Scott Armstrong will be talking about why the global warming alarmist message has been persuasive. Using evidence-based principles from advertising, Armstrong predicts the alarmist message will become less persuasive as people become more involved and learn more about climate forecasting. Their talks are part of a special session at the International Symposium on Forecasting on 27 June. A copy of Kesten Green's talk is available here, and a copy of Scott Armstrong's is available here.

Myth of killer mercury

The Environmental Protection Agency recently issued 946 pages of new rules, requiring that U.S. power plants sharply reduce the already low emissions of mercury and 83 other air pollutants. Willie Soon and Paul Driessen write that extensive evidence and clinical studies challenge the EPA's costly regulatory agenda. more...

Experts face manslaughter charges over forecasts

Three-hundred-and-eight people died when a 6.3-magnitude earthquake hit central Italy in 2009. Seven experts have now been charged: it is alleged that they failed to sufficiently warn residents. The story raises some important issues for public policy forecasting. more...

Polar bear numbers stable

The Polar Bear Specialist group have report no change in the bear population over the last 4 years, a result consistent with the 2008 forecast of Scott Armstrong, Kesten Green, and Willie Soon. more...

House Science Committee hearing on climate change

At a House Science Committee hearing on "Climate Change: Examining the Processes Used to Create Science and Policy" on 31 March 2011, Scott Armstrong presented findings and conclusions from research on forecasting for climate change. On the basis of the research findings, he recommended: (1) End government funding for climate change research, more...

Costs of German environmentalist policies

"Germany is among the world leaders when it comes to taking steps to save the environment. But many of the measures are not delivering the promised results. Biofuels have led to the clear-cutting of rainforests, plastics are being burned rather than recycled and new generation lightbulbs have led to a resurgence of mercury production". Read the 7-page SPIEGEL survey detailing failures to properly forecast all costs and benefits of environmentalist policies here...

IARPA forecasting competition

The Intelligence Advanced Research Projects Activity (IARPA) have invited experts to take part multi-year research program investigating the accuracy of individual and group predictions about global political, economic and military developments. More...

Costs and benefits of energy sources

Paul Driessen wondered whether policy makers are properly comparing evidence-based forecasts of all costs and benefits of alternative energy sources... and concluded that they are not. His article is available on the Global Warming Audit page, here.

Forecasting for foreign policy

In a Wharton Magazine blog, Scott Armstrong and Kesten Green describe problems with much current foreign policy decision making based on expert judgments. More importantly, they provide solutions in the form of evidence-based methods than can provide better forecasts of the effects of alternative policies. The blog is here, and a printable version is available here.

Betting on the climate

Three years ago, Scott Armstrong challenged Al Gore to bet the IPCC forecast of 0.03 C per annum global average warming would be more accurate over a ten year period than his evidence-based forecast of no-change. After some months of access troubles, theclimatebet.com is back up with a summary of the first three years of how the bet would have gone had Mr. Gore accepted, here.

Mercury in fish alarm description added to analogy list

Expectant mothers were urged not to eat fish with the prediction that the brain development of infants would be harmed. The forecast was contrary to the evidence provided by the intelligent and long-lived high fish eating populations such as the Japanese and more. A description by Soon and Ferguson of the "Mercury in fish" analogy to the alarming forecasts of man-made global warming here. Click on the 24th item in the table of analogies to see the description.

Global warming alarm analogy description: Population growth and famine

We have added a description of the "Population growth and famine (Meadows)" analogy to the alarming forecasts of man-made global warming. You can find it on the Global Warming Analogies Forecasting Project page (left menu bar or click here). Click on the 13th item in the table of analogies to see the description. You may have read it before: the description comes from Julian Simon's 1996 classic The Ultimate Resource 2, available online, here.

Forecasting the monsoon: Good science, or false hope?

Willie Soon and Madhav Khandekar examined a report titled "Climate change and India: A sectoral and regional analysis for the 2030s" released before the annual climate summit in Cancun, Mexico. They found the claims of the reports' authors that their computer models would soon be able to predict the timing and intensity of monsoon rainfall during the 2030s lacked credibility. Read the full story, here, on the Global Warming Audit page.

EPA regulation proliferation: Are they based on proper forecasts?

In a November 22 2010 article, the Wall Street Journal lamented the lack of proper processes behind the Environmental Protection Agency's proliferation of major regulations and policy rules. We suggest that the methods used to forecast the cost and benefits of such policies should be audited using our Forecasting Audit Software. We will happily list audit findings on these pages. Read the full story, here, on the Global Warming Audit page.

Bad forecasts don't justify wild spending and crazy regulation

Bjorn Lomborg says that we should accept forecasts of dangerous manmade global warming as valid but question spending recommendations. Scott Armstrong, Kesten Green, and Willie Soon in a letter to the Wall Street Journal argued that we should question the forecasts and demand evidence before countenancing any new spending or regulation.

Julian Simon's "Bad environmental and resource scares"

With the kind permission of the Simon family, we have provided an extract from Julian Simon's The Ultimate Resource 2 describing the alarming forecasts and the actual outcomes of environmental and resource scares over the, mostly recent, years. You can find a link from the left menu bar, or here.

Evaluation of Aussie emissions targets

Roger Pielke, Jr. of the University of Colorado has conducted an evaluation of Australia's proposed emission reduction targets and finds their achievement... unlikely. See the Global Warming Audit Page for details.

New draft analogy description for asbestos and lung disease

We have added a draft description of the asbestos and lung disease analogy. As with all our analogy descriptions, it is a work progress. We invite and welcome help in preparing analogy descriptions, and corrections and suggestions for the descriptions we have prepared already. See the Global Warming Analogies Forecasting Project page.

AFR's Mark Lawson asks why governments haven't demanded an audit of IPCC forecasting methods

Mark Lawson, senior journalist on the Australian Financial Review and author of the book A guide to climate change lunacy - bad forecasting terrible solutions, has written an article wondering why so little effort has been made to check the IPCC's climate forecasts. See his article on the Global Warming Audit page.

Tampa development forecasting audit

We have added a new audit page to the SIG on a Wharton student forecasting audit of the City of Tampa, FL, development forecasting procedures.

"Citizens' audit" of the Intergovernmental Panel on Climate Changes report

We have added a link to the NoConcensus.org site's citizens' audit of the United Nations' IPCC "dangerous manmade global warming" report. The link can be found on the Evidence-based Policy Sites and Publications page.

Researchers submission on the lack of scientific forecasting behind the U.S. State Department's latest "Climate Action Report"

Scott Armstrong, Kesten Green, and Willie Soon made a submission titled "Global Warming Alarm Based on Faulty Forecasting Procedures" on the State Department's report:

"Our research findings challenge the basic assumptions of the State Department’s Fifth U.S. Climate Action Report (CAR 2010). The alarming forecasts of dangerous manmade global warming are not the product of proper scientific evidence-based forecasting methods. Furthermore..."

Climate models, forecasting, and politics with legislators in Washington on 9 April

Scott Armstrong will be discussing the value of forecasts of global warming and where things are headed in a talk titled "Effects of the global warming alarm: An application of the structured analogies method" at the National Conference of State Legislatures in Washington, D.C. at 1:15 pm on 9 April, 2010.

Global warming alarm analogies forecasting project page now up

Is it possible to make useful forecasts about social phenomena such as the alarm over predictions of dangerous manmade global warming? Kesten Green and Scott Armstrong reasoned that it should be possible to do so using the structured analogies method. (In their 2007 paper, Green and Armstrong found that structured analogies provided forecasts that were more accurate than experts’ judgments when applied to the difficult problem of forecasting decisions in conflicts.) The Global Warming Analogies Forecasting Project page is now up on the Public Policy Forecasting SIG pages (PublicPolicyForecasting.com). The page includes a link to the Project working paper and lists 26 situations that have been identified as analogous to the current alarm. To date there are descriptions of six of them available. Kesten and Scott welcome evidence and analysis, especially if it contradicts their own efforts to date.

Have policy makers properly forecast all the costs and benefits of policies intended intended to reduce human CO2 emissions?

Two senior climate scientists think not. To read more about Harvard Astrophysicist Willie Soon and Delaware State Climatologist David Legates analysis of evidence on to what extent people can influence climate, the costs and benefits of changes in climate, and whether carbon trading schemes would lead to CO2 reductions, see the item on the Global Warming Audit page.

History shows manmade global warming alarm to be false – but that harmful policies will persist.

Using a forecasting method that they have developed, Dr. J. Scott Armstrong from the Wharton School and Dr. Kesten C. Green from the International Graduate School of Business at the University of South Australia conclude that alarm over “dangerous manmade global warming” is the latest example of a common social phenomenon involving alarming but unscientific forecasts that prove to be wrong. more...

Benchmark forecast for global average temperatures: No change for 100 years

24 October 2009 - In their new paper in a special section of the International Journal of Forecasting on decision making and planning under low levels of predictability, Kesten Green, Scott Armstrong, and Willie Soon asked whether it is possible to make useful forecasts for policy makers about changes in global climate up to 100 years ahead. Their findings and conclusion will surprise many people. more...

Letter to Environmental Protection Agency regarding regulation of greenhouse gases

8 October 2009 - Thirty-five scientists signed a letter addressed to the EPA Administrator, the Honorable Lisa P. Jackson, on October 7 2009 expressing concern that proposed rulemaking on the regulation of greenhouse gases would be based on unscientific forecasting procedures. more...

Climate policy, temperatures, CO2, and alarms

16 July 2009 - Scott Armstrong, Kesten Green, Andreas Graefe, and Willie Soon examined climate change forecasting for public policy decision making in a paper presented at the International Symposium on Forecasting in Hong Kong in June. They found official forecasting efforts unscientific and presented some surprising findings that should help decision makers make decisions that will be better for almost everyone. more...

Climate Change Reconsidered: NIPCC Report

15 July 2009 - Craig Idso and Fred Singer with the help of a team of expert contributors and reviewers have compiled a report of more than 700 pages that examines the evidence on climate change and effects, that provides useful information for forecasters. more...

Forecasting and scheming to trade CO2

11 May 2009 - A Select Committee of the New Zealand government are hearing submissions on the Emissions Trading Scheme. Among the submissions are one by Bob Carter and another by Kesten Green that argue that the Scheme is based on unscientific forecasts of climate change and the cost of the scheme. more...

Public sector forecasting at the ISF

30 April 2009 - Eighteen papers on forecasting for public sector problems have been accepted for the International Symposium on Forecasting in Hong Kong (June 21-24). The papers address forecasting issues in education, employment, hospitals, performance, population, terrorism, and transport. The hot topic is climate and related matters, about which there are eight papers.

BBC interview on polar bear population and climate forecasts

30 April 2009 - Scott Armstrong was interviewed on the BBC "Five Live" show about forecasting the population of polar bears and climate while he was attending the International Conference on Climate Change in New York. more...

Dissenting analysis of climate policy

29 April 2009 - Former OECD Head of Economics and Statistics David Henderson delivered his contrarian analysis at the Oxford Business and Environment Conference "Beyond Kyoto: Green Innovation and Enterprise for the 21st Century". more...

Forecasting the economic impact of new policies

25 April 2009 - There has been little research on how best to forecast the effectiveness of altenative strategies for implementing government policies. It seems reasonable to assume that better forecasts of the effects of policy implementation would lead to better better policies, and better implementation of those policies. Nicolas Savio and Konstantinos Nikolopoulos propose in their working paper an approach that combines two evidence-based forecasting methods: strunctured analogies (Green and Armstrong 2007) and econometric modelling.

Noconsensus.org Citizen Audit of the IPCC report

“This Citizen Audit focused its attention on the peer-reviewed literature claim. A team of 43 volunteers from 12 countries examined the list of references at the end of each chapter...”

The Global Warming Policy Foundation

“…to bring reason, integrity and balance to a debate that has become seriously unbalanced, irrationally alarmist, and all too often depressingly intolerant.”

Frontier Centre for Public Policy

“…to broaden the debate on our future through public policy research and education and to explore positive changes within our public institutions that support economic growth and opportunity.”

Center for Policy Research at the University at Albany

“…research in decision and policy sciences that will ultimately contribute to the improvement of policy processes and decision making at all levels of society.”

The Campbell Collaboration:

“…aims to help people make well-informed decisions about the effects of interventions in the social, behavioral and educational arenas.”

Coalition for Evidence-based Policy:

“…promote government policymaking based on rigorous evidence of program effectiveness.”

Science and Environmental Policy Project (SEPP) and The Week That Was (TWTW):

“…sound, credible science must form the basis for health and environmental decisions.”

Evidence and Policy Journal:

“…dedicated to … the relationship between research evidence and the concerns of policy makers”

Science and Public Policy Institute (SPPI):

“…public policies for energy and the environment rooted in rational science and economics.”


Tell us about other sites and publications that are concerned with evidence-based public policy: Email This email address is being protected from spambots. You need JavaScript enabled to view it.

Kesten C. Green & J. Scott Armstrong

August 9, 2008

The precautionary principle is a political principle, not a scientific one. The principle is used to urge the cessation or avoidance of a human activity in situations of uncertainty, just in case that activity might cause harm to human health or the natural environment. There is an interesting discussion of the history of the term in Wikipedia.

In practice, the precautionary principle is invoked when an interest group identifies an issue that can help it to achieve its objectives. If the interest group is successful in its efforts to raise fears about the issue, the application of the scientific method is rejected and a new orthodoxy is imposed. Government dictates follow. People who dissent from the orthodox view are vilified, ostracized, and may have their livelihoods taken away from them.

Consider the case of "climate change'. Warnings of dangerous manmade global warming from scientists, politicians, and celebrities have received much publicity. They admonish us to dramatically reduce emissions of CO2 in order to prevent disaster over the course of the 21st Century. Efforts have been made to stifle a scientific approach to the issue. In an article titled "Veteran climate scientist says 'lock up the oil men'", James Hanson, who heads the NASA Goddard Institute for Space Studies, was quoted as suggesting that those who promote the ideas of global warming skeptics should be "put on trial for high crimes against humanity." The skeptics themselves have been ejected from, for example, State Climatologist positions and prevented from publishing research in mainstream journals, and they and their views are routinely attacked.

Much complexity and uncertainty surround climate change. The cumulative empirical evidence on proper forecasting procedures suggests that the most appropriate method in this case is naive extrapolation. In simple terms, this means to forecast no change. Of course there will be change, but with current knowledge there is no more reason to expect warming than to expect cooling.

As we describe in our paper, we have been unable to find any forecast derived from evidence-based (scientific) forecasting methods that supports the contention that the world faces dangerous manmade global warming.

Appeals for urgent curtailment of human activity "just in case" are often couched in ways that imply that industrial societies are inherently sinful, rather than that there might be a problem to be dealt with. Indeed, interpretation of the precautionary principle is subjective and it is arguable that it is being misapplied to the issue of climate change.

Firstly, even if forecasts of increasing temperatures turned out to be accurate, predicted temperatures and other conditions are within the range of variations that have been experienced in the past. There is no evidence that the natural environment "prefers" relatively cool to relatively warm average temperatures. In fact, life in general prefers warmth.

Secondly, curtailing human activity would harm people's health by making them poorer than they would otherwise have been. This is likely to be the case even if curtailing human activity happened to reduce global average temperatures. When the situation is framed in this way, the precautionary principle dictates that it is policies to curtail economically efficient human activity that should themselves be curtailed.

The outlook for the climate over the 21st Century is highly uncertain. There is a word in the English language to express high uncertainty. That word is "ignorance". And ignorance is not a basis for responsible government action. We should expect our politicians to have the courage to resist interest groups' calls for action in the face of ignorance. The precautionary principle brings to mind the slogan on the Ministry of Truth building in George Orwell's 1984: "Ignorance is Strength". Instead of this political principle, we hope that politicians will turn to scientific principles for making public policy.

Kesten C. Green & J. Scott Armstrong

October 1, 2008

Assume that you are considering investing $700,000 and that this is a large sum for you. Experts give you the names of two reputable investment houses of long standing. When you visit the first of these, Benjamin Company, Inc., the manager recommends Bailout Bonds as an excellent investment for you. He gives you good reasons for his recommendation and reassures you that he is very confident in his advice, which is based on the analysis of top experts. He urges you to act swiftly to avoid missing out.

As a cautious investor, you visit the other investment house that was recommended to you, Franklin Company, Inc., their manager advises you that Bailout Bonds would be a poor investment and that you would be better off holding on to your money than putting such a large amount into such a speculative investment. Like Benjamin’s manager, Franklin’s manager provides good reasons and she too is confident in her advice.

Given the conflicting advice, you decide to check the track record of each of these investment houses. Fortunately, numerous academic studies have been published on the accuracy of each firm’s investment recommendations since 1930. Each firm retains the advice of some of the best experts in the world. You find that on average, the Benjamin experts have been right for 50% of their forecasts – and wrong on 50%. Interestingly, the Franklin experts have exactly the same performance record.

What would you do? Our guess on what most small business people would do is to skip this investment opportunity.

The year 1930 is roughly the date of the first studies on the value of expert forecasts that relate to complex and uncertain situations. Contrary to popular opinion, the findings are consistent with our Benjamin versus Franklin story. For example, Philip E. Tetlock’s 2005 book, Expert Political Judgment describes how he recruited 284 people whose professions included “commenting or offering advice on political and economic trends.” He asked them to forecast the outcomes for various situations. By 2003, he had accumulated 82,361 forecasts.

The conclusions from the studies are that:

1) expert opinions are useless for forecasting related to complex and uncertain situations.

2) expertise does not help; College students do as well as seasoned experts at such forecasting.

3) experts’ statements about confidence have virtually no value. Indeed, if you put a group of experts in a room and have them make forecasts, their confidence goes up rapidly, but this has no relationship to accuracy.

The country faces a similar problem. But our leaders in Washington are not debating about their own investments. Instead, they are thinking about how to spend other people’s money. They have provided no scientific basis for their decision and they show no awareness of how one should properly approach such a forecasting problem.

There are ways to study this problem, but they should not be done in a rush, and they should not be done in group meetings.

We don’t know what will happen, but we do know what procedures to use to obtain scientific forecasts of the outcomes of various plans. Researchers in the field have been trying to spread the word on scientific (evidence-based) forecasting by making forecasting knowledge easily and freely available to others at http://forecastingprinciples.com

The question for our leaders is whether they should invest $700 billion when, despite their confidence, they are completely ignorant of the outcome of the investment plan.

Dr. J. Scott Armstrong. Professor, The Wharton School, University of Pennsylvania
Dr. Kesten C. Green. International Graduate School of Business, University of South Australia.

J. Scott Armstrong, Kesten C. Green, and Willie Soon (2008)
"Polar Bear Population Forecasts: A Public-Policy Forecasting Audit." Interfaces (with commentary), 38, 382-404.

Polar bear fears groundless (31 March 2008)

The U. S. government commissioned studies to support the listing of polar bears as a threatened or endangered species. Polar bear numbers are currently high and the population has been increasing rapidly in recent decades. Everyone likes polar bears, so this is good news. A decision to list would require forecasts that the current upward population trend will reverse. The government studies concluded that polar bear populations would decrease substantially.

Decision makers and the public should expect people who make forecasts to be familiar with the scientific principles of forecasting just as a patient expects his physician to be familiar with the procedures dictated by medical science. Three scientists, J. Scott Armstrong of the University of Pennsylvania, Kesten Green of Monash University, and Willie Soon of The Harvard-Smithsonian Center for Astrophysics, audited the government studies to assess whether they were consistent with forecasting principles. Their paper, “Polar Bear Population Forecasts: A Public-Policy Forecasting Audit,” has been accepted for publication in the management science journal Interfaces. It is the only peer-reviewed paper on polar bear population forecasting that has been accepted for publication in an academic journal.

They concluded that the government forecasts were based on false assumptions and their polar bear population forecasts contravened many principles for scientific forecasting. Indeed, the reports followed fewer than one-sixth of the relevant principles. Given the importance of the forecasts, all principles should be properly applied. In short, the government reports do not provide relevant information for this decision.

Research shows that for issues such as the population of polar bears—situations that are complex and where there is much uncertainty—the best forecast is that things will follow a “random walk;” in effect, this model states that the most recent value is the best forecast for all periods in the future. Because the polar bear population has been increasing over recent decades, however, a continuation of that trend over the short term is possible.

Polar bears listed as threatened under the Endangered Species Act in defiance of scientific evidence (14 May, 2008)

“Secretary of the Interior Dirk Kempthorne announced on May 14, 2008 that he is accepting the recommendation of U.S. Fish and Wildlife Service Director Dale Hall to list the polar bear as a threatened species under the Endangered Species Act (ESA). The listing is based on the best available science, which shows that loss of sea ice threatens and will likely continue to threaten polar bear habitat. This loss of habitat puts polar bears at risk of becoming endangered in the foreseeable future…”. See the U.S. Department of the Interior website for the full announcement.

This extraordinary announcement is at odds with evidence that the polar bear population is currently thriving, and is based on false assumptions and unscientific forecasting procedures. The forthcoming Interfaces paper by Armstrong, Green, and Soon, provides evidence that the “best available science” does not support a listing.

BBC “Five live” Interview with Scott Armstrong during the International Conference on Climate Change (9 March, 2009)

Host: The New York conference “Global Warming: Was It Ever Really A Crisis?” has been described as the world’s largest ever gathering of global warming skeptics. Professor Scott Armstrong from Wharton School of Management at the University of Pennsylvania and a world renowned forecaster has turned his attention to the global polar bear population. Professor Armstrong, hello, welcome to Five Live. more... Audio.

No change in polar bear population (May, 2011)

The Polar Bear Specialist Group of the International Union for the Conservation of Nature (IUCN) has reported that there was no change in the polar bear population in the most recent four-year period studied.

The finding is consistent with the conclusion of a 2008 paper by Scott Armstrong, Kesten Green, and Willie Soon (“Polar Bear Population Forecasts: A Public-Policy Forecasting Audit”) that “the inconsistent long-term trends in the polar bear population suggest that it is best to assume no trend in the long-term.”

The polar bear population finding contrasts with Senator Boxer’s hearings in January 2008 in which she expressed the view that the number of polar bears would decline rapidly. Professor Armstrong offered to bet her that the number of polar bears would not decline, but she did not respond to the challenge.

The Polar Bear group’s report can be found here.

The Armstrong, Green, and Soon paper on polar bear population forecasts can be found here.