Guessing Games

Overconfidence

Predictive overconfidence often leads to misguided decisions, as illustrated by the story of Archie Cochrane, a pioneer in evidence-based medicine. He mistakenly trusted a specialist's intuition over empirical evidence, resulting in unnecessary surgery 1. This example highlights the dangers of substituting intuition for rigorous analysis. Philip E. Tetlock compares this to political pundits who use vague language to hedge their predictions, avoiding accountability 2.

Pundits didn't know as much about the future as they thought they did.

--- Philip E. Tetlock

Such overconfidence can lead to significant errors in judgment, emphasizing the need for a more disciplined approach to forecasting.

   

Biases

Cognitive biases, such as hindsight bias, significantly impact our ability to make accurate predictions. Shankar Vedantam discusses how people often forget their incorrect forecasts, believing they were right all along 3. This bias, along with the tendency to attribute success to skill rather than luck, distorts our understanding of prediction accuracy.

Many people who get the right answer are just lucky, and some people who guess wrong are just unlucky.

--- Shankar Vedantam

To truly assess predictive skill, it's crucial to evaluate a large number of predictions over time, as demonstrated by a federal forecasting tournament.

   

Limits

Long-term predictions often fall prey to triviality and inaccuracy. Philip E. Tetlock notes that forecasting tournaments can become trivial pursuits if not carefully managed, with accuracy diminishing over longer horizons 4. Despite this, such tournaments remain valuable for honing forecasting skills.

Forecasting accuracy falls virtually to chance to the dart throwing chimpanzee when you try to forecast more than about ten years out.

--- Philip E. Tetlock

Interestingly, non-experts sometimes outperform professionals in these settings, challenging assumptions about who can make accurate predictions 5.