How accurate are prediction markets, really?

Prediction markets have a reputation for accuracy, and much of it is deserved. But the reputation gets oversold in both directions. Fans treat every price as gospel; skeptics point to a famous miss and dismiss the whole idea. The truth sits between the two, and it is more interesting than either. Prediction markets are very good under specific conditions, mediocre under others, and understanding the difference is the key to reading them well.

What accuracy actually means

The right way to judge a probabilistic forecast is calibration, not whether any single call was right. A forecast is well-calibrated when the stated probabilities match the observed frequencies over many events. Take everything a market priced at 70 percent. If the market is calibrated, close to 70 percent of those events happened and roughly 30 percent did not.

A 70 percent forecast that fails is not a broken forecast. If markets are calibrated, roughly three in ten of their 70 percent calls are supposed to fail. The failure is the model working, not the model breaking.

This is the single most misunderstood point about prediction markets. When an event priced at 30 percent happens, headlines call it a shock and declare the market wrong. But a 30 percent event is meant to happen roughly once every three times. You cannot evaluate a probability from one outcome. You need the whole distribution of calls, checked against how often they came true.

Judged that way, well-run markets on liquid questions tend to be strikingly well-calibrated. Their 20 percent events happen about a fifth of the time, their 80 percent events about four fifths, and so on across the range. Studies of the Iowa Electronic Markets, which have run US election contracts since 1988, are a well-known example: their prices frequently landed closer to the eventual result than the contemporaneous polls. For a plain-English primer on what that calibration means, see how to read a market-implied probability.

The track record against polls and pundits

The academic and practical record generally favors markets over the obvious alternatives, though with caveats.

• Versus polls. Polls are snapshots of opinion at one moment, subject to sampling error and turnout assumptions. Markets absorb polls as one input among many and update continuously, so they usually lead polls rather than lag them.
• Versus pundits. Commentators are rarely scored, so their confident predictions carry no cost. Markets price conviction in money, which filters out cheap talk.
• Versus statistical models. Here it is closer. Good quantitative models — the election forecasts published by outfits like FiveThirtyEight and The Economist, for instance — can match or beat markets, and markets often incorporate those models anyway. The market is best understood as an aggregator that folds models, polls and private information into one number.

None of this makes markets infallible. It makes them a strong default: on average, hard to beat, and transparent about their uncertainty in a way pundits never are. For how markets and polls diverge in practice, see prediction markets vs polls.

When markets are sharp

Accuracy is not uniform across every market. It depends on the conditions under which the price is set. Markets tend to be sharpest when three things line up:

• Liquidity. Deep markets with real trading volume have prices that many participants have stress-tested. A price backed by serious money is worth far more than one set by a handful of small bets.
• Clear resolution. The question has an unambiguous answer and an agreed source to settle it. Everyone knows exactly what they are trading.
• Enough time and attention. Markets that many people care about, running long enough for information to accumulate, converge on good estimates. A market that opened yesterday on an obscure topic has had little chance to find its level.

When all three hold, a market price is about as good a single-number forecast as you will find anywhere. Watching how that price moves is often more informative than the level itself, which is why sudden shifts are worth tracking. You can see which questions are moving fastest right now on the movers page.

The failure modes

A fair account of prediction markets has to name where they break down. The main failure modes are well understood, and we go deeper on each in can you trust prediction markets.

• Thin markets. Low volume means a price can be set by one motivated trader or a stale order. Wide spreads and tiny size are a warning that the number is fragile.
• Longshot bias. Very unlikely outcomes tend to be slightly overpriced and near-certain outcomes slightly underpriced. People overpay for a small chance at a big payout, so extreme probabilities can be a few points off.
• Manipulation. Because prices are public signals, someone with an agenda may push a price to influence perception. Deep, liquid markets shrug this off; thin ones can be moved, at least briefly.
• Ambiguous resolution. If the rules for settling a contract are vague, traders price in the uncertainty about how it will be judged, not just the underlying event. The number then measures two things at once and means less.
• Reflexivity and thin news. On quiet questions with little fresh information, prices can drift on sentiment rather than evidence.

The practical takeaway is to read the market, not just the price. A 65 percent on a deep, clearly-defined, heavily-traded question is a serious forecast. A 65 percent on a thin market with a fuzzy resolution rule is a rough guess dressed up as precision.

Reading them well

So how accurate are prediction markets, really? On the questions that matter most, the ones with liquidity, clarity and attention, they are among the best forecasting tools available, and they own their uncertainty in a way few sources are willing to. On thin or ambiguous questions, they are only as good as the market behind them, which sometimes is not very.

The skill is in telling the two apart, and in resisting the urge to judge a probability by a single outcome. If you want the fuller picture of how WillThisHappen sources prices and turns them into trackable probabilities, see how it works. The number is a starting point. Knowing how much to trust it is the real edge.