Scott Alexander, on the Astral Codex Ten blog, makes a strong argument for AI getting better and better at forecasting – and now regularly competing with (and sometimes outperforming) human superforecasters. The idea is simple and logical: Given that AIs have been trained on vast amounts of data, with the right harness, they should be able to make predictions with calculated confidence levels.
Ironically, we’ve spent the last couple of years making AI refuse to have opinions, and now we’re building a certification layer specifically so it can – which also means that the question isn’t whether AI forecasters are accurate; it’s whether we’ve thought at all about who decides which opinions get to count as “calibrated” versus “biased.”
But the AI prediction markets of the future should be far superior to the human markets of the present. The biggest barrier to current markets is liquidity - there isn’t enough money riding on most questions to convince top superforecasters to drop their jobs at Google or the NSA in order to think hard about them and bet on them. But AI forecasters bring the cost of forecasting labor down to near-zero, so we can have hundreds of different AI agents betting on each question and be pretty sure its error has been driven down to the theoretical minimum. This, in turn, means we can vastly expand the number of questions, including (finally!) allowing randos to submit their own questions (probably with AI assistance in proposing un-rules-lawyerable resolution criteria). […] This, then, is my prediction for the AI superforecaster future: for basic questions, your off-the-shelf AI chatbot will be able to offer opinionated probabilities superior to those of any human. For more controversial or bias-laden questions, a new era of prediction markets will smooth over differences in brand and model and efficiently aggregate all AIs’ opinions.
