Startups Taking on 'Future Forecasting' With AI [Tech Talk]

AI Ranks 8th in Future Forecasting Competition for the First Time
Mantic AI, Founded by Former DeepMind Scientist
Collective Intelligence AI Narrowing the Gap With Human Intuition
Aiming to Automate Future Forecasting With Machine Speed

In the United States, there is a future forecasting competition called the "Metaculus Cup." Each quarter, the person who makes the most accurate predictions about the future is selected and awarded a prize. Since its inception in 2015, hundreds of experts have participated in this competition every year. In the most recent event, artificial intelligence (AI) made headlines by entering the top 10 for the first time.

AI Breaks Into the Top 10 of a Future Forecasting Competition for the First Time

The Metaculus Cup poses questions on all kinds of events happening around the world. For example, questions from the autumn Metaculus Cup, which began last month, include: "Will there be a political protest in France this year involving more than 600,000 participants?", "Will the Palestinian militant group Hamas release all hostages by the end of this year?", and "Will the United States provide bailout funds to Argentina this year?" There are a total of 30 such questions.

The prediction competition "Metaculus Cup," where future forecasting experts from around the world gather to compete. Metaculus homepage

Because the topics are so wide-ranging, participants' professions, expertise, or general knowledge are of little help. Contestants must answer all questions relying solely on the brief background information and related materials provided by Metaculus. The top three participants who get the most answers right share the prize. In other words, the Metaculus Cup is a competition to identify the best future forecaster who can make the most accurate judgments with limited information.

Can AI surpass humans in future forecasting? In the first quarter of this year, an AI developed by Mantic entered the overall top 8 of the Metaculus Cup for the first time. Mantic is a startup founded by AI scientists such as Toby Shevlane, formerly of Google DeepMind, and is conducting research to develop an AI that can outperform humans in future forecasting.

AI Catching Up to Human Intuition and Insight Through Collective Intelligence

Until now, future forecasting has been a domain dominated by humans. The biggest obstacle is incomplete information. Future predictions are made when relevant data has not yet been fully collected. While this is a critical weakness for AI, whose performance depends on data quality, it is also when human intuition and insight shine brightest.

Screen of the betting platform Polymarket predicting the 2024 U.S. presidential election. Polymarket

Mantic AI overcame this disadvantage through "collective intelligence." Instead of a single large AI model solving the problems, dozens of smaller models each make judgments, and their average is used to predict the future. Interestingly, collective intelligence forecasting has consistently ranked in the top 4 or 5 in the actual Metaculus Cup each year, making it a highly effective method. Online betting on sports outcomes and platforms like Polymarket, which offer betting opportunities on various events, are also examples of collective intelligence forecasting.

Automating Future Forecasting With Speed Beyond Human Capability

Future forecasting has been a subject of research for decades in advanced countries such as the United States and the United Kingdom.

It began with Philip Tetlock, a professor at the University of Pennsylvania, who in 2005 launched the "Good Judgment Project," which aimed to predict various future events using only limited information. The Good Judgment Project attracted the attention of the U.S. and U.K. governments. Both countries have since conducted future forecasting training for senior officials, diplomats, and intelligence officers through the INFER (Integrated Forecasting and Risk Estimation) project and the Cosmic Bazaar project, respectively.

The INFER project and the Cosmic Bazaar project were integrated into a joint project by the two countries in 2023, and are expected to aid future national policy decisions. The RAND Corporation, a U.S. public policy think tank, stated, "By training thousands of forecasting experts and gaining useful perspectives and insights, we can enhance the effectiveness of policy decisions. In addition, forecasting will be used to monitor the outcomes of future policies and to provide early warnings of unknown risks."

Scenario predicting 'US Air Force airstrike possibility' demonstrated by Mantic. In June, as the timing of the US and Israel airstrikes on Iran approached, the possibility surged sharply. Mantic homepage

Mantic predicts that AI-based forecasting will eventually become an essential technology for public institutions. The first reason is that the accuracy of AI forecasts is improving rapidly. Mantic's AI chatbot ranked 300th in last year's Metaculus Cup, but now it is in the top 10. Regarding this, Dagger Turan, CEO of Metaculus, said in an interview with the technology media "Futuro Prossimo," "At the current pace, humans will be overtaken by AI in forecasting by 2029."

The second reason is AI's speed. Human forecasters take a long time to solve even one or two problems, and their mental stamina is quickly depleted when making predictions across a wide range of topics. However, AI can update its predictions instantly whenever it receives new information. This means it will be possible to compare the likelihood of specific future events in real time. Shevlane, CEO of Mantic, stated, "Our AI will reduce the time required to forecast the future and evaluate its accuracy to the level of milliseconds," adding, "Our goal is to achieve the 'industrialization of forecasting,' something that was impossible with the human brain alone."