Predicting: Quick Start

duck_master1 Jul 2023 3:43 UTC
9 points
3 comments14 min readLW link

Have you ever wanted to predict the future? Did you ever take a peek at Metaculus or Manifold or some other site, suddenly feel super intimidated about the effort you’d need to put in, and never try again? If so, fear not, for this post is for you![1]

This post is intended to be a how-to guide. I will add pictures because apparently people really like reading posts with pictures. However, since there are exactly zero good explainers on how to be a good predictor as far as I know, I’m hoping that more experienced predictors will find value in this post too.

Thanks to Emily Thomas for proofreading this post!

Epistemic status/​content warnings: Basically correct in the big picture, but I might have missed or misstated some details. May be too wordy or technical in parts.

1. What is probabilistic predicting, anyways?

Throughout history, people have often been interested in predicting the future. In ancient times, people would ask the local prophets or soothsayers. (This is called “prophecy”, and it’s mostly extinct.) In more recent decades, they have tried other, more systematic approaches, such as extrapolating trend lines on a graph into infinity. (This was called “futurism”, and it still exists to some extent these days.) Needless to say, the track record of such efforts is pretty bad.

The key insight of probabilistic prediction is simple: Instead of saying that things will definitely happen, just soften it to saying that they will only probably happen. And quantify the probability!

1.1 What is probability?

Probability is a weird and mystical concept that does not sit well with some people. For our purposes, a probability is a number associated with an event that describes what fraction of the times it could happen that it actually does happen. For example, if an event has a probability of 0.8, then given 10 different chances, the event should happen about 8 times. To make our lives easier, we predictors usually use percentages to write probabilities (for example, “70%” instead of “0.7″). Note that the probability of any event is always between 0% and 100%.

You should be aware of discrete distributions, which have a fixed list of outcomes whose probabilities are nonnegative and sum to 100%. You should also be aware of continuous distributions, where the true value can range continuously over some interval. Probabilists describe using a probability density function (roughly, how likely the value could come from a small interval around that point) and a cumulative density function (the probability the value could be less than or equal to the specified point).[2]

To make this concept feel more concrete to you, here are some examples:

  • “Will Zero equal One tomorrow?” has probability 0%, because it’s always false no matter what.

  • “Will the Sun explode tomorrow?” has probability <<1% because it’s really unlikely (as the Sun has lasted for many eons, and we as a society understand the types of things which cause stars to explode quite well), although you never know for certain with physical objects.

  • “Will a fair die land on 1 when I toss it again?” has probability 17% since each of the die’s six sides are equally likely to appear and 1⁄6 ≈ 17%.

  • “Will a fair coin land on 1 when I toss it again?” has probability 50% because the two sides are equally likely.

  • “If I flip millions of fair coins, will the total number of heads be a standard deviation above normal?” has probability 68% because that’s how the normal distribution works.

  • “Will Asia be north of Australia tomorrow?” has probability >>99% because the geography is this way and has also been stable for eons (and plate tectonics is way too slow to affect this)

Importantly, note that I’ve written that everything can be conceived of as having a probability. This is called “radical probabilism”. Radical probabilism is nothing new—it’s been in the rationalist milieu since Yudkowsky, and many other people believe it too. However, since predictors have to think about the probabilities of things all the time, they tend to believe even more in radical probabilism than rationalists proper!

If the concept of assigning probabilities to real world events is foreign to you, it may be helpful to read the first book from Rationality: From AI to Zombies, titled “Map and Territory”. That said, reading all this is totally not required for you to become a predictor. Not only is the Sequences ridiculously long, but I suspect that it may have deeper content issues as well.

1.2 Types of questions

In general, probabilistic predictors work with three types of questions: binary questions, multiple-choice questions, and numeric questions:

  • A binary question has two possible answers: YES and NO. When dealing with this question, your task is to provide a number, which is your estimate of the probability that the question resolves to YES. Binary questions are by far the easiest types of questions to wrap your head around, and have excellent support for calibration graphs.

  • A multiple-choice question has a fixed list of answers, and asks you to assign a probability for each answer. When answering such questions, make sure to check that all your subjective probabilities sum up to 100%. Sometimes, these may be “free-response questions” where you may to add new answers at will to the existing list.

  • A numeric question has an answer which is a number. Platforms vary, but typically your task is either to guess this number, give a confidence interval for this number, or (most sophisticatedly) draw a probability distribution which you expect the number to be drawn from. In some cases, this may be reframed as a “date” question, where you are asked to predict the point in time that something will occur.

Calibration on binary questions has excellent support from the (admittedly very small) forecasting software ecosystem. However, support for calibration graphs is slight for multiple choice questions and basically nonexistent for numeric or date questions. (This is uncharted territory so I encourage people to build this anyways!)

1.3. How do I predict?

At a high level, you should be aware that there are two major types of mechanisms in use: absolute predicting (where you give a specific probability or probability distribution), and relative predicting (where you make bets based on your disagreement with the previous market probability).

When it comes to making a concrete prediction, the most important thing that you could do is to do your own research! I cannot stress this point enough. Even just searching some relevant terms on Google or DuckDuckGo or Wikipedia or what-have-you and skimming the results gives you a massive advantage over a predictor who just trades based on the question title alone. If you have the motivation for this, the minute you spend learning the background can easily turn into an hour.

As for coming up with a specific prediction, there is no fixed recipe. However, predictors often find it helpful to employ the concept of “base rates” (the fraction of times that similar events have occurred in similar circumstances in the past). More advanced predictors may also want to build crude mathematical models of the situation (for those with a good math background) or imagine very detailed storylines in which the question resolves in different ways (for those with a good humanities background).

The exact UI elements that you’ll need to navigate in order to record your prediction vary from website to website, so see section 3.

2. Calibration training

Before you start predicting on live questions, I strongly recommend that you do calibration training, which is essentially “predicting” on “dead” questions (i.e. questions where the answer has already been determined). At the bare minimum, you should answer at least a dozen questions; ideally, answer a hundred.

To do calibration training, pick the training. The app will then ask you a “question” and provide you an interface for giving an answer. Use your intuition when answering the question! (The purpose of calibration training is to give you a good sense of probabilistic intuition, anyways.) Don’t look up the question because that defeats the purpose, and don’t memorize the answers either. Unfortunately, there are only so many calibration training questions, so unless (or until?) we enter the cyberpunk golden age where millions of fact-checked calibration questions abound, try your best to forget the questions you already answered if you’re answering them again.

A rough guide for converting feelings to subjective probabilities:

  • ~90-99% when you’re certain of the answer being YES (e.g. “Did the sun set in the west yesterday?” or “Is 1 a number?”)

  • ~70-80% if you think YES is very likely but not certain

  • ~60-70% if you have a vague hunch (sorry I don’t have a good example for this)

  • ~50% when you have basically no clue at all (e.g. “Does [obscure country] have a higher GDP than [similar obscure country]?”).

If you follow this guide, then you’re off to a great start! For probabilities below 50%, just reverse the numbers (for example, you should guess ~1-10% if you think the answer is almost certainly NO).

2.1. List of calibration trainings

There are plenty of good calibration trainings out there, and I’m relying on Isaac King’s List of Probability Calibration Exercises. However, the ecosystem is kind of scattered, so to help the reader, I’ll group each exercise by question type, and within each grouping the listing is roughly from most user-friendly to least.

Binary questions:

Multiple-choice questions:

  • https://​​bayes-up.web.app/​​ - a great web app, which even provides you a calibration graph! Don’t be scared by the loginwall—you can hit “proceed as guest” to start a fresh session. However, if you want to save your results over time then your Google account, email address, or phone number will do.

Numeric questions:

Before you move on, you should have a close-to-perfect calibration graph. If not, just try your hand again on a different quiz.

3. Go forth and predict!

Once you have a good calibration graph, the next step for you is to predict on live questions. After signing up, predicting only takes a few clicks.

Each platform, however, has its own idiosyncrasies, so I will discuss each one separately.

3.1. Metaculus

URL: https://​​www.metaculus.com/​​home/​​

This is the Metaculus homepage. Again, sorry for the spoilers.

Metaculus was founded in 2015. It mainly collects very serious questions about major world events or scientific/​technological advancements. Predicting on Metaculus is also absolute, but supports many different types of questions, including binary, numeric, date (basically numeric), “group” (a collection of related binary, numeric, or date questions), and “conditional” (conditioning a binary or numeric question on a different related binary question).

To sign up for Metaculus, hit the “Log In” button, then navigate to a separate “Sign up” dialog, which asks you for a username, a password, and an email address. Once you’re logged in, the homepage should now be a question feed.

Metaculus offers three tutorials (you’ll need to be signed in to see them): “Basics”, “Calibration Practice”, and “AI Tutorial”. If you’re a newbie and chose Metaculus, I strongly suggest that you complete all three of them in order.

Each tutorial rewards you with ten “tachyons” upon its completion (although making predictions and such also gives you tachyons), and you can use tachyons to unlock various Metaculus features (it’s in the “Profile” dropdown underneath your username):

  • personal track record (level 1)

  • hiding the community prediction (level 2)

  • “personal track record histograms” (i.e. a histogram of how many points you gained or lost from each question; level 3)

  • “personal track record calibration” (i.e. a calibration graph; level 4)

  • accessing a question’s “Metaculus prediction” once (note that this requires tachyons every time; level 5)

  • accessing other people’s track records

To make a new prediction on Metaculus depends on the type of question:

  • For binary questions Metaculus provides a slider ranging from 1% to 99%; drag it to your subjective probability of the question resolving to YES, and hit the “Predict” button to record your prediction.[3]

  • For numeric questions, your prediction consists of a probability distribution, which depends on several different sliders. You’ll be building your distribution with one to five unimodal components. Each component is controlled with three sliders – a middle one for centering the component, a left slider to resize the left tail, and a right slider to resize the right tail – and, if you’re using more than one component, there are also sliders to set weights for each component (very roughly speaking this is the relative probability of the scenario reflected by that component).

  • For date questions the mechanism is essentially the same as a numeric question, except the UI will show you days and months instead of just numbers.

  • For group or conditional pair questions, the mechanism is essentially a collection of all binary, all numeric, or all date questions (conditional pairs are a special case where the questions come in pairs), so predict each question individually. It’s encouraged to predict all the questions, although you don’t have to.

Making a new question, however, is much more stringent. The interface, which appears under the “Write a New Question” tab, is mostly intuitive. However, it’s important to note that the “title” should be shorter than the “question”, and that the resolve date can be later than the close date. Also, once you submit a question, it will need to be approved by one of Metaculus’s moderators. Then it will spend a day or two as “upcoming” so that other users can suggest other potential changes to the question formulation. After all of that, then your question will open for predictions.

3.2. Manifold

URL: https://​​manifold.markets/​​

Again, sorry for the question spoilers.

Manifold (formerly “Manifold Markets”, and before that, “Mantic Markets”) was founded in or around December 2021[4] by Austin Chen, James Grugett, and Stephen Grugett, and owes its existence quite heavily to the one and only 2021 ACX Grants round by Scott Alexander. In the intervening months, it has quickly become very popular. In fact, it could be on track to eclipse PredictIt, Metaculus, Polymarket, and Kalshi combined in total pageviews before 2023 is over! It’s also the most user-friendly and beginner-friendly platforms of all the ones I know.

Predicting on Manifold is relative. That is, instead of expressing a specific probability outright, you make bets that express your disagreement. To make a prediction on Manifold, you hit the button of the appropriate directionality (YES or NO for binary markets[5], the YES or NO of each specific option for multiple-choice markets, and HIGHER or LOWER for numeric markets). Then, you “buy” an appropriate amount of mana[6] in that directionality. This is a bet. Then, if or when the market resolves, you are rewarded with more mana if you are directionally right, but you lose mana if you’re directionally wrong (and this effect is stronger for bigger bets).

Another oddity of Manifold is that it does not have its own login system. Instead, you sign up for or log into Manifold using your Google account (hit the “Sign up” or “Sign in” buttons respectively).[7] A dialog may pop up asking you to select an account from a list (though often this list will only have one item); if so, just select your own account. If you don’t have a Google account, you can sign up for one by giving a username, password, and phone number.

Lastly, making markets on Manifold is super easy! Hit the “Create a question” button (formerly “Create a market”), select between either “Yes/​No” or “Multiple choice” ( creating numeric markets is impossible now, very sadly), write a title and description, categorize, and set a close date. All Manifold users, except for banned users, are given the power to create markets.

3.3. PredictionBook

URL: https://​​predictionbook.com/​​

Again, sorry for the spoilers.

PredictionBook is the oldest probabilistic predicting website, though it’s very close to defunct. Predicting on PredictionBook is absolute (i.e. you give a specific probability), and supports binary questions only.

According to this post by Eliezer Yudkowsky, PredictionBook was launched in 2009 (or maybe 2008?) by a no-longer-existing company(?) called “Tricycle Developments”. Since the mid-2010s, however, PredictionBook has been steadily in decline. These days, you’d be lucky if even 1 other user predicted on a question that you asked. Since PredictionBook is so low volume, its main use case is to record personal predictions.

To use PredictionBook, sign up using a username and password.

To make a new prediction on an existing question, view it while logged in. Then, under the “Add your estimate or comment” sidebar, type in your subjective probability and/​or a relevant comment (limit 240 characters; URLs and hashtags allowed). Hit “Record my prediction” (oddly enough this is also the button to hit to post a comment only), and your prediction will publicly appear.

To make a new question, click on the “New prediction” tab. Then enter your question into the “What do you think will (or won’t) happen?” textbox (add hashtags to categorize your question), your subjective probability into the “What’s your estimate of this happening?” textbox, and the close date in the “When will you know?” textbox (the website accepts several different date formats). Then, hit the “Lock it in!” button to publish.

3.4. Other platforms

Other platforms I’m not covering in similar detail (due to the sheer length of this blogpost and personally being unfamiliar with them) include:

Note that some of these platforms are blockchain-based and thus would require you to set up or use a cryptocurrency wallet in order to join.

4. Learning on the fly

So, if you’ve been following my instructions in this post well, you’ve made a few dozen predictions/​bets or so. Some of these may have even resolved! If you are disappointed by your results, however, or if you want to just get better at this because of the gamer instinct, then keep reading: there is still plenty of room for improvement.

4.1. Bad calibration

You might find that your calibration graph is much wonkier on live questions than on dead ones. This is not unusual! It can be tough to translate your feelings on dead questions, where instant feedback is just a few clicks away, to live questions where no one really knows the ultimate answer. If this is how you feel, I have suggestions:

First off, do more calibration training. It never hurts to do calibration training, so long as you are not actively memorizing the answers. If you haven’t exercised your sense of radical probabilism in the last few days or weeks, it might be a little off when you try to do so again.

Secondly, look at your live calibration graph, and specifically look at how your predictions have fared against question resolutions. If your calibration graph looks like a curve that got too far off the diagonal, then fortunately you can just read off the conversions of what-you-thought-the-probability-should-be (on the x-axis) to what-the-probability-should-actually-be (on the y-axis). If it looks noisier, then more likely your intuition is only weakly correlated with the truth, or you just haven’t predicted enough questions yet. If that’s the case, then it’s time for you to look at specific question resolutions, especially for the points farthest off the diagonal. Try to recall why you made these specific predictions in the first place (this is easiest if you make comments explaining your reasoning as you go along). Then, look at what actually happened, not just with respect to the resolution of the literal question statement, but any factors or omens portending that resolution which you may have missed. Rinse and repeat for each resolved question, or at least each question that you were directionally wrong about.

4.2. Talking with other predictors

It can also be possible to learn more about the art of predicting by actually talking to other predictors.

I’m not aware of much in this vein, as a lot of what predictors do say or write about predicting is one-way info-sharing rather than actual dialogue. However, Metaculus does sponsor talks and socializing in the “Forecast Friday” events, which runs in a gather.town world (warning: may be buggy!). They also host tangentially-related-to-predicting talks under the “Metaculus Presents” series, which may also be worth listening to either live or in recorded form.

5. Other considerations

This section is for miscellaneous considerations that don’t really fit into the other four sections.

5.1. Ambiguous or N/​A resolutions

Occasionally, some questions may resolve to Ambiguous (Metaculus’s wording) or N/​A (Manifold’s wording). This means that the circumstances made it nonsensical to ascribe a meaningful answer to the question. As an example, the Metaculus question “Which language modelling benchmark will be most popular in the calendar year 2022?” resolved to Ambiguous because there were no submissions at all for any of the four language modeling benchmarks listed.

If such a resolution occurs, you should konw that your track record remains unaffected, as if the question didn’t exist at all. When dealing with questions that have a substantial chance of resolving to N/​A, pretend that such outcomes don’t exist. (In more technical terms, consider the conditional probability, conditioned on the question not resolving to N/​A.)

5.2. Asking good questions

In order to even have questions to predict on, one has to ask questions in the first place! Asking questions at all is generally pretty easy, but not all questions are equally fit for forecasting. As I wrote above, forecasters generally work with either binary, multiple-choice, or numeric questions. (So, if your question is some kind of ultra vague open-ended one like “What is the best way to solve AI alignment?”, then you’ll have really tough luck getting predictors to give you an answer.)

Despite this, most natural questions do have sort-of-falsifiable answers. If there’s no natural close date, just add an arbitrary deadline on the right timescale. See also this 2015 post by Julia Galef on 16 specific types of useful predictions, which may be of further interest.

If you want shorter-term questions, follow the news! As you read, ask yourself about the plot twists that could occur for each story. (Putting yourself in mythic mode might help with this.) Then, the questions about what plot twists will occur do make for good, forecastable questions.

It bears mentioning bad questions (which chiefly appear on Manifold). The first type is self-resolving questions, whose resolution depends on only the specific trades or comments on it, instead of anything external. These are uneasily tolerated on Manifold, although that website’s admins do actively delist them to keep them small. The other type is spam, where the question’s title and description are lifted wholesale from some kind of marketing copy. Creating such questions is extremely discouraged and can get you banned on sight.

5.2.1 Dealing with long term questions

By default, you should be a little suspicious of any predictions that resolve decades into the future, even if the people making them are the world’s best superforecasters. This is because probabilistic predicting is still a young field and very few long-term questions have been resolved. For similar reasons, you should be suspicious of any of your own predictions that must wait for longer than you’ve been predicting.

That said, people few techniques for dealing with long-term questions:

  • Asking what the best predictors on short-term questions are predicting on your long-term question.

  • Coming up with short-term factors which could influence the long-term result, and predicting the short term factors instead. (For example, “Will Metaculus exist in 2100?” could be addressed with “Will Metaculus fire lots of employees in the next few years?”. Credit to SirSalty for this idea.)

  • Putting your bet into an index fund (extremely speculative; suggested by this paper in the Journal of Prediction Markets, which is a real thing)

5.3. Prediction bots

Bots that make predictions exist, although they are quite primitive. You can find a list of existing Manifold bots by viewing the “Bots” league. If you want to make your own bot, Metaculus and Manifold both have APIs (here are Manifold’s docs and Metaculus’s docs), although Manifold is much more bot-friendly than any of the other platforms. If you have a ML/​AI background, you may also be interested in the Autocast competition. There is little work in this regard but I think prediction bots are another potentially big opportunity.

6. Coda

I hope you found this post useful in starting or continuing your predicting journey.

  1. ^

    Of course, I’m assuming that you, dear reader, are a human. For bots, a very different set of considerations applies; see section 5.4.

  2. ^

    I was going to write a lot more about this, but I decided against it because most of you don’t really have to care about the minutiae of probability theory.

  3. ^

    The two arrows on either side of the slider allow you to shift the value by increments of 0.1%, for anyone that fastidious. However, in my experience, its main use case is to express a subjective probability of less than 1% or greater than 99% - in other words, really unlikely or really likely.

  4. ^

    Sources: Crunchbase states Manifold’s founding date as “Dec. 1, 2021”; the oldest market on Manifold that I can find was created in mid-December 2021; the oldest Internet Archive snapshots of Manifold’s homepage date from January 2022.

  5. ^

    On Manifold, questions are usually called “markets”, and the UI used to use that specific word until earlier this June.

  6. ^

    Mana (denoted M$) is Manifold’s play currency. It’s not real money, although you can get it for real money or donate your mana as real money at the fixed ratio of 100 mana:1 USD. Each new user starts with 500 mana.

  7. ^

    I suspect that this is because Manifold uses Firestore, a Google product, to store its data (as stated in the Manifold monorepo).

  8. ^

    This is taken from Metaculus’s primary focus list; I am not claiming that it’s exhaustive. Also, I’ve reordered it to make it prettier.

duck_master1 Jul 2023 3:43 UTC
9 points
3 comments14 min readLW link