We (jacobjacob and Ben Pace) decided to award $100 (out of the total bounty of $800) to this answer.
It compiles a useful summary of the literature (we learnt a lot from going through on of the papers linked), and it attaches handy links to everything, which is a task which is on the one hand very helpful to other people, and on the other tedious and without many marginal benefits for the writer, and so likely to be under-incentivised.
I’ll PM you for payment details.
We (jacobjacob and Benito) decided to award $150 (out of the total bounty of $800) to this answer (and the additional points made in the discussion).
It offers relevant and robust evidence about the role of info-cascades in forecasting environments, together with a discussion of its interpretation.
I’ll PM you about payment details.
We (jacobjacob and Ben Pace) decided to award $250 (out of the total bounty of $800) to this answer. It does several important things.
It references existing (and novel) work in economics and mechanism design, which might have been time-consuming to discover otherwise
It distills a technical paper, which is a valuable service that is usually underfunded (academic institutions comparatively incentivise novel and surprising insights)
The insights provided are quite action-guiding, and caused me (jacobjacob) to have ideas for how one can experiment with new kinds of forecasting tournaments that use a threshold-mechanism to change participant incentives
I’ll PM you for details about payment.
UPDATE.
We (jacobjacob and Ben Pace) have finally settled on the allocation of the $800 bounty for this question. All the motivations are summarised in this comment, together with links to the relevant prize-winning answer/comment.
We will also post individual notices with motivations next to each comment for ease of discussing them.
We’ll PM all prize winners to sort out logistical details of payment.
Main post
David Manheim (answer and additional points made in discussion) $150
This answer offers relevant and robust evidence about the role of info-cascades in forecasting environments, together with a discussion of its interpretation.
Jan Kulveit (answer and additional comment below) $200
This answer seems to offer a learnt summary of the relevance of network science (which offers a complementary perspective on the phenomenon to the microeconomic literature linked by other commenters), which not implausibly took Jan at least an order of magnitude less time to compile than it would have taken us. (For example, the seemingly simple fact of using a different Google scholar keyword than “information cascade” might have taken several hours to realise for a non-expert.) It also attempts to apply these to the case of forecasting (despite Jan’s limited knowledge of the domain), which is a task that would likely have been even harder to do without deep experience of the field.
Pablo (1 and 2) $100
These answers compile a useful summary of the literature (we learnt a lot from going through on of the papers linked), and it attaches handy links to everything, which is a task which is on the one hand very helpful to other people, and on the other tedious and without many marginal benefits for the writer, and so likely to be under-incentivised.
Michael McLaren $50
This answer:
It offers a novel mechanism which is relevant to the context of intellectual progress, and ties it in with literature cited in the OP
Rather than just linking the paper, it distills a technical paper, which is a valuable service that is usually underfunded (academic institutions comparatively incentivise novel and surprising insights)
Ways of responding
David Manheim $50
This answer offers a practical example of a cascade-like phenomenon, which is both generally applicable and has real economic consequences. Also, the fact that it comes with a game to understand and practice responding is rare and potentially quite valuable (I (jacobjacob) am of the opinion that deliberate practice is currently a neglected virtue in the rationality/EA spheres).
rossry $250
This answer does several important things.
It references existing (and novel) work in economics and mechanism design, which might have been time-consuming to discover otherwise
It distills a technical paper, which is a valuable service that is usually underfunded (academic institutions comparatively incentivise novel and surprising insights)
The insights provided are quite action-guiding, and caused me (jacobjacob) to have ideas for how one can experiment with new kinds of forecasting tournaments that use a threshold-mechanism to change participant incentives
We (jacobjacob and Benito) decided to award $50 (out of the total bounty of $800) to this answer.
It offers a practical example of a cascade-like phenomenon, which is both generally applicable and has real economic consequences. Also, the fact that it comes with a came to understand and practice responding is rare and potentially quite valuable (I’m of the opinion that deliberate practice is currently a neglected virtue in the rationality/EA spheres).
.
I am quite uncertain but have an intuition that there should be an expectation of more justification accompanying stronger negative aversions (and “hate” is about as strong as it gets).
(Naturally not everything has to be fully justified, that’s an unbearable demand which will stifle lots of important discourse. This is rather a point about the degree to which different things should be, and how communities should make an unfortunate trade-off to avoid Moloch when communicating aversions.)
Ah! I read “it” as the comment. That does change my mind about how adversarial it was.
Well, I strong downvoted because adversarial tone, though I’d be pretty excited about fighting about this in the right kind of way.
Curious if you could introspect/tell me more about the aversion?
Something interesting happens when one draws on a whiteboard ⬜✍️while talking.
Even drawing 🌀an arbitrary squiggle while making a point makes me more likely to remember it, whereas points made without squiggles are more easily forgotten.
This is a powerful observation.
We can chunk complex ideas into simple pointers.
This means I can use 2d surfaces as a thinking tool in a new way. I don’t have to process content by extending strings over time, and forcibly feeding an exact trail of thought into my mind by navigating with my eyes. Instead I can distill the entire scenario into 🔭a single, manageable, overviewable whole—and do so in a way which leaves room for my own trails and 🕸️networks of thought.
At a glance I remember what was said, without having to spend mental effort keeping track of that. This allows me to focus more fully on what’s important.
In the same way, I’ve started to like using emojis in 😃📄essays and other documents. They feel like a spiritual counterpart of whiteboard squiggles.
I’m quite excited about this. In future I intend to 🧪experiment more with it.
I downvoted this even though it followed instructions, because the final sentence has a scornful tone that does not seem conducive to good-faith intellectual discourse.
Problem
The space of possible DRT-induction method pairs is much larger than this would suggest.
I think the space of things you could try is quite large indeed, both when it comes to DRT-induction as well as what you choose to include in the control condition. I can also imagine this being a major point of contention/annoyance post-study (“This is nice, but for me to really change my mind I’d want you to have used this induction/control”).
Solution
Before the experiment, we have prediction markets/forecasting tournaments on the results of the pre-registered statistical tests, given a particular induction x control combination.
When the markets close, your experiment runs as planned—but you only test the induction x control combinations that had the most disagreement/variance in their estimates.
Prediction market participants are then paid according to a proper scoring rule based on the outcome of the experiment.
So overall, even if you just test 1-3 experimental designs, we could have these markets on 10-20 designs, and get priors for all of them!
This is also a more transparent way of picking conditions to run for the experiment.
___
I’ve messaged you privately to discuss this further and organise eventual funding and operational support.
Blackberries and bananas
Here’s a simple metaphor I’ve recently been using in some double-cruxes about intellectual infrastructure and tools, with Eli Tyre, Ozzie Gooen, and Richard Ngo.
Short people can pick blackberries. Tall people can pick both blackberries and bananas. We can give short people lots of training to make them able to pick blackberries faster, but no amount of blackberries can replace a banana if you’re trying to make banana split.
Similarly, making progress in a pre-paradigmatic field might require x number of key insights. But are those insights bananas, which can only be picked by geniuses, whereas ordinary researchers can only get us blackberries?
Or, is this metaphor false, such that having a certain number of non-geniuses + excellent tools, we can actually replicate geniuses?
This has implications for the impact of rationality training and internet intellectual infrastructure, as well as what versions of those endeavours are most promising to focus on.
This is a good point. I think the epistemic ability to predict and evaluate arguments independently of the truth of the conclusion is something we want to heavily select for and reward, see e.g. Eliezer’s writing on that here.
If Elizabeth is interested, I’m definitely interested in funding and experimenting with prediction markets on argument validity for the next round of amplifying epistemic spot checks.
Your cruxes are formulated as “Why do I believe what I believe?”
This is quite different from what seems to me important to get at in double crux—“What would change my mind?”
For example, case 2 is only crux if it is true that: “Were you to believe that a person can not have contact with God through prayer, then you would change your mind and think there’s no God”.
Is that correct?
As a metaphor, think of a ceiling supported by a few walls. It’s usually the case when constructing houses that not all walls are equally important to keeping the ceiling up. Some are merely decorative—you can knock them over and put up a new one elsewhere, and the thing will be fine. But others are load-bearing—if you take those walls down, the ceiling itself will come crashing in.
From experience, I find something similar happens with belief. For a given belief, I can often list many arguments supporting it. And those usually that take the form “I believe X”. But it often turns out that most of them aren’t actually the load-bearing reason I believe it. Because were I to knock them down and stop believing them, I still would not change my mind about X. To find the ones which are actually load-bearing, it’s more useful to use as a search query “If false, would this change my mind?”, rather than “Do I believe this?”
Nitpick. Mildly triggered by:
These are posts about moving from thought A to thought B, and whether thought B is allowed given thought A.
“Allowed” is of course a very social term, and one that sounds a lot like “will my teacher accept it if I make this inference?”
Which is different from the mathematical mindset of what happens if I make that inference, and is that thing interesting/elegant/useful. What does it capture to have those kinds of inference rules, and does it capture the kind of process I want to run or not?
Moreover, when it comes to Bayesian reasoning and its various generalisations, the correct inference is _inevitable_, and not optional. There is one single credence which is correct to hold given your priors and the evidence you’ve observed. (Compare this to old school rationality, like Popper and Feynman, thought more in terms of you being “allowed” to hold a variety of beliefs as long as you hadn’t been refuted by experiment. I can’t find the reference post for this now, though.)
Just riffing a bit on the same project you started :)
There’s integrity and accountability—integrity (Level 3) as following a certain decision theory and making it common knowledge that you do, such that others can reliably simulate you, and coordinate and make trades with you; and accountability as choosing who you want to do your individual holistic regulation (Level 4).
On another note, predictions and calibration training is often pitched as a kind of Level 1⁄2 intervention, but I’m more bullish on it as a Level 2 intervention with important Level 5 consequences.
It’s certainly often helpful to quantify your beliefs, and to form an all-things-considered opinion as an ensemble model of all the things you might trust. But to restrict your trains-of-thought to always follow an all-things-considered view, never veering off into resonating with a single model or world-view, is, as you point out, not that great. However, spreading the meme of being able to zoom out to an all-things-considered, quantitative opinion when necessary, and engaging with that level regularly enough to build a track-record of being able to do that, seems like a core part of having a healthy Bayesian community, even if you actually use it quite infrequently compared to other modes of thinking (just like professional mathematicians riff on a post-rigorous level but can drop down to the rigorous level when need be). This is part of my current framing for the forecasting class I’m teaching at CFAR mainlines.
There’s also a long list of other CFAR techniques one could analyse.
Eliezer’s and Abram’s posts are interesting Level 1 interventions, but look at lot like improvements to your slow, deliberate, conscious thinking processes, perhaps eventually becoming ingrained in your S1. I’d compare that with TAPs, which seem to intervene quite directly at Level 2 (and probably with backchaining effects to Level 1): “what thoughts do I want to follow from other thoughts?” [1]
This also seems to me to be the core of what makes CBT therapy work, whereby you uncover unwanted trains (“Get invite to social event” → “Visualise public shame from making an embarrassing comment” → “Flinch away from invite”), and then intervene to change their trajectory.
This causes the question of whether there are any more direct interventions at Level 1. Interventions determining which thoughts, in and of themselves, are even desirable or not. I interpret Selective reporting and Lines of retreat as analysing such interventions. The former (a bit extrapolated) as noting that if there are some unitary thoughts we cannot think, regardless of whether we actually believe them, this can cause large mistakes elsewhere in our belief system. The latter tries to tackle the problem when the blocker is motivational rather than social, by embedding the thoughts in conditionals and building a backup plan before considering whether it has to be used.
Then there’s goal factoring, closely related to separation of concerns. Don’t take actions which confusedly optimise for orthogonal goals, separate out your desires and optimize them separately. This probably has implications at Levels 1 through 4.
I could go on through the CFAR techniques and might at a later point, but that will do for now.
[1] This looks more like “epistemic TAPs”, or “internal TAPs”, which haven’t yet become a standard part of the mainline curriculum, where TAPs are often more external, and for things like “Deciding to take the stairs instead of the elevator as soon as I come into the office and look at them”.
As we’re thinking about _intervention_, we’re hoping to _change_ something, or accomplish some _effect_. And in this vein, it’s interesting to note how the levels aren’t that independent.
For example, incentives tend to backpropagate from one level to the other. I expect that if you regularly give someone negative reinforcement for expressing half-formed ideas (Level 3 intervention), they might not just stop expressing ideas, but also stop _having_ original ideas altogether (Level 1 / 2 effect).
Or if you establish a meme of sharing the causes of your beliefs (Level 3 intervention), your community as a whole will run into fewer info-cascades (Level 5 effect).
Some of the most powerful interventions are those which create loops between levels. Helping people become stronger rationalists (Level 1 / 2) will enable them to make important changes to their and their community’s environment (Level 4 / 5) which will then feedback into their ability to think true thoughts and enact further important changes.
Similarly, bad equilibria emerge when Level 5 interventions change the optimal strategy at Level 3, and people doubling down on that then further entrenches those Level 5 changes.
This sounds like a very exciting project and a solution to an open exposition problem.
I look forward to reading the posts!
I also wonder if you know what kinds of things would motivate you to write them?
Interesting discussion? Getting data about your ability to tutor concepts? Money (maybe we could organize a Patreon with interested LW users if so)?
We (jacobjacob and Ben Pace) decided to award $200 (out of the total bounty of $800) to this answer (and the additional comment below).
It seems to offer a learnt summary of the relevance of network science (which offers a complementary perspective on the phenomenon to the microeconomic literature linked by other commenters), which not implausibly took Jan at least an order of magnitude less time to compile than it would have taken us. (For example, the seemingly simple fact of using a different Google scholar keyword than “information cascade” might have taken several hours to realise for a non-expert.)
It also attempts to apply these to the case of forecasting (despite Jan’s limited knowledge of the domain), which is a task that would likely have been even harder to do without deep experience of the field.
I’ll PM Jan about payment details.