First I should note that Nate is the one who most believed this; that we not share ideas that come from Nate was a precondition of working with him. [edit: this wasn’t demanded by Nate except in a couple of cases, but in practice we preferred to get Nate’s input because his models were different from ours.]
With that out of the way, it doesn’t seem super implausible to us that the mindset is useful, given that MIRI had previously invented out of the box things like logical induction and logical decision theory, and that many of us feel like we learned a lot over the past year. On inside view I have much better ideas than I did a year ago, although it’s unclear how much to attribute to the Nate mindset. To estimate this it’s super unclear how to make a reference class—I’d maybe guess at the base rate of mindsets transferring from niche fields to large fields and adjust from there. We spent way too much time discussing how. Let’s say there’s a 15%* chance of usefulness.
As for whether the mindset would diffuse conditional on it being useful, this seems pretty plausible, maybe 15% if we’re careful and 50% if we talk to lots of people but don’t publish? Scientific fields are pretty good at spreading useful ideas.
So I think the whole proposition is unlikely but not “extraordinary”, maybe like 2.5-7.5%. Previously I was more confident in some of the methods so I would have given 45% for useful, making p(danger) 7%-22.5%. The scenario we were worried about is if our team had a low probability of big success (e.g. due to inexperience), but sharing ideas would cause a fixed 15% chance of big capabilities externalities regardless of success. The project could easily become -EV this way.
Some other thoughts:
Nate trusts his inside view more than any of our attempts to construct an argument legible to me which I think distorted our attempts to discuss this.
It’s hard to tell 10% from 1% chances for propositions like this, which is also one of the problems in working on long-shot, hopefully high EV projects.
Part of why I wanted the project to become less private as it went on is that we generated actual research directions and would only have to share object level to get feedback on our ideas.
First I should note that Nate is the one who most believed this; that we not share ideas that come from Nate was a precondition of working with him. [edit: this wasn’t demanded by Nate except in a couple of cases, but in practice we preferred to get Nate’s input because his models were different from ours.]
With that out of the way, it doesn’t seem super implausible to us that the mindset is useful, given that MIRI had previously invented out of the box things like logical induction and logical decision theory, and that many of us feel like we learned a lot over the past year. On inside view I have much better ideas than I did a year ago, although it’s unclear how much to attribute to the Nate mindset. To estimate this it’s super unclear how to make a reference class—I’d maybe guess at the base rate of mindsets transferring from niche fields to large fields and adjust from there. We spent way too much time discussing how. Let’s say there’s a 15%* chance of usefulness.
As for whether the mindset would diffuse conditional on it being useful, this seems pretty plausible, maybe 15% if we’re careful and 50% if we talk to lots of people but don’t publish? Scientific fields are pretty good at spreading useful ideas.
So I think the whole proposition is unlikely but not “extraordinary”, maybe like 2.5-7.5%. Previously I was more confident in some of the methods so I would have given 45% for useful, making p(danger) 7%-22.5%. The scenario we were worried about is if our team had a low probability of big success (e.g. due to inexperience), but sharing ideas would cause a fixed 15% chance of big capabilities externalities regardless of success. The project could easily become -EV this way.
Some other thoughts:
Nate trusts his inside view more than any of our attempts to construct an argument legible to me which I think distorted our attempts to discuss this.
It’s hard to tell 10% from 1% chances for propositions like this, which is also one of the problems in working on long-shot, hopefully high EV projects.
Part of why I wanted the project to become less private as it went on is that we generated actual research directions and would only have to share object level to get feedback on our ideas.
* Every number in this comment is very rough
Interesting, thanks!