The picture this post paints seems to me wildly unrealistic, in a “this is ignoring one of the most taut binding constraints of socioeconomic reality” kind of way.
The overwhelmingly most likely result of a monk going to their private space to muse and meditate for 10000 days is that they end up completely unmoored from reality, and produce nothing of any practical use at all. This is not a matter of “the 10-day monks aren’t able to recognize why the 10000-day monk’s products are useful”. It’s a matter of “the 10000-day monk’s products are not useful”, because that’s what happens when someone spends very long periods trying to do something without a feedback signal. It’s that lack of feedback signal which is the problem.
Now, in principle this could be circumvented. The 10000-day monks themselves could recognize the lack-of-feedback-signal as a problem, and go looking for useful feedback signals. Unfortunately, the whole “just give all the 10000-day monks whatever resources they need” thing means that there’s no incentive or selection pressure to ensure that the 10000-day monks actually do that.
What actually happens in monk-world is that massive amounts of resources are thrown at 10000-day monks who almost-all do absolutely nothing useful. Even if some of them do figure out useful things, there is no signal with which to distinguish the useful-monks from the masses of 10000-day monks producing crap. Even if some 10000-day monk realizes that that AGI-related X-risk is coming, there will be thousands of other 10000-day monks warning about thousands of other long-term problems, most of which are in fact completely imaginary because none of these monks has any feedback signal. All the memetic dynamics of our world would still apply. “Delegation via chain of trust” would not work any better in monk-world than it does in our world, because there is no better mechanism to enforce truth in monk-world than in our world; there is no better feedback signal in which to ground trust. The 100000-day monks’ work would be even less grounded in reality than all those scientific papers in our world which fail to replicate; after all, the 100000-day monks have no particular incentive to care about their work failing to replicate on a 1-year timescale, they can just say “the reasons are mysterious to you” and their bills will continue to be paid.
The bottleneck to good long-term thinking is not just “we don’t throw enough resources at long-term thinking”. It’s “long-term means bad feedback loops, so we can’t easily distinguish which long-term thinking is actually useful”. If we were able to solve that problem, I expect resources would follow, but simply throwing resources around will not cause the feedback-loop problem to be solved.
The author of the story in question takes a fairly similar attitude, specifically noting that, without feedback or external contact, often the 10,000-day monks produce only useless things or their monastery opens up on schedule and everyone is dead (I think sometimes of mass suicide but it’s been a while). On the other hand, you also at least once get monks who have spooky multiverse-walking quantum-immortality abilities and can defeat aliens. I get the impression that it’s intended a bit analogous to an even more extreme ‘intellectual venture capital’; the idea being that the long-term monasteries are long-shots which usually fail but just once is enough to pay for them all. The abilities of the quantum-immortal monks are self-verifying in the sense that if you hit the tail of the power law, the incidental spin-off capabilities are so impressive that asking is otiose—similar to a VC investing in Facebook or Airbnb or Bitcoin or Stripe; if you did, you don’t really need to do a detailed audit to figure out if you turned a profit! (You did!)
So the real point of disagreement, perhaps, is whether even 1 such success is plausible.
Small correction, the level which goes insane or dies is the second level down, out of four total; 100 years rather than 1000. Though it is roughly 3x the 10,000 day mark would equal, though. The weaponized use of Penrose Quantum Mind is devised by the top level, who are seen only once per 1000 years. (As it is written: SciFiWritersHaveNoSenseOfScale. Even the particularly clever ones who play around with big ideas and write extremely ingroup-y doorstoppers.)
(It’s also the case, as some readers have pointed out in the comments below, that certain problems cannot be solved by isolated thought alone, and require feedback loops or regular contact with the territory. For monks working on such problems, it is less that they sequester themselves completely for thousands of days at a time and more that, during those thousands of days, none can make demands of them.)
The claim is not that just giving people resources and letting them think is sufficient; the objections you raise in your last paragraph seem true and correct to me (e.g. there’s an incentive thing to be solved, among other problems) but it wasn’t so much that I intend the worst possible answer to those objections as that the above essay didn’t speak to them at all. =P
(Didn’t speak to e.g. how do you select 1,000 and 10,000 and 100,000-day monks, how do you sort them to their respective problems, how do you motivate and evaluate them, etc.)
How much work was “culture of Duncans” supposed to be doinng?
In particular, I kind of read it as “we can assume (among other things) common knowledge that everyone is basically cooperative with everyone else”. So people don’t become 10,000 day monks just because they want to be supported while they doss around. Is that intended?
If Duncans don’t all basically cooperate with other Duncans, or if it’s supposed to be more like “a culture where Duncans have a lot of institutional power but a lot of the population isn’t a Duncan”, I become a lot more skeptical, while acknowledging that if you do think it would work in that sort of situation, you’ve probably thought of the same objections I have.
This, in so many words was likely one of the biggest factors why Anthropic/OpenAI/Deepmind were so much more successful than any LW person or group at AI safety until maybe 2021 at the earliest, and even then the lead shifted.
A lot of AI safety proposals before deep learning were basically useless because of the feedback loop issue.
I think this is also connected to ambition issues, but even then lack of feedback loops was way worse for LW than they thought it was.
It’s also why longtermism should be bounded in practice, and a very severe bound at that.
Edit: The comment that johnswentworth made is pointing indirectly at a huge problem that affects LW, and why I’m not inclined to treat arguments for doom seriously anymore, in that there are no feedback loops of any kind, with the exceptions of the AI companies.
The picture this post paints seems to me wildly unrealistic, in a “this is ignoring one of the most taut binding constraints of socioeconomic reality” kind of way.
The overwhelmingly most likely result of a monk going to their private space to muse and meditate for 10000 days is that they end up completely unmoored from reality, and produce nothing of any practical use at all. This is not a matter of “the 10-day monks aren’t able to recognize why the 10000-day monk’s products are useful”. It’s a matter of “the 10000-day monk’s products are not useful”, because that’s what happens when someone spends very long periods trying to do something without a feedback signal. It’s that lack of feedback signal which is the problem.
Now, in principle this could be circumvented. The 10000-day monks themselves could recognize the lack-of-feedback-signal as a problem, and go looking for useful feedback signals. Unfortunately, the whole “just give all the 10000-day monks whatever resources they need” thing means that there’s no incentive or selection pressure to ensure that the 10000-day monks actually do that.
What actually happens in monk-world is that massive amounts of resources are thrown at 10000-day monks who almost-all do absolutely nothing useful. Even if some of them do figure out useful things, there is no signal with which to distinguish the useful-monks from the masses of 10000-day monks producing crap. Even if some 10000-day monk realizes that that AGI-related X-risk is coming, there will be thousands of other 10000-day monks warning about thousands of other long-term problems, most of which are in fact completely imaginary because none of these monks has any feedback signal. All the memetic dynamics of our world would still apply. “Delegation via chain of trust” would not work any better in monk-world than it does in our world, because there is no better mechanism to enforce truth in monk-world than in our world; there is no better feedback signal in which to ground trust. The 100000-day monks’ work would be even less grounded in reality than all those scientific papers in our world which fail to replicate; after all, the 100000-day monks have no particular incentive to care about their work failing to replicate on a 1-year timescale, they can just say “the reasons are mysterious to you” and their bills will continue to be paid.
The bottleneck to good long-term thinking is not just “we don’t throw enough resources at long-term thinking”. It’s “long-term means bad feedback loops, so we can’t easily distinguish which long-term thinking is actually useful”. If we were able to solve that problem, I expect resources would follow, but simply throwing resources around will not cause the feedback-loop problem to be solved.
The author of the story in question takes a fairly similar attitude, specifically noting that, without feedback or external contact, often the 10,000-day monks produce only useless things or their monastery opens up on schedule and everyone is dead (I think sometimes of mass suicide but it’s been a while). On the other hand, you also at least once get monks who have spooky multiverse-walking quantum-immortality abilities and can defeat aliens. I get the impression that it’s intended a bit analogous to an even more extreme ‘intellectual venture capital’; the idea being that the long-term monasteries are long-shots which usually fail but just once is enough to pay for them all. The abilities of the quantum-immortal monks are self-verifying in the sense that if you hit the tail of the power law, the incidental spin-off capabilities are so impressive that asking is otiose—similar to a VC investing in Facebook or Airbnb or Bitcoin or Stripe; if you did, you don’t really need to do a detailed audit to figure out if you turned a profit! (You did!)
So the real point of disagreement, perhaps, is whether even 1 such success is plausible.
Small correction, the level which goes insane or dies is the second level down, out of four total; 100 years rather than 1000. Though it is roughly 3x the 10,000 day mark would equal, though. The weaponized use of Penrose Quantum Mind is devised by the top level, who are seen only once per 1000 years. (As it is written: SciFiWritersHaveNoSenseOfScale. Even the particularly clever ones who play around with big ideas and write extremely ingroup-y doorstoppers.)
Yeah, this was intended to be addressed by:
The claim is not that just giving people resources and letting them think is sufficient; the objections you raise in your last paragraph seem true and correct to me (e.g. there’s an incentive thing to be solved, among other problems) but it wasn’t so much that I intend the worst possible answer to those objections as that the above essay didn’t speak to them at all. =P
(Didn’t speak to e.g. how do you select 1,000 and 10,000 and 100,000-day monks, how do you sort them to their respective problems, how do you motivate and evaluate them, etc.)
How much work was “culture of Duncans” supposed to be doinng?
In particular, I kind of read it as “we can assume (among other things) common knowledge that everyone is basically cooperative with everyone else”. So people don’t become 10,000 day monks just because they want to be supported while they doss around. Is that intended?
If Duncans don’t all basically cooperate with other Duncans, or if it’s supposed to be more like “a culture where Duncans have a lot of institutional power but a lot of the population isn’t a Duncan”, I become a lot more skeptical, while acknowledging that if you do think it would work in that sort of situation, you’ve probably thought of the same objections I have.
A lot of the work was being done by “culture of Duncans.”
All of that is double-clickable/expandable, but it’s hard to expand all of it for any single essay; that’s why so many essays. =)
This, in so many words was likely one of the biggest factors why Anthropic/OpenAI/Deepmind were so much more successful than any LW person or group at AI safety until maybe 2021 at the earliest, and even then the lead shifted.
A lot of AI safety proposals before deep learning were basically useless because of the feedback loop issue.
I think this is also connected to ambition issues, but even then lack of feedback loops was way worse for LW than they thought it was.
It’s also why longtermism should be bounded in practice, and a very severe bound at that.
Edit: The comment that johnswentworth made is pointing indirectly at a huge problem that affects LW, and why I’m not inclined to treat arguments for doom seriously anymore, in that there are no feedback loops of any kind, with the exceptions of the AI companies.