The biggest issue here is not even any particular talking point or thinkpiece, but the overall ecosystem of discussions on the topic.
I didn’t even consider this point. I’m sure most capability researchers already have capability advancement strategies as their top-of-mind idea, but I guess if the discussion on LW is sufficiently orthogonal to the discussion inside capability-researcher communities (seems likely?), then I would agree that this is bad. I would guess that this is non-obvious to most people on this site.
But on the other side we have, as @Kaj_Sotalaoutlined, the costs of the LW community potentially-falsely believing that there are no blockers to AGI left.
Do you feel like this is a direct tradeoff? Or is there another way to address some or all of these costs without pointing directly at concrete blockers to AGI? Maybe the right way to navigate this issue is some alternative approach(es)? The following options seem fine but not amazing.
Getting the community to hedge our bets more? It’s great that there are people working on things for worlds with no blockers left to AGI, but it feels like the pendulum has swung too far in that direction as of late. On the other hand, a lot of this might be more due to streetlighting than anything, so just telling people to hedge their bets more doesn’t really help with that. And “hedge your bets” is already an obvious strategy, so I don’t know if telling people to do it more helps at all. Although maybe its obvious only in some implicit / retrospective-y way s.t. hearing it said or emphasized is helpful to some people?
There might be some way to argue abstractly about blockers to AGI convincingly without pointing at specific ones. I know Tsvi has attempted that here, here, here, and here. I find these arguments more convincing the more I read Tsvi’s other work, which suggests that there is maybe more work to be done in making these arguments fully explicit and less reliant on background intuitions. Also, maybe there are other abstract arguments that can be made?
The extent to which it’s rational to hedge your bets obviously depends on what the gamble is, i. e., how uncertain you are regarding what the outcome would be. If you’re near-certain about short timelines, it doesn’t actually make sense to “hedge your bets” by investing in some long-timelines approaches. So you have to argue away from that. Which can be done either by:
Making object-level arguments against short timelines – which mostly does necessarily require discussing blockers to AGI. And if you want to make your arguments more convincing, the best way to do so is by making them more concrete at the technical level, which in turn makes them more usable as capability insights.
Making meta-level arguments trying to argue that people should be more a-priori uncertain about any given model of theirs that is predicting short timelines. This is part of what @TsviBT is doing, I think. It’s indeed a meaningful alternate route, but (1) it’s fundamentally very abstract and hard to make strongly convincing (meta-epistemics, really?), (2) it may not be sufficient to move people enough without object-level arguments.[1]
Further, the fundamental problem is that even highly abstract/meta arguments that there are still blockers to AGI are arguments that there are still blockers to AGI. If such blockers exist, then (a) convincing alignment-pilled rationalists of this is good, (b) convincing capability researchers of this is bad, because then they’ll start figuring out what those blockers are and how to resolve them. (b) gets worse the more accurately and concretely you describe what the blockers are, but the main thing is that whether the impact of your argument is positive or negative depends on who reads it and is convinced by it. Admittedly, you can still control that to some extent:
You can post your arguments in places alignment-pilled rationalists are more likely to read them and capabilities researchers are less likely to read them (up to only sharing them privately with trusted people). But this is tricky, because there’s a lot of overlap in where the two demographics hang out, plus good arguments have the propensity to escape containment.
You can try to somehow phrase your arguments in manners that would be differentially more convincing to alignment-pilled rationalists over capabilities researchers. But this is also tricky, because the reasons people find arguments convincing are ultimately convergent (e. g., the more concrete and technical the arguments are, the more convincing they’d be both to alignment-pilled rationalists and to capabilities researchers).
As in, if people are falsely strongly convinced of short timelines because the discourse ecosystem only produces concrete arguments towards them and no concrete arguments against them, then convincing people to have appropriate over-model uncertainty may still lead to them being strongly overconfident in short timelines, because the object-level disbalance accounts for most of the update and it hasn’t been corrected.
(1) it’s fundamentally very abstract and hard to make strongly convincing (meta-epistemics, really?), (2) it may not be sufficient to move people enough without object-level arguments.
This may be true of people’s behavior, but I don’t agree with the characterization of “meta-epistemics”. It only becomes “meta-epistemics” because people, like, update without paying attention to why (narrator: it was because everyone else was updating). So then I ask “what convinced you of this” and then we go around and around in circles because no one can give a straight answer, and then we have to get into meta-epistemics I guess. I didn’t start the meta-epistemics!
I mean, to be fair, of course you’re supposed to make lots of updates all the time without being able to immediately explicitly account for every background assumption that went into every update. But like, it’s a really big update and a really big enthymeme.
This may be true of people’s behavior, but I don’t agree with the characterization of “meta-epistemics”. It only becomes “meta-epistemics” because people, like, update without paying attention to why
Well yes, and also obviously meta-probabilities of any level are flattenable to just object-level probabilities, so “arguing about meta-epistemics” is really just arguing about normal epistemics. But...
So then I ask “what convinced you of this” and then we go around and around in circles because no one can give a straight answer, and then we have to get into meta-epistemics I guess. I didn’t start the meta-epistemics!
… that. People perceive it as meta-epistemics, so you still end up doing things that look like “arguing meta-epistemics”. Which often parses as some perilously abstract navel-gazing. ¯\_(ツ)_/¯
Somewhat agree on the later half (and definitely agree on the last sentence...) but two counterpoints:
There are two benefits to the capability-researcher from pointing out a concrete blocker: potentially convincing them to switch research directions because of the blocker, and the actual information gained about the blocker. Whereas the alignment researcher would get the same benefit even if all they knew was that there was a blocker at all. Thus meta-arguments that don’t have information about the blocker are more asymmetric, even if they still help the capability researcher. I feel like the benefits outweigh the harms on meta-arguments because of this, but maybe I’m wrong. Also, I have this feeling that meta-arguments are more likely to be taken seriously by the kind of people who take the standard theoretical/abstract arguments for doom seriously (so less capability researchers and more alignment researchers), but maybe this feeling is unjustified.
I don’t think you have to make timeline arguments to improve the way the community hedges it’s bets. It seems like even in a world where the optimal subjective probability assessment puts 90% on short timelines, there are still resources that could be better spent if people hedged their bets better. As one example, there’s a lot of young people who I’ve talked to who are basically just desperately searching for a way to throw themselves at anything that could help with short timelines (I was one of them no more than a year ago), who could otherwise possibly make a substantial difference in a few years if they took their time and accumulated more skills and knowledge (probably multiplying their impact by OOMs or even sign-flipping it). The fact that @TsviBT still needs to ask for money for HIA is another example. Generalizing, there are probably a lot of misplaced resources right now. Especially because leverage on moderately long (10-30 years?) timelines seems highest.
I didn’t even consider this point. I’m sure most capability researchers already have capability advancement strategies as their top-of-mind idea, but I guess if the discussion on LW is sufficiently orthogonal to the discussion inside capability-researcher communities (seems likely?), then I would agree that this is bad. I would guess that this is non-obvious to most people on this site.
Do you feel like this is a direct tradeoff? Or is there another way to address some or all of these costs without pointing directly at concrete blockers to AGI? Maybe the right way to navigate this issue is some alternative approach(es)? The following options seem fine but not amazing.
Getting the community to hedge our bets more? It’s great that there are people working on things for worlds with no blockers left to AGI, but it feels like the pendulum has swung too far in that direction as of late. On the other hand, a lot of this might be more due to streetlighting than anything, so just telling people to hedge their bets more doesn’t really help with that. And “hedge your bets” is already an obvious strategy, so I don’t know if telling people to do it more helps at all. Although maybe its obvious only in some implicit / retrospective-y way s.t. hearing it said or emphasized is helpful to some people?
There might be some way to argue abstractly about blockers to AGI convincingly without pointing at specific ones. I know Tsvi has attempted that here, here, here, and here. I find these arguments more convincing the more I read Tsvi’s other work, which suggests that there is maybe more work to be done in making these arguments fully explicit and less reliant on background intuitions. Also, maybe there are other abstract arguments that can be made?
I think it mostly is, yeah.
The extent to which it’s rational to hedge your bets obviously depends on what the gamble is, i. e., how uncertain you are regarding what the outcome would be. If you’re near-certain about short timelines, it doesn’t actually make sense to “hedge your bets” by investing in some long-timelines approaches. So you have to argue away from that. Which can be done either by:
Making object-level arguments against short timelines – which mostly does necessarily require discussing blockers to AGI. And if you want to make your arguments more convincing, the best way to do so is by making them more concrete at the technical level, which in turn makes them more usable as capability insights.
Making meta-level arguments trying to argue that people should be more a-priori uncertain about any given model of theirs that is predicting short timelines. This is part of what @TsviBT is doing, I think. It’s indeed a meaningful alternate route, but (1) it’s fundamentally very abstract and hard to make strongly convincing (meta-epistemics, really?), (2) it may not be sufficient to move people enough without object-level arguments.[1]
Further, the fundamental problem is that even highly abstract/meta arguments that there are still blockers to AGI are arguments that there are still blockers to AGI. If such blockers exist, then (a) convincing alignment-pilled rationalists of this is good, (b) convincing capability researchers of this is bad, because then they’ll start figuring out what those blockers are and how to resolve them. (b) gets worse the more accurately and concretely you describe what the blockers are, but the main thing is that whether the impact of your argument is positive or negative depends on who reads it and is convinced by it. Admittedly, you can still control that to some extent:
You can post your arguments in places alignment-pilled rationalists are more likely to read them and capabilities researchers are less likely to read them (up to only sharing them privately with trusted people). But this is tricky, because there’s a lot of overlap in where the two demographics hang out, plus good arguments have the propensity to escape containment.
You can try to somehow phrase your arguments in manners that would be differentially more convincing to alignment-pilled rationalists over capabilities researchers. But this is also tricky, because the reasons people find arguments convincing are ultimately convergent (e. g., the more concrete and technical the arguments are, the more convincing they’d be both to alignment-pilled rationalists and to capabilities researchers).
It’s pretty cursed.
As in, if people are falsely strongly convinced of short timelines because the discourse ecosystem only produces concrete arguments towards them and no concrete arguments against them, then convincing people to have appropriate over-model uncertainty may still lead to them being strongly overconfident in short timelines, because the object-level disbalance accounts for most of the update and it hasn’t been corrected.
(1) it’s fundamentally very abstract and hard to make strongly convincing (meta-epistemics, really?), (2) it may not be sufficient to move people enough without object-level arguments.
This may be true of people’s behavior, but I don’t agree with the characterization of “meta-epistemics”. It only becomes “meta-epistemics” because people, like, update without paying attention to why (narrator: it was because everyone else was updating). So then I ask “what convinced you of this” and then we go around and around in circles because no one can give a straight answer, and then we have to get into meta-epistemics I guess. I didn’t start the meta-epistemics!
I mean, to be fair, of course you’re supposed to make lots of updates all the time without being able to immediately explicitly account for every background assumption that went into every update. But like, it’s a really big update and a really big enthymeme.
Well yes, and also obviously meta-probabilities of any level are flattenable to just object-level probabilities, so “arguing about meta-epistemics” is really just arguing about normal epistemics. But...
… that. People perceive it as meta-epistemics, so you still end up doing things that look like “arguing meta-epistemics”. Which often parses as some perilously abstract navel-gazing. ¯\_(ツ)_/¯
It is written:
Somewhat agree on the later half (and definitely agree on the last sentence...) but two counterpoints:
There are two benefits to the capability-researcher from pointing out a concrete blocker: potentially convincing them to switch research directions because of the blocker, and the actual information gained about the blocker. Whereas the alignment researcher would get the same benefit even if all they knew was that there was a blocker at all. Thus meta-arguments that don’t have information about the blocker are more asymmetric, even if they still help the capability researcher. I feel like the benefits outweigh the harms on meta-arguments because of this, but maybe I’m wrong. Also, I have this feeling that meta-arguments are more likely to be taken seriously by the kind of people who take the standard theoretical/abstract arguments for doom seriously (so less capability researchers and more alignment researchers), but maybe this feeling is unjustified.
I don’t think you have to make timeline arguments to improve the way the community hedges it’s bets. It seems like even in a world where the optimal subjective probability assessment puts 90% on short timelines, there are still resources that could be better spent if people hedged their bets better. As one example, there’s a lot of young people who I’ve talked to who are basically just desperately searching for a way to throw themselves at anything that could help with short timelines (I was one of them no more than a year ago), who could otherwise possibly make a substantial difference in a few years if they took their time and accumulated more skills and knowledge (probably multiplying their impact by OOMs or even sign-flipping it). The fact that @TsviBT still needs to ask for money for HIA is another example. Generalizing, there are probably a lot of misplaced resources right now. Especially because leverage on moderately long (10-30 years?) timelines seems highest.