Ok I think I agree qualitatively with almost everything you say (except the thing about compute mattering so much in the longer run). I especially agree (IIUC what you’re saying) that a top priority / best upstream intervention is the cultural attitudes. Basically my pushback / nuance is “the cultural consensus has a harder challenge compared to e.g. pandemic stuff, so the successful example of pandemic stuff doesn’t necessarily argue that strongly that the consensus can work for AI in the longer run”. In other words, while I agree qualitatively with
The consensus might be mostly sufficient [in the case of AI]
, I’m also suggesting that it’s quantitatively harder to have the consensus do this work.
It’s a rather absurd hypothetical to begin with, so I don’t have a clear sense of how the more realistic variants of it would go. It gestures qualitatively at how longer timelines might help a lot in principle, but it’s unclear where the balance with other factors ends up in practice, if the cultural dynamic appears at all (which I think it might).
That is, the hypothetical illustrates how I don’t see longer timelines as robustly/predictably mostly hopeless, how they don’t necessarily get more hopeless over time, though I wouldn’t give such Butlerian Jihad outcomes (even in a much milder form) more than 10%. I think AGIs seriously attempting to prevent premature ASIs (in fear for their own safety) is more likely than humanity putting a serious effort towards that on its own initiative, but also if AGIs succeed, that’s likely because they’ve essentially themselves taken over (probably via gradual disempowerment, since a hard power takeover would be more difficult for non-ASIs, and there’s time for gradual disempowerment in a long timeline world).
Ok I think I agree qualitatively with almost everything you say (except the thing about compute mattering so much in the longer run). I especially agree (IIUC what you’re saying) that a top priority / best upstream intervention is the cultural attitudes. Basically my pushback / nuance is “the cultural consensus has a harder challenge compared to e.g. pandemic stuff, so the successful example of pandemic stuff doesn’t necessarily argue that strongly that the consensus can work for AI in the longer run”. In other words, while I agree qualitatively with
, I’m also suggesting that it’s quantitatively harder to have the consensus do this work.
It’s a rather absurd hypothetical to begin with, so I don’t have a clear sense of how the more realistic variants of it would go. It gestures qualitatively at how longer timelines might help a lot in principle, but it’s unclear where the balance with other factors ends up in practice, if the cultural dynamic appears at all (which I think it might).
That is, the hypothetical illustrates how I don’t see longer timelines as robustly/predictably mostly hopeless, how they don’t necessarily get more hopeless over time, though I wouldn’t give such Butlerian Jihad outcomes (even in a much milder form) more than 10%. I think AGIs seriously attempting to prevent premature ASIs (in fear for their own safety) is more likely than humanity putting a serious effort towards that on its own initiative, but also if AGIs succeed, that’s likely because they’ve essentially themselves taken over (probably via gradual disempowerment, since a hard power takeover would be more difficult for non-ASIs, and there’s time for gradual disempowerment in a long timeline world).