I largely agree. But I think not-stacking is only slightly bad because I think the “crappy toy model [where] every alignment-visionary’s vision would ultimately succeed, but only after 30 years of study along their particular path” is importantly wrong; I think many new visions have a decent chance of succeeding more quickly and if we pursue enough different visions we get a good chance of at least one paying off quickly.
Edit: even if alignment researchers could stack into just a couple paths, I think we might well still choose to go wide.
I disagree with this view that someone’s vision could “succeed” in some sense. Rather, all visions (at least if they are scientifically and methodologically rigorous), if actually applied in AGI engineering, will increase the chances that the given AGI will go well.
However, at this stage (AGI very soon, race between AGI labs) it’s now the time to convince AGI labs to use any of the existing visions, apart from (and in parallel with, not instead) their own. While simultaneously, trying to make these visions more mature. In other words: researchers shouldn’t “spread” and try to “crack alignment” independently from each other. Rather, they should try to reinforce existing visions (while people with social capital and political skill should try to convince AGI labs to use these visions).
The multi-disciplinary (and multi-vision!) view on AI safety includes the model above, however, I haven’t elaborated on this exact idea yet (the linked post describes the technical side of the view, and the social/strategic/timelines-aware argument for the multi-disciplinary view is yet to be written).
This seems like a better model of the terrain: we don’t know how far down which path we need to get to find a working alignment solution. So the strategy “let’s split up to search, gang; we’ll cover more ground” actually makes sense before trying to stack efforts in the same direction.
+1. As a toy model, consider how the expected maximum of a sample from a heavy tailed distribution is affected by sample size. I simulated this once and the relationship was approximately linear. But Soares’ point still holds if any individual bet requires a minimum amount of time to pay off. You can scalably benefit from parallelism while still requiring a minimum amount of serial time.
I agree. It seems like striking a balance between exploration and exploitation. We’re barely entering the 2nd generation of alignment researchers. It’s important to generate new directions of approaching the problem especially at this stage, so that we have a better chance of covering more of the space of possible solutions before deciding to go in deeper. The barrier to entry also remains slightly lower in this case for new researchers. When some research directions “outcompete” other directions, we’ll naturally see more interest in those promising directions and subsequently more exploitation, and researchers will be stacking.
I largely agree. But I think not-stacking is only slightly bad because I think the “crappy toy model [where] every alignment-visionary’s vision would ultimately succeed, but only after 30 years of study along their particular path” is importantly wrong; I think many new visions have a decent chance of succeeding more quickly and if we pursue enough different visions we get a good chance of at least one paying off quickly.
Edit: even if alignment researchers could stack into just a couple paths, I think we might well still choose to go wide.
I disagree with this view that someone’s vision could “succeed” in some sense. Rather, all visions (at least if they are scientifically and methodologically rigorous), if actually applied in AGI engineering, will increase the chances that the given AGI will go well.
However, at this stage (AGI very soon, race between AGI labs) it’s now the time to convince AGI labs to use any of the existing visions, apart from (and in parallel with, not instead) their own. While simultaneously, trying to make these visions more mature. In other words: researchers shouldn’t “spread” and try to “crack alignment” independently from each other. Rather, they should try to reinforce existing visions (while people with social capital and political skill should try to convince AGI labs to use these visions).
The multi-disciplinary (and multi-vision!) view on AI safety includes the model above, however, I haven’t elaborated on this exact idea yet (the linked post describes the technical side of the view, and the social/strategic/timelines-aware argument for the multi-disciplinary view is yet to be written).
This seems like a better model of the terrain: we don’t know how far down which path we need to get to find a working alignment solution. So the strategy “let’s split up to search, gang; we’ll cover more ground” actually makes sense before trying to stack efforts in the same direction.
Yeah, basically explore-then-exploit. (I do worry that the toy model is truer IRL though...)
+1. As a toy model, consider how the expected maximum of a sample from a heavy tailed distribution is affected by sample size. I simulated this once and the relationship was approximately linear. But Soares’ point still holds if any individual bet requires a minimum amount of time to pay off. You can scalably benefit from parallelism while still requiring a minimum amount of serial time.
I agree. It seems like striking a balance between exploration and exploitation. We’re barely entering the 2nd generation of alignment researchers. It’s important to generate new directions of approaching the problem especially at this stage, so that we have a better chance of covering more of the space of possible solutions before deciding to go in deeper. The barrier to entry also remains slightly lower in this case for new researchers. When some research directions “outcompete” other directions, we’ll naturally see more interest in those promising directions and subsequently more exploitation, and researchers will be stacking.