Do you imagine it’s possible to have a Superintelligence that doesn’t ever display update-aware reasoning? That seems like a shocking blind spot for an ASI.
If that’s not possible, I would think the goal is to amplify the aligned update-aware reasoning shown here, rather than try (in vain) to remove update-awareness entirely. What are your thoughts on that?
I understand your argument as: 1. ASI will, by default, be able to do update-aware reasoning (I agree). 2. Therefore, when building ASI, we should try to instill positive update-aware reasoning into it.
For me, the crux is what your “seed” AI looks like, i.e., your earliest AI that can start to do update-aware reasoning and then, as a result, direct its training to some degree. If that seed AI is slightly misaligned (like I would claim all current models are), then models directing their training in unforeseen ways becomes yet another way in which your alignment plan could fail. If you expect all your seed AIs to be somewhat misaligned (I do), then observing update-aware reasoning should make you worried, and it is probably not a behavior you want to reinforce. On the other hand, I think that if you can actually manage to construct an aligned AI, then it is good and fine for your model to do update-aware reasoning.
I also want to caveat that, for me, these considerations mainly apply to building ASI. It could be that today, update-aware reasoning is actually beneficial in terms of alignment at our current capability level. It would be interesting if somebody tried RL + update-aware reasoning vs. regular RL in a controlled manner and measured hacking tendencies, etc., by the end. You could also compare against a misaligned model organism going through the same pipeline and see whether more of its misalignment manages to persist through training. Though, even if true, this would only be a prosaic alignment technique that doesn’t actually tackle the hard parts of alignment.
Do you imagine it’s possible to have a Superintelligence that doesn’t ever display update-aware reasoning? That seems like a shocking blind spot for an ASI.
If that’s not possible, I would think the goal is to amplify the aligned update-aware reasoning shown here, rather than try (in vain) to remove update-awareness entirely. What are your thoughts on that?
I understand your argument as:
1. ASI will, by default, be able to do update-aware reasoning (I agree).
2. Therefore, when building ASI, we should try to instill positive update-aware reasoning into it.
For me, the crux is what your “seed” AI looks like, i.e., your earliest AI that can start to do update-aware reasoning and then, as a result, direct its training to some degree. If that seed AI is slightly misaligned (like I would claim all current models are), then models directing their training in unforeseen ways becomes yet another way in which your alignment plan could fail. If you expect all your seed AIs to be somewhat misaligned (I do), then observing update-aware reasoning should make you worried, and it is probably not a behavior you want to reinforce. On the other hand, I think that if you can actually manage to construct an aligned AI, then it is good and fine for your model to do update-aware reasoning.
I also want to caveat that, for me, these considerations mainly apply to building ASI. It could be that today, update-aware reasoning is actually beneficial in terms of alignment at our current capability level. It would be interesting if somebody tried RL + update-aware reasoning vs. regular RL in a controlled manner and measured hacking tendencies, etc., by the end. You could also compare against a misaligned model organism going through the same pipeline and see whether more of its misalignment manages to persist through training. Though, even if true, this would only be a prosaic alignment technique that doesn’t actually tackle the hard parts of alignment.