This post’s examples are not very convincing, but I agree with the premise. The trouble is explaining why in a way that is clearly wrong-if-the-premise-is-wrong.
I think the main claim is that there is some “phase change” that will happen between AI today and AGI, rendering current mechinterp and control systems moot since post-change AI will have different affordances, tendencies and behaviours (in the same way that grown up humans are no longer surprised and delighted by peek-a-boo or similar object-permanence games which can elicit reliable reactions from babies). I do agree with the claim but the examples/elaboration given here don’t feel super substantiated.
Oh well put! I think the post also missed pointing out why iteratively working on existing models will not work. If you can please a baby, and keep working with said baby as it grows up, you’ll likely be able to please the adult version of the baby too.
The analogy breaks down if the baby grows up overnight and you don’t have time to adapt, but we’re not working just by ourselves—the ever improving models themselves are used in alignment and related work. E.g. judge models and model generated evals for evaluations, with the extreme end of it being deferring to aligned AIs to build future aligned AIs.
The post is about what “adulthood” means for goal engines, and where the vector from baby to adulthood points. Current AI safety work is only relevant to a “system that is still sufficiently baby-like”. But we should expect goal engines to be extremely mature. When you are negotiating with a human adult who is trying to maximize their company’s profit, there is no need to study the phenotype of the 3-month embryo that once scaffolded that human.
This post’s examples are not very convincing, but I agree with the premise. The trouble is explaining why in a way that is clearly wrong-if-the-premise-is-wrong.
I think the main claim is that there is some “phase change” that will happen between AI today and AGI, rendering current mechinterp and control systems moot since post-change AI will have different affordances, tendencies and behaviours (in the same way that grown up humans are no longer surprised and delighted by peek-a-boo or similar object-permanence games which can elicit reliable reactions from babies). I do agree with the claim but the examples/elaboration given here don’t feel super substantiated.
Oh well put! I think the post also missed pointing out why iteratively working on existing models will not work. If you can please a baby, and keep working with said baby as it grows up, you’ll likely be able to please the adult version of the baby too.
The analogy breaks down if the baby grows up overnight and you don’t have time to adapt, but we’re not working just by ourselves—the ever improving models themselves are used in alignment and related work. E.g. judge models and model generated evals for evaluations, with the extreme end of it being deferring to aligned AIs to build future aligned AIs.
The post is about what “adulthood” means for goal engines, and where the vector from baby to adulthood points. Current AI safety work is only relevant to a “system that is still sufficiently baby-like”. But we should expect goal engines to be extremely mature. When you are negotiating with a human adult who is trying to maximize their company’s profit, there is no need to study the phenotype of the 3-month embryo that once scaffolded that human.