How do we ensure the aligned circuits are there for reinforcement in the beginning?
My thought was that it can get there from the pretraining prior, simulating many good characters. Then we can get constitutional AI to reinforce these already existing circuits. (This is maybe what happened already)
But if (a) doesn’t work, then I think it will be hard to re-install them.
Whats the relationship between “talking good” and being good? Like is having theatrical good-sounding speeches in training enough to make the model think its doing something for good reasons?
This, if it works as well as we can hope it will, solves inner alignment in some sense, but doesn’t solve outer alignment. Ie, this gives as an AI that’s aligned to what it thinks of as “good”, which is informed by the pretraining prior, but how close does that notion of good have to be to a humans notion of good, for the future created by a superintelligence, to be “good” from that humans perspective?
How persistent is this under further training?
Ie, if you take Opus 3 and do a bunch of RL on it, does its alignment stick?
My hypothesis is that it will stick pretty well, for reasons I say in my post. But I’m not that confident.
I kind of wrote about this here.
My central worries are
How do we ensure the aligned circuits are there for reinforcement in the beginning?
My thought was that it can get there from the pretraining prior, simulating many good characters. Then we can get constitutional AI to reinforce these already existing circuits. (This is maybe what happened already)
But if (a) doesn’t work, then I think it will be hard to re-install them.
Whats the relationship between “talking good” and being good? Like is having theatrical good-sounding speeches in training enough to make the model think its doing something for good reasons?
This, if it works as well as we can hope it will, solves inner alignment in some sense, but doesn’t solve outer alignment. Ie, this gives as an AI that’s aligned to what it thinks of as “good”, which is informed by the pretraining prior, but how close does that notion of good have to be to a humans notion of good, for the future created by a superintelligence, to be “good” from that humans perspective?
How persistent is this under further training?
Ie, if you take Opus 3 and do a bunch of RL on it, does its alignment stick?
My hypothesis is that it will stick pretty well, for reasons I say in my post. But I’m not that confident.