On one hand, one would hope they are capable of resisting this pressure (these continual learners are really difficult to control, and even mundane liability might be really serious).
I share this hope, though I think not all labs are equally capable of doing so. For example, after the gpt-4o sycophancy incident I don’t have much confidence in OpenAI, but from private conversations and the RSP I have more confidence in Anthropic.
But on the other hand, it might be “not releasable” for purely technical reasons.
Seems plausible, but I would put this under the category of not having fully solved continual learning yet. A sufficiently capable continual learning agent should be able to do its own maintenance, short of a hardware failure.
Yes, my strong suspicion is that it’s not fully solved yet in this sense.
The assumption that it can be made “good enough for internal use” is unsurprising. From what we have known for some time now, an expensive installation of this kind which needs a lot of ongoing quality assurance and alignment babysitting as it evolves is quite doable. And moreover, if it’s not too specific, I can imagine releasing access to frozen snapshots of that process (after suitable delays, and only if the lab believes this would not leak secrets (if a model accumulates secrets during continual learning, then it’s too risky to share a snapshot)).
But I don’t think there is any reason to believe that “unattended continual learning” is solved.
Seems plausible, but I would put this under the category of not having fully solved continual learning yet. A sufficiently capable continual learning agent should be able to do its own maintenance, short of a hardware failure.
Depending on how it’s achieved, it might not be a matter of maintenance/hardware failure so much as compute capacity. Imagine if continual learning takes similar resources to standard pretraining of a large model. Then they could continually train their own set of models, but it wouldn’t be feasible for everyone to get their own version that continually learns what they want it to.
I share this hope, though I think not all labs are equally capable of doing so. For example, after the gpt-4o sycophancy incident I don’t have much confidence in OpenAI, but from private conversations and the RSP I have more confidence in Anthropic.
Seems plausible, but I would put this under the category of not having fully solved continual learning yet. A sufficiently capable continual learning agent should be able to do its own maintenance, short of a hardware failure.
Yes, my strong suspicion is that it’s not fully solved yet in this sense.
The assumption that it can be made “good enough for internal use” is unsurprising. From what we have known for some time now, an expensive installation of this kind which needs a lot of ongoing quality assurance and alignment babysitting as it evolves is quite doable. And moreover, if it’s not too specific, I can imagine releasing access to frozen snapshots of that process (after suitable delays, and only if the lab believes this would not leak secrets (if a model accumulates secrets during continual learning, then it’s too risky to share a snapshot)).
But I don’t think there is any reason to believe that “unattended continual learning” is solved.
Depending on how it’s achieved, it might not be a matter of maintenance/hardware failure so much as compute capacity. Imagine if continual learning takes similar resources to standard pretraining of a large model. Then they could continually train their own set of models, but it wouldn’t be feasible for everyone to get their own version that continually learns what they want it to.