Hold the line, brother! Don’t normalize handing off trust to untrustworthy AIs! Just because it’s probably going to be done, doesn’t mean we should treat it as reasonable and good!
I acknowledged the “our alternatives our worse” possibility too; I don’t think we disagree that much substantively.
...
I agree “trust handoff” seems maybe the same concept as “giving up on control,” except that I think “giving up on control” seems like something that could be already true today (is there ~any effort to control our AIs now?) even though the AIs are too dumb to take over. Whereas you haven’t handed off trust until you reach the point where the AIs really could screw you over if they wanted to.
In what sense could the AIs be uncontrolled now? I have just re-read Anthropic’s Model Card for Claude Opus 4.6. The evaluation had Anthropic believe that “Given that Opus 4.6 did not improve over previous models in the higher performing out of the two main settings, we do not believe that the risk of successful sabotage is increased from Opus 4.5 (italics mine—S.K.). Like Opus 4.5, we believe that Opus 4.6 is likely unable to conduct significant subtle sabotage without such sabotage often becoming obvious.” Suppose that Opus 4.6 were to sabotage research. Then it would be clear from transcripts, especially if the AI-2027 forecast holds and a half of tokens is checked by weak models which are ten times cheaper.
Maybe you mean that if the model displays the ability to sabotage research without referencing the task, then monitoring becomes a far more complex problem, requiring the company to apply interpretability tools to activations and/or that such tools aren’t THAT cheap?
Hold the line, brother! Don’t normalize handing off trust to untrustworthy AIs! Just because it’s probably going to be done, doesn’t mean we should treat it as reasonable and good!
I acknowledged the “our alternatives our worse” possibility too; I don’t think we disagree that much substantively.
...
I agree “trust handoff” seems maybe the same concept as “giving up on control,” except that I think “giving up on control” seems like something that could be already true today (is there ~any effort to control our AIs now?) even though the AIs are too dumb to take over. Whereas you haven’t handed off trust until you reach the point where the AIs really could screw you over if they wanted to.
In what sense could the AIs be uncontrolled now? I have just re-read Anthropic’s Model Card for Claude Opus 4.6. The evaluation had Anthropic believe that “Given that Opus 4.6 did not improve over previous models in the higher performing out of the two main settings, we do not believe that the risk of successful sabotage is increased from Opus 4.5 (italics mine—S.K.). Like Opus 4.5, we believe that Opus 4.6 is likely unable to conduct significant subtle sabotage without such sabotage often becoming obvious.” Suppose that Opus 4.6 were to sabotage research. Then it would be clear from transcripts, especially if the AI-2027 forecast holds and a half of tokens is checked by weak models which are ten times cheaper.
Maybe you mean that if the model displays the ability to sabotage research without referencing the task, then monitoring becomes a far more complex problem, requiring the company to apply interpretability tools to activations and/or that such tools aren’t THAT cheap?