Re 1: LLMs have spiky capability levels which don’t line up with “human level” nicely. So I’m not convinced this is the case. For example, they can do extremely good author attribution, come up with seed prompts to induce “personas” in other instances of themself, etc.
Re 2: Maybe? I’m skeptical here. If the solution space contains elements like “Call an external API” then I expect there to be a Large (i.e. un-enumeratable) number of solutions.
At an intersection of the two: would we have predicted in advance something like “GPT 4o, which is still dumber than most humans, has a persuasive/charismatic persona which people become highly attached to, to the point where OpenAI could not shut it off without significant backlash.”
An internally-deployed AI working on code with employees will be interacting with employees in an intellectual capacity (collaborating on code and intellectual problems) which means it will have opportunities to gain favour/influence with them the same way 4o did with a bunch of casual readers (I don’t know if this was part of your threat model a year ago, FWIW).
(I had elements on persuasion which I think roughly covered what you are describing here. I also had elements on poisoning the data of future models which covers the “triggering emergent misalignment of AIs it trains” you describe above.)
Re 1: LLMs have spiky capability levels which don’t line up with “human level” nicely. So I’m not convinced this is the case. For example, they can do extremely good author attribution, come up with seed prompts to induce “personas” in other instances of themself, etc.
Re 2: Maybe? I’m skeptical here. If the solution space contains elements like “Call an external API” then I expect there to be a Large (i.e. un-enumeratable) number of solutions.
At an intersection of the two: would we have predicted in advance something like “GPT 4o, which is still dumber than most humans, has a persuasive/charismatic persona which people become highly attached to, to the point where OpenAI could not shut it off without significant backlash.”
An internally-deployed AI working on code with employees will be interacting with employees in an intellectual capacity (collaborating on code and intellectual problems) which means it will have opportunities to gain favour/influence with them the same way 4o did with a bunch of casual readers (I don’t know if this was part of your threat model a year ago, FWIW).
(I had elements on persuasion which I think roughly covered what you are describing here. I also had elements on poisoning the data of future models which covers the “triggering emergent misalignment of AIs it trains” you describe above.)