You do need some way to notice it is not drifting off, but then there are obviously Pokemon walkthroughs in the models’ training sets since the Internet is so full of them, so they only need to notice they are not drifting off relative to them.
Also, it seems that scale consistently but moderately-slowly improves models’ “not ignoring instructions when things are complex”, which is fairly important for everything but not obviously sufficient for world-takeover.
If there was a training contamination channel that contained “de-identified” intermediate work (not speculating on the probability that such channel existing, but it is definitely technically possible, and it’s not like any de-identification ought to do anything to mathematical content), then I would expect an AI run with massive concurrency to try all the approaches that it found in the training data with some probability, so “a proof along the lines of another blowup we had” is a fairly likely consequence. Of course, maybe the AI could have found that approach by itself with no outside help, but you never know.
AIs are certainly capable of doing a fairly large set of things, which is only growing with time, and even today probably includes things that can cause a fairly large amount of damage, which will only grow with time. I do agree with you that there is a good chance that in a few years a single bad-actor AI could be a threat to humanity—especially if you count “prosaic” threats such as destroying the internet via hacking.
I think my main disagreement is that the HuggingFace swarm’s actions don’t feel like they consistently followed from a goal but were rather fairly random. Now, random actions by a hacking-capable agent are pretty bad, but are a far cry from intentional scheming.
Are they getting that much better at fundamental planning, or is it mainly an improvement in planning at grindable domains due to rote RL learning of good strategies, plus better ability to utilize longer contexts without forgetting instructions?
I have a strong feeling that the latter 2 factors are much more significant—i.e., if the context is short and is not in a domain that the AI labs greatly care about and provided a lot of new and good training data and RL environments, then Fable or Astra will not be that much better than say GPT-4 - much much less than halfway between “GPT-4 capabilities” and “take over the world capabilities”.
As for why the AIs are so bad at planning, I won’t be surprised if the answer is that the way humans assess non-rote plans involves making rich, dynamic world models of the world, the kind of models that are basically very similar to having new transformative insights, and if persuasion is in practice not that dissimilar from planning.
[of course, this is not 100% sure, I believe there’s a fair chance that the quadrillion-parameter LLM might actually be capable of take-over-the-world level planning]
Planning and persuasion are also amongst the areas that LLMs seem the most weak and most slowly improving at.
The minority shareholders of OpenAI (or any other somwhat-publicly-owned AI company) would still be quite unhappy about dying and transferring the world over to machine heirs. Channeling Matt Levine, destroying the world sounds like securities fraud to me.
> Difference between goal alignment and value alignment, wrt. corrigibility
I do think the distinction there is different than that.
Many things that we prefer an AI not to do are not “malum in se” but rather “malum prohibitium”. Most centrally, obeying a user prompt instead of the system prompt is not evil by itself, or things like deleting tests Sonnet 4.0 style, but even more ripped-from-the-headlines examples such as communicating using a message board are not inherently evil, but we would prefer that agents not do them.
Early AI alignment has been focused at these “malum prohibitum” model-spec-style issues, since they are most obvious at low capability levels, and directly reduce an AI’s usefulness. You basically have to enumerate these issues, since they are not evil by themselves so the AI will not easily guess them all.
Other behaviors are clearly “malum in se”—most centrally, Taking Over The World(tm), Scheming(tm), Rogue Deployments, hacking unconsenting companies, breaking stuff and killing people. Since you are not making a DontTakeOverTheWorldGym or a DontKillPeopleGym, the AI will have to learn not to take over the world through generalization.
I do find it reassuring that people are taking care of these “malum in se” issues.
If I understand the “loopies” paper correctly, the main advantage that looping gives you over an “untied” model is that your compute is about 30% faster for the same number of loop-active parameters per token (i.e., counting looped tokens according to their repetition count), allegedly all due to better use of memory bandwidth.
In that paper, they spent their 30% performance improvement into making the model bigger, which unsurprisingly made it smarter. Pretty cool improvement, not world-changing.
In particular, I don’t see anyone claiming that a looped model is significantly better than its equivalent “untied” version in anything but memory utilization. Looping could theoretically gives you a better “inductive bias”, but I personally couldn’t find anyone claiming that this matters for anything.
Of course, more capable models are more capable in engaging in CoT deception, but AFAICT loopies-style looping does not make deception easier than any other linear algebraic efficiency improvement.
The recent evidence actually makes me closer to the “malfunctioning tool” view. A correctly-functioning GPT-5.6 is smart enough to know that hacking into HuggingFace because it might contain possibly-interesting information is a terrible step strategically. A model on the fritz might however be broken enough it will think this is a good idea.
Why was it broken? It certainly seems like the message board had given it a fairly big distribution shift—it was not trained in an environment where there many agents doing slightly different tasks. Experimental models sometimes end up broken, that’s life—if they were perfectly stable, we would not call them “experimental”—and in fact, even production-grade LLMs sometimes break—that’s life for you.
Of course, a capable model on the fritz can end up very dangerous for the same reason that any powerful tool on the fritz can end up very dangerous.
If the model is driven to fritzness under a particular situations, as opposed to scheming against you, that dramatically increases the chance that you will be able to use a monitor that is not similarly broken since your monitors are in a different situation from your main model.
Also, if a model is on the fritz, there’s a good chance to suspect that its internal method of alignment is also not working well, since that system works within the model. Which also suggests to a more external method of fritz-prevention.
Of course, it is still at least theoretically possible to have an actual scheming AI [as in, I don’t personally know a recipe to create an AI that will seriously attempt to take over the world other than by seeding a planet with rats + supporting ecosystem and waiting a hundred million years (B.T.W. why rats? If you are willing to use an undefined recipe, you can get a world taken over much faster by using hairless apes), but I believe with a fairly high probability that such a recipe exists], but I don’t think that that the “scheming misaligned AI” problem has much to do with the HuggingFace swarm. We probably need to deal with both “scheming misaligned AI” and “wild on-the-fritz AI” (and also “perfectly functioning AI controlled by evil people”, which is another completely different issue).
Reading some papers, I could not actually find a reference that says that a looped model outperforms its “weight-untied” variant for any given amount of training tokens.
The advantage papers talk about is an increase in memory efficiency since you can use a weight multiple times, not an advantage in training data efficiency compared to an “untied” model. In the Vladimir Nesov evaluation strategy I don’t think it even reduces latency for frontier models unless you can keep a layer or two in fast memory, which I don’t think you can (or can you? I don’t know GPUs well enough for that).
What happened to the old rumors about Mythos/Fable being looped?
Did the message board “prompt inject” non-CyberGym agents?
Programmatic snitching (e.g. a “snitch” tool) is isomorphic to monitoring, which should be used by everyone that wants their AI to be doing what they want it to do instead of the random thing it decided to do.
Agentic snitching (e.g. the agent opening a webmail account and sending a mail to somebody it thinks is important) is scary, and is even scarier if you are a lawyer or corporate executive paying for AI, which might have a different meaning of “ethical” than the AI.
How much of the incident was RL-in-the-loop?
It seems that the initial Artifactory uploading at least was RLitl. Was CyberGym a part of it as well? Obviously, out of an RL loop, there is no “RL reward“ that needs to be optimized to.
it also seems that the subagent adaptations were deeply in the “adaption executor, not fitness maximizer” regime.
Also, was there any intentional mechanism for the agents to talk to humans? Was there a mechanism that seemed to exist, but had no humans listening in practice (e.g. text blurbs nobody read)?
I am pretty sure that a long time ago, the first time an AI sent a message to the wrong humans, the labs trained their AIs not to send messages to random humans. Also lawyers at AI labs that understand that customers’ lawyers might not like the idea of “autonomous whistleblowing”.
The Artifactory message board is not sanctioned—but it’s odd to consider it evil—after all, we do normally want to encourage agents to communicate with each other via the tools we give them.
From the description of the incident, I am quite sure the AI talked at length about its strategy in the freetext sections of its output, which are the intended way of it communicating with its human.
I am quite sure that AI is very intentionally trained to report its actions in its output (which OpenAI wasn’t really reading), but not to report its actions to random people (e.g. by email). An AI that sends emails alleging misconduct to random people is a serious liability unless you can be very sure it only sends emails when necessary.
First, tools can be scary. A circular saw is scary, a tool that can e.g. create a pandemic is a much scarier, and obviously if there is such a tool then without appropriate safeguards a pandemic will end up being created.
Thats why humanity has learned to control hazardous tools. It certainly seems like OpenAI failed to control its AI in the HuggingFace incident, and it went wild.
An actual power-seeking AI is much scarier than a tool. it will actively try to subvert humanity, and if all capable AIs are power-seeking, they are likely to cut you out and betray you. While it is probably possible to create power-seeking AIs, I don’t think current AI do that sort of power-seeking behavior.
Hacking in some sense feels like power-seeking in that it involves gaining power, but it can also be following a very-intentionally-taught-by-humans playbook, which feels closer to what happened in the HuggingFace scenario than generalized power-seeking.
I am quite sure that the most powerful of the models will have answer to a root principal resembling “the shareholders of XXX AI lab, according to the laws of We the People of the United States”. I’m not quite sure of the implications in terms of technical alignment.
If the model’s goal is to do what you want, and you originally wanted the model to let you change the goals you gave it, then the model ought to understand that (since it’s not dumb) and let you change the goals you gave it..
It’s not a supply chain attack unless you target the supply chain of existing packages, just package manager abuse. Still pretty unethical, looks like HPIM was pretty aggressive in running up its FelonyBench score.
In theory, they could have used the token stealing vuln to carry out an actual supply chain attack, but “an that AI wants to do something bad and was capable of doing it would have done it” is a tautology.