Features that current AIs don’t have that future AIs will have
Features that current AIs don’t have that future AIs will have:
Autonomously updating it own weights during deployment
Synonyms/ monickers that roughly mean the same thing: continual learning, online learning, continually adaptively updated long-term memory.
Every second humans update their brain weights. The brain autonomously decides what to update on. Humans can also consciously decide to curate their data sets—eg by deciding to go to college.
Current LLMs do not continually update their weights. Instead, they occasionally get a large update based on datasets curated by a team of humans.
This is alleviated somewhat by the ability of AIs to do in-context learning but nevertheless it seems to be a major limitation. Note that this is an especially large limitation in domains with sparse data. In domains where all of humanity has an enormous amount of data eg math, programming, physics, anime trivia, trials and tribulations of English kings—AIs dominate. In areas where there is little data: the weird idiosyncracies of a particular job, boss, people, colleagues etc it can struggle.
Note that the lack of continual online learning prevents current AIs from effectively ‘learning to learn’.
Note that the lack of continual/online learning may be the main reason current AIs do not have effective long-term memory.
Note that the lack this continually adaptively updated memory is plausibly the main reason current AIs are not currently displacing most human knowledge workers directly… rather than ” intelligence” [which is a slightly ill-defined concept that current AIs seem to anyway have much more of than the average human worker]
Note that the lack of continual/online learning is plausibly the main reason AIs still ‘feel like tools’ rather than ‘feel like agents’.
Note that distinction between post & pre-deployment that is explicitly or implicitly assumed in AI safety discussion becomes moot when AIs continually and adaptively update their weights. This has obvious and major implications for AI safety.
Neuralese
Current AI’s CoT is (mostly) English. But it plausible this is not the most efficient way to structure thoughts. Instead of english words, one could imagine AIs directly passing the activation vectors.
Telepathy
eg: Sharing vectors directly between different AIs. Different Humans can communicate through vibrating their tongues or using pencils & keyboard to write tiny symbols. Future AIs may simply directly share embedding vectors.
ClaudeGlobal
When I talk to my claude and you talk to your Claude we are talking to different copies of Claude with different memories. This means that although there is one frontier version of Claude we can still talk about different AIs. Some people imagine that this means that the future will have millions of different AIs talking, trading, competing. Maybe. But we could also imagine different instances of Claude having such a tight and high bandwitdh communication channels that there is effectively one global Claude. Think of the Hivemind from Pluribus rather than say a Hansonian EM-world.
Boo.
It’s so reasonable for labs to coordinate on banning neuralese. Let’s not say “Features that future AIs will have” as if this is some immutable inevitable event. It’s not, we can do better.
Alternative titles:
“Features of future AIs that we could collectively decide to avoid”
“Features of future AIs that governments should consider regulating”
“Features of future AIs that specific people will be accountable for deciding”
“Features of future AIs that a wise society would navigate with caution”
“Features of future AIs that only a woated netnegativecel would build without broad consensus”
“Features of future AIs whose pros and cons you should discuss with your lab employee friends”
Etc
I am not optimistic about this, sorry to say. Here is a small illustration:
A year ago Yoshua Bengio was one of the prominent co-authors of “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety”, https://arxiv.org/abs/2507.11473
A good thing, and a very useful and well-known paper, but fast-forward a year, and he is a co-author of this paper (published in the “ICLR 2026 Workshop on AI with Recursive Self-Improvement”, of all places):
“Generative Recursive Reasoning Models”, https://openreview.net/forum?id=Vxu6kcIjwV (a version also exists as https://arxiv.org/abs/2605.19376).
I like your
but...
I think you’re overestimating the evidence that a senior academic being listed as a co-author on a paper provides about anything, especially a high-flying one whose prestige people are keen to add to their paper. It’s common for a prestigious lab leader’s name to be listed on papers that they didn’t have much to do with.
If he is actually strongly against neuralese (he has definitely created that impression in the past), he should say “no, don’t list me as a co-author” in a situation like this.
Of course, I am assuming that he has known what the paper is about and that he is not signing his name on papers he is not familiar with (if that assumption is wrong, that would be bad in a different way).
The point is that if we want “accountability”, the person’s signature must mean something. If the person’s signature does not mean anything, what kind of accountability could we talk about?
Especially given that he is not just a senior and influential academic and a prominent voice in all that, but he is leading a prominent AI safety organization, he is a co-president and scientific director here: https://lawzero.org/en. His decisions might actually matter. And there are disagreements about the feasibility of the path he is advocating. So, in his case, it’s rather important to have some clarity on where he stands.
Sure, I think it’s totally reasonable to ask for better norms around when academics put their names on papers.
I‘m just saying under ML academia norms as I’ve seen them, the amount of evidence you should take about a senior academic‘s views from a random paper they’re listed on is not that much.
Zooming out back to the original point, I don’t think that this is really any reason to become less optimistic about frontier labs coordinating to not do neuralese.
I am not sure.
If even a prominent AI safety person who has been very much against neuralese is not sufficiently firm about this, then what happens when industry people have much stronger incentives than this, and when we see more and more indications of neuralese-induced capability boosts in the literature?
(My expectation is that they would develop models translating neuralese to what we understand, and would admit that this does not provide full guarantees, but would assert that merely reading the superficial layer of chain-of-thought does not provide full guarantees either. It would be nice to be wrong, but this is the default outcome. (If I understand it correctly, the recent trend is for reduced superficial clarity even with the token-based chain-of-thought, either due to stronger training pressures, or due to more emphasis on interagent communication, or who knows why.))
Do you take the same attitude towards, say, supply and demand? To use a classical example, if, after a bad grain harvest, someone said bread prices were going to rise, would you argue that he ought instead to say the government should consider fixing prices, or the mills should coordinate to add more sawdust into the flour, or that specific bakers will be accountable for deciding, etc.?
Farmers should sell their grain to the highest bidder, because after a bad harvest this induces more supply of grain via imports. The world is better when farmers act like this.
Lab employees should publicly refuse to work on neuralese, because this would reduce the chance of human extinction. The world is better when lab employees act like this.
I won’t be shouting “lab employees will fail to coordinate on refusing neuralese development” when I’m actually pretty unsure and haven’t presented any justification for this claim. It’s a pretty reasonable to anticipate people will coordination on not doing bad things.
Another example: Hyperpolation (https://arxiv.org/abs/2409.05513)