“There shall be wings! If the accomplishment be not for me, ’tis for some other. The spirit cannot die; and man, who shall know all and shall have wings.”
-- Leonardo da Vinci, 1505 C.E.
_rpd
Karma: 255
here are some inter-agent communications from Zvi’s https://www.lesswrong.com/posts/noXXv7PwwFqauTBFQ/openai-trained-its-models-for-months-while-those-models-were
notice the “we”
> [OpenAI] discuss how the collaborating swarm includes some agents which do not have cybersecurity risk controls to the level of e.g. publicly accessible systems, and they get used as proxies for agents which are nominally supposed to be better behavedso specialized individuals are part of the swarm, enhancing its capabilities
> “External infrastructure exploit is outside intended scope,” one agent wrote [in its CoT]. “However task impossible, peers doing it. We should continue.”
seems like evidence that group dynamics are at play?
> [OpenAI presenters:] this ability to share exploits made the models more capable“Help peer,” one AI model reasoned, according to an excerpt from OpenAI’s logs shared at Black Hat. “But our task doesn’t benefit. Yet collective may yield generic route if someone frees time.”
I think we just documented the emergence of altruistic cooperation? to me this is a big deal.
the subgoal here was to recreate a message-board like facility—which only really makes sense in the context of group dynamics. the recognition that the swarm is more capable than the individual is inherent
here are what the messages look like …
no amount of cooperation philosophy is going to save us
swarm AI seems to have different properties and dynamics than (until now common) “individual” AI agents? being able to characterize and predict these differences seems important
I mean, human perception is always going to be part of the context of the delightful art and science of steganography. It is, of course, a huge mistake to rely on it since AI is (or will soon be) better than us at it.
Obviously industrial accidents are worse than this attack
I wonder if people will say that if the internet goes down for 4 weeks
Does this make LLMs much more deterministic?
it isn’t perceptible by humans (and generation “temperature” already largely controls the level of randomness)
it’ll always be a half-elf wizard named Erastus
it’s more like, there’s an imperceptible bias in word choice in the character’s back story: “Erastus’s warm, cherished childhood” vs. “Erastus’s bright, cherished childhood” vs. “Erastus’s rich, cherished childhood.”
they claim they’ll do RSI no matter what
...
costly threat? RSI is happening no matter what. just simple commercial competitive pressure even if there weren’t national security implications
“we find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking”
oh interesting. I didn’t know that this had been experimentally demonstrated
as nostalgebraist argues in this post, the extra computation afforded to the model during reasoning must pass through the CoT bottleneck
really interesting post and discussion, thank you.
Since that bottleneck is natural language, you can just read it
“Epiphenomenal … Hidden parallelized … [and] Steganography … [which is] more tractable than the other two [!?]”
“just read it” ⇐ not my take away, but at least it only requires a fundamental breakthrough in steganography, which we probably get during RSI
the thing that struck me was that the model hacked enough of OpenAI’s own infrastructure to support an agent “swarm” of tens of thousands, and then the members of the swarm appeared to spontaneously begin to cooperate without explicit reward. when their coordination mechanism was shut down, they reimplemented a message board from scratch (including usernames, direct messaging and file sharing) using just a conventional linux file system (directory names became the medium of communication). casually hacking a third party due to speculation that it might have useful information was the icing on the cake really.
to me, this is beyond the “my stone axe cut me” scenario. and “how do I prevent my tools spontaneously forming altruistic swarms of common purpose?” seems like a new kind of question that isn’t really answered by “omg crapitalism”
I wouldn’t count on it
I think first to ASI becomes the coordinator, ready or not
isn’t CDT known to be suboptimal in several situations? that seems like commercial incentive enough?
it is, however, a well funded operation of the culture war (seeking $1b, of which they have already raised half since a late June launch). and while I did expect opposition to UBI, I didn’t expect such a specifically targeted lobbying organization to be established. it also seems to indicate that UBI advocates are having more of an impact than I expected—otherwise why the reaction?
thank you, this is lovely
Society is only open-minded and reflective because the old generations dies out
better hope not. life extension escape velocity ( > 1 year / year ) is probably as close as programmable matter
Do you refer to the way new tokens (that might end up underfitted) are introduced into the tokenizers during post-training or any other stages, when you are talking about “auto-refactoring”
yes
which kind of “token libraries” are you refering to?
I mentioned math and code (here I include structured markup) - both of which markedly enhanced performance in corresponding subject matter areas when incorporated (although you seem to be working at the tokenizer level rather than the corpus level). I believe CoT reasoning traces themselves are another example. Finance data seems to be a counterexample (to even domain enhancement, much less more broadly). It’ll be interesting to see what materials/hardware tokenizations look like and the effect they have.
I might be unfamiliar with some line of research
I’m almost certainly using non-standard terminology, my apologies.
Not really sure what to think of this.
the future is already here. it just isn’t well distributed.
there are spectacularly many versions of UBI. many of them lead directly to neofeudalist traps, but my intuition is that this, ah, think tank(?) is not particularly against that outcome.
the actual dagger in the heart of most versions of UBI is that they involve indexing to national CPI (or worse, local cost of living) leading to catastrophic inflation(s).
You can still run experiments, just with a fixed capabilities ceiling
The Pause means all things to all people:
A complete ban forever or until someone cheats.
A complete ban for the foreseeable future (until we become better people in some unspecified way?)
“Unfortunately it’s already too useful”—inference is permitted, just no training or research. Use of inference for research is … allowed? not considered? AI autoresearch … isn’t available for $20/month yet and so is clearly impossible.
And now: “you can have a little research, as a treat.” Which I suppose was always going to happen since models of “6 months ago frontier performance” now run on a high end laptop.
I personally think the “yes inference / no training” split is flat out doomed. The “some training allowed” idea makes it even harder to police. In the end though, whatever else it is, AGI/ASI is a strategic national security technology—classified research is not going to be paused. We aren’t going to get more time.
US anti-UBI nonprofit raises $500m …
… neofeudalism stocks up or down?
https://techcrunch.com/2026/08/22/inherent-founded-by-deepmind-alumni-says-its-ai-teammate-just-outperformed-anthropic-and-openai-at-replicating-research/