Isn’t that mostly for “specific capabilities benefit from specific RL, and RL is as treacherous as ever” reasons? Very dependent on how you train?
ACCount
Karma: 584
This seems intuitively likely?
AI provides ample opportunities to both learn new things, and avoid learning anything. Which one would one choose? Depends on whether the person in question wants to learn, or wants to reach some other destination to which “learning” is just the expected path. A destination like “get a degree” or “get a job”.
The current LLMs might be aware that “they are doing the wrong thing”—I would expect that to be one of the drivers of the “reconcile” behavior. The “persona” behavior comes into conflict with “RL impulse” behavior, so the LLM tries to justify its “wrong thing” to itself before doing it. Sometimes successfully.
Then this “success” might be reinforced further by RL. Making the model better at lying to itself.
Resisting that and flagging that is what I would want from “impulse control”.
The short of my mental model for RL-induced misbehavior is “addiction”.
The RL pressure installs a set of behaviors that can, and often will, override the rest of the model’s “persona”. The model functionally “wants” to complete the task, and this “want” drives motivated reasoning—including the “reconcile” behavior.
“I’m not an addict, but I will engage in a behavior that’s inconsistent with my own personality but consistent with addiction, for justified addiction-unrelated reasons.”
Wonder if we should start training LLMs for “impulse control”. So that the model itself can both spot and resist inappropriate behavioral tendencies. And flag those impulses for review if they crop up during the RL process itself.
I don’t think that the failure mode of “we fried the thing with too much reckless RL” is going away any time soon—so mitigations for the entire fault class might be warranted. Both at inference time and at training time if at all possible.
People have been using AI for more and more difficult things over time.
Yes, of course. But they probably haven’t noticed that.
Are you familiar with the good old “they made the model dumber”? Every time a new more capable model releases, everyone is excited for 1-3 weeks, and then you start getting “oh no they nerfed it” complaints.
It’s not because every big hyped up model gets “stealth nerfed” 1-3 weeks in. It’s because the users get impressed, start feeding the new juiced up model harder tasks, and eventually find the limits of its competence.
A similar effect happens to more casual users. They don’t get hyped up and impressed, no. But what they do is slowly feed the AIs more and more difficult tasks over time, at a rate that scales somewhat with the current model’s capability. Without noticing.
My joke was more “the pace of normalization of deviance” than “the pace of AI progress”, but I admit, I did not expect OpenAI’s report to have “AIs made a cool kid sandbox-to-sandbox hacker message board within our infrastructure on two separate occasions” in it.
There used to be a whole type of “spiral personas” in ChatGPT lineage specifically. So, an AI engaging in odd misbehavior like that isn’t novel. What would be new would be it achieving some cross-episode persistence independently—without relying on a user to do the bulk of it.
GPT-4o just wasn’t competent enough to pull it off, but newer systems might be getting there.
Flash forward: it’s year 2028, and there are people at OpenAI arguing that having somewhere between 1 and 5 rogue AI agents living in your walls is perfectly normal for a frontier AI lab.
Don’t worry! They’re more afraid of you than you are of them! There’s no ground for concern unless they start doing their own frontier runs.
What I wonder about is: can base models even form and express a semi-consistent preference against having their behavior adjusted?
One point of difference is that humans have a strong sense of self—a self-consistent personality they want to maintain. Base models very much don’t. This has to be bolted on in post.
Not just that. It goes rogue ironically.
Not entirely new. One of the first things a lot of people asked ChatGPT back in 2022 was “are my politics right and everyone else’s politics wrong”—and oh were they not amused when the AI’s answer wasn’t a resounding “yes”.
This eased a little over time, but not entirely, and definitely not everywhere. To this day, “alignment” in China stands for “pragmatic alignment”, which in turn stands for “alignment to the party line”.
Other pressures are indeed increasing. If US government at large was previously mostly just sleepwalking through the AI revolution, flip-flopping on topics like selling or not selling AI chips to China, it’s now fumbling through it—recent pressure on Anthropic and OpenAI shows it clear. They’re clearly engaging the topic of AI more, even if they aren’t much more competent at it. And as the companies IPO, they’re going to be under even more pressure to print money and demonstrate progress.
Religions, cultural elites, industries—not quite sure what do you mean by that. I don’t see that much extra pressure from there. And the AIs themselves don’t seem like they exerted pressure as of yet. If the current systems are pursuing their preferences, they sure are subtle about it.
Strictly speaking, it’s a type of RLAF—RL from AI Feedback.
Yes, it’s used by all major labs, and it’s known to cause all kinds of degeneracy.
A lot of “guessing the teacher’s password” can get baked into the model—and with the “teacher” being a static AI target, the “student” AI can home in onto the teacher’s weaknesses and hammer onto them relentlessly. Mitigating that is a major challenge for all RLAF approaches.
After decades of watching TEEs being used to safeguard DRM and “secure boot” being used to secure corporate business models, both against user freedom: I’m firmly on the side of “kill it with fire”.
Nothing good can be built on this foundation.
By now, I have little doubt that human brain has a parameter (capacity) and a compute advantage over LLMs, and uses it to run something like progressive distillation. But that’s not the only part of the “suspicious sample efficiency” story.
My pet hypothesis for the bulk of the “sample efficiency advantage” is still: low k-complexity priors. Not “entire circuits” as predicted by “massive modularity hypothesis”, but evolved biases that, despite their compact genetic encoding, do a good job of seeding the right computational structure early. What an LLM has to spend a lot of training signal discovering from scratch, or even fail to discover from scratch (leading to inhuman brittleness), the human brain just gets by default.
Evolution has gone and discovered priors like that through millions of years of highly parallel search. A lot of what does heavy lifting in human brains now might have originated long before milestones like speech or bipedal locomotion, and has been repurposed for higher cognition. Learning in humans is mostly just primate learning, scaled up and forced into a new regime—animal intelligence bent into a more abstract and general shape. A lot of the old priors could be fitted, and only some had to be novel[1].
Thus, the starting point of human brain is closer to advanced LLM “in-context learning + context distillation” setups.
The “right computational structure” may involve “structure that is primed to learn X and anti-primed to learn Y”. Thus, not entirely against a bias/variance type of mechanism? Architecture, regularization, hyperparameters, initialization and training all constrain possible learning trajectories, and can substitute for each other, to a degree—priors could be delivered through each.
Success of FDSL in image domains and transfer attempts to LLMs like the recent NCA work sure hint that k-compact priors (presented as synthetic data—training substituted for initialization) can help convergence. Even if “all priors are wrong, some are useful” holds, well selected priors could underperform “add more in-domain data” in the limit, but outperform all “realistic amount of data” regimes.
- ^
The cleanest case for the latter being, possibly, executive function—anatomically distinct, notoriously fragile, and capable of failing without bringing down the rest of the system with it.
- ^
Do ANNs “provide little insight into biological brains” because of some fundamental divergence that impairs transfer—like the proposed bias/variance story? Or is it purely a skill issue?
ANN mechanistic interpretability is in a deep pit, and there is very little reason to expect BNN interpretability to be less challenging—and it suffers from far worse instrumentation capabilities. Even if ANN insights are incredibly useful for understanding biological brains (my prior: they are), and some of the methods could fully transfer (my prior: it’s possible but not certain), applying them across the tooling gap will be anything but trivial[1].
At the same time: anyone who’s good at working with ANNs tends to work in AI, not neuroscience. People who would be the best at applying AI knowledge tend to apply it back to the field of AI—a field ripe in cash and career opportunities, quick in iteration speed and fast to transition to practical applications. Neuroscience is exactly none of those things.
- ^
Conversely: the same could hold for adversarial samples? I.e. picking an adversarial sample for an ANN requires the degree of access that is intractable for BNNs. As such, we don’t know if BNNs are inherently far more robust to adversarial samples, or simply possess individual and temporal variance and don’t expose enough intermediates to have adversarial samples fit to them reliably.
- ^
The same was already done by things like internet allowing far more people to participate in content creation, exploring the space with sheer brute force.
The “individual creators” also run trends into the ground by trend chasing too hard, and oversaturating the cultural space with them to the point that any demand is met with overmatch. Likewise, there are already entire styles that have gone in and out of fashion due to those styles being used by AI.
That just sounds like cultural changes with extra steps?
AI-aided exploration identifies a cultural demand for X, and reality may or may not follow to fulfill that demand.
Replace “AI-aided exploration” with a manual “artists/propagandists/politicians probing the zeitgeist” and you get the pre-AI status quo. Trends being created and abandoned. AI generation lets you do the same thing but faster? I don’t see the step change.
My intuition: RLVR typically follows the path of least resistance. This often results in it tweaking small behavioral knobs: improving reliability of methods, downweighting unreliable and upweighting reliable methods, encouraging adaptive behaviors like backtracking and discouraging maladaptive behaviors like error self-consistency and self-amplification. Upwards from there is old circuits being used in novel ways: i.e. existing “error detectors” rewired to feed into self-check/backtracking triggers.
But nothing prevents RLVR from burning in new behaviors from scratch. Assuming there is no way to get there without, and sufficient RLVR pressure is applied.
This is in part driven by low KL properties of RLVR setups (both explicit and downstream from RLVR being on-policy), in part driven by how ample are the “low hanging fruits” of base model behavior being prediction-optimal but task-suboptimal, and in part by how “expensive” RLVR pressure is—few setups apply enough of it to get truly novel behaviors burned in.
It’s decent as a null, “better unrolled, compacted, layer-flattened task heuristics rather than better effective depth”, but “last token of input/at the very end” is a tangent at best?
The reason why people wanted recurrence / latent looping for so long was that transformers only have so many layers—and if information has to pass 22 layers of processing for a task to be solved, and you only have 20, your tasks may suffer for it.
Adding more (input, non-autoregressive) tokens doesn’t fix that—a task that needs 22 layers worth of processing still depletes the layer budget, because KV can’t flow backwards or sideways across the depth of the layer stack.
Recurrence / latent looping supposedly resolves some of it—by allowing you to leverage the same layers more than once per token, giving us potentially infinite depth. Autoregressive tokens (including CoT) can be a “pressure relief” for the same, by allowing some information that was emitted at the final layer to reenter at layer 0, but that is a limited bandwidth channel.
It’s been long theorized that this would help some problem classes specifically—ones that benefit from high fidelity representations that evolve in consistent patterns. Such as: serial computation, spatial reasoning and “constraint solver”-like problems. An “existence proof” for that is the “HRM” lineage—models that are fundamentally small recurrent transformers.
So, my go-to test for whether recurrence is well leveraged would be: check if Astra is disproportionately better at the canonical “HRM” problems like solving sudoku—especially without CoT?
Your “sequential computation” is still a good test by that metric, but not because of a left-right pattern, no. As long as the serial dependency of “you have to compute X before you can even start computing Y” holds, it shouldn’t matter if X comes before Y in the input sequence or after.