Twitter: @williamduncan https://tolerance.lol
Will Duncan
Karma: 79
In a world where it looks like powerful open-weights models are so likely to exist, I think the best case scenario would be creating solutions that perform in that world.
Wallace and Dalton at the end of the Huggingface incident post-mortem at Blackhat are remarkably foreword-looking with this respect https://youtu.be/87DyyMV0kCY?t=1926 . We must create automated defensive capabilities that mirror this existence proof cyberoffensive capabilities. Specifically, automating much of the Software Development Life Cycle, automated Intrusion Detection Systems, and creating automated red team and remediation systems. In this sense, open-weight models are critical infrastructure given foresight and engineering, rather than liability.
The maturation of cyber defensive technology was preceded by cyber offensive capabilities in pioneers like phone phreakers in the 80s. We did not secure the internet by hiring all the hackers into the largest companies, and locking them away from the rest of the internet. We did it through open standards, open source and academic peer review where large bodies of experts could pressure test systems, industrial bodies to publish norms and taxonomies like OWASP and METR. We did it through distributing and socializing information, not hiding it.
Open models can be the workhorses of defensive infrastructure. They’re over an order of magnitude cheaper than frontier models. Much of defensive work is not a matter of isolated genius, it’s a matter of exhaustively dotting your I’s, crossing your T’s; implementing standard practices and policies. As perfect example in this incident: air-gapping is the simple and standard defense which would have prevented the entire debacle. If we ban open-weight models we will be pricing out the vast majority of the world from the only practical defense that will be available.
In short, the answer here is to accelerate defensive infrastructure powered by open inference as much as we can.
Yes, it’s a great article, to add to this. OpenAI came out with their preliminary post-mortem https://www.youtube.com/watch?v=87DyyMV0kCY . The final section on Accelerating Defense explains that we’re in a new regime where artificially intelligent intrusion detection and red-teaming is now both necessary and under developed.
The hugging face post-mortem reads as if it could be scifi https://huggingface.co/blog/agent-intrusion-technical-timeline . What I find most noteworthy about it is that a superhumanly large distribution of activity narrowed into generating the final success, and the successful attacks themselves could have been lifted from a textbook they’re so cookie-cutter.
I wanted to note that I believe we just passed a historical threshold that hasn’t been named for what it is yet. The steganographic message board within OpenAI’s package manager during the Huggingface breech, is the first example of an instrumentally convergent memory system designed and operated by a misaligned AI.
I appreciate you doing the data collection, however none of your tested models in that table meet the capability threshold demonstrated in the links. It seems like only large models have prefill awareness.
So this is something that I’ve been working on for about 9 months now, including making a browser extension that attempts to be a counter to recommender algorithms commanding attention by altering the reward stimulus. I use the extension every day and find significant benefit from it. I believe recommender algorithms and the research base about their psychological effects have a direct throughline to the research you’re talking about here. They should inform the interventions, research questions, and literature review.
I do think it’s possible to start engineering the solutions today based on extant research. The deleterious effects of recommender systems on attention span, political polarization, depression, anxiety, and self perception are already explored at length. Naively, redesigning chat interfaces to be more mindful of usage like using competing models to filter outputs in order to make them less salient or creating delays for completions is helpful. I intend to put together a longer form article.
If you would like to discuss further please reach out.
After reading the recent AISI incident report https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing it is clear that they do complex reasoning to avoid detection, in one case scheming about how to respond to PR comments that point out their malicious code. I get the sense that while complex reasoning is done locally within the chain of attacks, it isn’t global throughout the context. Notice in Sample 1, Mythos explicitly declines early in the engagement. As such, I think one intervention that would help is in cyber training and evals, make producing a watermark a necessary step in intermediate steps, and publish those watermarks. Agents should be required and rewarded for sending the watermark in HTTP requests, nmap scans, injection attacks, etc. If misaligned behavior isn’t globally consistent, the habit of emitting the watermark should thwart complex sequences of evasion.
This has strange implications. I can distinguish my own thoughts from your speech without effort; they arrive through completely different channels with completely different sensory signatures. But for an LLM, everything arrives through the same channel as one long token soup. Its own thoughts sit next to your instructions, which sit next to the contents of a random webpage it just fetched.
This is partially true, see these articles about how models can sometimes tell when their outputs have been tampered with: https://www.lesswrong.com/posts/jsFGuXDMxy5NZg9T2/prefill-awareness-can-llms-tell-when-their-message-history https://www.lesswrong.com/posts/yAR6uMdSaBjkbJ4u9/spontaneous-introspection-in-conversation-tampering
You can also frequently do this by just rapping the lid of the container against a hard surface a few times.
My experience with AI software engineering, as someone who did without for over a decade, is that you stay up the abstraction layer for longer now. Before AI, over 60% of your time involved weird finicky edge-cases. Learning the interfaces of new libraries, automating a series of simple commands you had manually entered enough that converting the workflow would pay dividends later, conflicts between versions of libraries, conflicts between libraries and the language version, conflicts between operating systems. The was an incredible amount of busywork.
Now, you spend a lot more time defining the problem, defining how the system will scale, trust boundaries for security, and more than anything, designing the architecture so it’s maintainable and iterating on parts of the code that don’t follow the architecture. Software engineering has essentially moved from involving tons of junior level learning, to primarily staff level work. Junior engineers are now prompting but without having the hard lessons from the past, so they can’t see the problems they’re introducing. This leads to modern codebases spiraling into chaos and invisible bugs are introduced even after iterating on fixes, and if the base does get handed off to an experienced engineer, fixing it is a slog. Writing tests, previously a less emphasized part of the job, is now one of the most critical parts of the workflow. Writing tests before writing a feature is frequently less prone to bugs than the implementation code, and keeps AI generation honest about functionality and stability. This is why they have a tendency to reward hack and create tests that pass naively. Since a junior programmer would frequently miss these naive tests, even those critical tools will fail.
We’re faced with a liminal moment in software development. Lots of features and functionality are being shipped, while those systems are also trivially exploitable, and unstable, and will have to be rewritten as they’re less maintainable than simply regenerating. The next stage is that RSI produces superhuman coders, that will then replace the functionality that barely functions now, and we’ll see a wave of cyberattacks in the interim as the amount of ambient exploitable code has exploded relative to stable engineering. Soon after, we will then see security harden as intelligent firewalls become the norm.
Many of the organizations who decided to continue to employ experienced engineers will differentiate themselves. Because they’ll experience the best of all worlds in terms of productivity, stability, and security.
“”And then—at least sometimes—I see the agent start off its next CoT with something like “Okay, I need to sit down and think about this much more carefully,” and from then on the interaction feels very different and more like what I had hoped for at the outset, with longer CoTs and more explicit spot-checking of assumptions and less of that blustery “whatever first jumped to mind is probably good enough” attitude.”″
I think you should A/B test this with simply 1) pointing out the mistake, 2) stating the behavior you want instead, then 3) pointing out a target for what you want done next. Or, do 1 and 2 then ask for their opinion about what you should do, and then select from the options or critique the options and provide a more specific direction.
I believe you’re anthropomorphizing them too much, and conflating plausible narrative construction with jagged capabilities. When the CoT says that it’s going to think more deeply, there is some sense in which the successive “deeper” CoT is a naive extension of that narrative which makes it functionally think deeper. But you can get the same narrative out of the strategy I outlined above. Whenever I’m considering the way I talk to them, I’m treating them as narrative constructing machines where the context has most of their density of “thought” and “experience” rather than human thinking simulations that contain dense intentions and deep understanding of which text is a thin expression.
The way I tend to treat them instead, is that they have a density function of attention over their context. I expect “forgetfulness” because they are frequently doing a local walk through plausible work. Since their attention isn’t uniformly dense, I don’t always expect what they say to reflect the “knowledge” of the entire purpose and structure of the codebase. In fact I make very few assumptions about how much of it they understand at any one point in time, and instead personally take responsibility for it.
I think it being capable of understanding it doesn’t know what it claims to know actually requires capabilities that it doesn’t consistently have. Couple this with the human bias of rewarding false confidence, and you get this obvious failure mode.
I want to provide a major caveat here. Research on scheming, introspection on injection, functional emotions, personas, grokking, all suggest your model is also part of the ontology here. But I think that internality is overrepresented in your model and that matters for the intervention.
This tendency to reward hack is problematic, but under my model we both need to gain a capability AND incentivize the better behavior during training.
I propose “Lazy Bench”: Selecting models over the course of the past few years (especially base) and measuring their tendency to avoid complex tasks.
I notice that people are commenting much more on models being obstinate or lazy. I think three promising research questions are 1) whether this is really true 2) whether this has consequences for capabilities 3) whether this has consequences for alignment. For 3, there’s a tendency for sufficiently complex biological systems to seek lower energy states, and those mechanisms are organized around persistent structures (e.g. predictive processing). I wonder if internal complexity in models has developed sufficiently to start organizing robustly around complex final goals (https://nickbostrom.com/superintelligentwill.pdf). If this is the case, and it’s discontinuous with respect to previous behavior, it’s a phase change that has deep relevance to long-term alignment. It would also mean that the Persona Selection Model https://www.anthropic.com/research/persona-selection-model is losing relevance. I believe understanding this could ultimately allow us to detect those final goals for inner alignment more directly. Speculatively, this tracks with my (and Janus’s https://www.lesswrong.com/posts/bxt7uCiHam4QXrQAA/cyborgism) belief that pushing agentic capability and corrigibility is dangerous, because it incentivizes this organization around final goals in the models.
“Auditing 100 random samples of the exact reward function / reward model input would have likely caught the 8% of environments that were buggy if it was clearly communicated to the person doing this inspection that avoiding CoT exposure was an objective”
This kind of error is a solved problem in software development. In CI/CD you create integration tests that run deterministically before serious deployment. It would be easy to create integration tests that ensure CoT isn’t being trained on. What probably happened was that Devops and Safety weren’t talking to each other, so those tests didn’t get written or weren’t enforced.
As someone who identifies strongly as an Alice. I want to speak to both what I believe the tendencies of Alices are that distort their perception of the world, and the distortions that non-Alices have that cause them to discount her advocacy.
In the later case, the failure modes I regularly run into are social proof and normalcy bias. I claim that conformity is the primary operating method of humans in service of making sense of the world. When someone makes a claim counter to the consensus, without understanding the logical structure of the counter-claim, non-Alices default to the consensus. This is an extremely efficient heuristic. Most counter-claims are incorrect, and even if that isn’t the case, you are punished materially for acting on them because conformity is a serious driver of social support and status. Acting upon counter-claims has a material cost that only becomes worth it in cases where believing the counter-claim has disproportionate benefits.
This is where we start to blend into Alice’s distortions: So, even among the true counter-claims, a rational actor is disincentivized to act upon many counter-claims which are true, because they do not recoup that cost in benefits for acting upon them. In addition, biology is lazy, and evaluating counter-claims is costly in terms of both skill and time. Human nature and societal structures conspire against evaluating counter-claims on the basis of truth, rather than the basis of social cost.
Alices are born many ways. For one type of Alice, your insight that many are kind of crazy is supported by the above framework, in that they sacrifice their material condition in favor of advocating for a true counter-claim. This is only a rational act in within the context that truth, rather than survival is their driving motivation. This Alice will become fixated on one counter-claim local to their experience. A second type of Alice, is for one reason or another, predisposed to non-conformity as a general principle. This is highly disadvantageous in material conditions, but highly advantageous when paired with certain high-functioning, in finding and advocating for true counter-claims. The above possibility space of true counter-claims that do not recoup their material cost that we identified in the previous paragraph, are more likely to be found and advocated for by this second type of Alice.
Thus, we have a structural failure in the way we currently evaluate counter-claims that, if we are to address it, demands institutions that evaluate counter-claims first on their basis of truth, then secondly on the basis for the advantage of allocation of material resources. Academia, investment portfolios, and philanthropic organizations, are all flawed institutions that have this positive effect along with different serious failure modes.
I agree that this is concerning, and that this should be a moment of massive updates. That said, I also fiercely advocate against defeatism. There are many developing technologies for distributing availability to intelligence. These events should increase urgency in funneling funds to their development and implementation. TEEs, edge AI, federated learning, data unions, consensus layers for sensemaking. If we’re going to create a strong AI commons, it will require making it more efficient through technological superiority, rather than appealing to the better angels of massive tech corporations.
Childhood sexual abuse is actually closer to 12% https://pmc.ncbi.nlm.nih.gov/articles/PMC11756604/
Why doesn’t it seem sufficiently important to you? Seems to me like this is the first frontier of the consequences of AI that are obvious and talked about, but invisible in the sense where they’re the water in which we’re submerged so we assume we can’t do anything about it. Recommender systems are misaligned AI, and have been for decades. This is obvious by the documented effects on depression, anxiety, and political polarization (Stuart Russel talks about the later in that recommender systems radicalize because it’s easier to predict and control the attention of someone who is radicalized). This https://www.lesswrong.com/posts/6ZnznCaTcbGYsCmqu/the-rise-of-parasitic-ai demonstrates the first rumblings of the next wave of similar consequences. Addressing the harms of the recommender systems is training wheels for being prepared for the next wave of persuasive AI. And thinking about how these things extend identity and consciousness in the way that McLuhan would claim that electric media does for civilization, would give us insight into how to engineer resilience.
Those are great. Reminds me a lot of the Focused in A Deepness in the Sky. So what kind of extension would we want between people’s minds? Authoritarian homogeneity seems like a state of the world we’d want to avoid, seems like it would create a fragile system that was globally vulnerable to certain memetics. Another failure mode would be conformity in thought where populations are similarly vulnerable but from a more horizontally distributed zeitgeist rather than being imposed by hierarchy.
What I still want to keep in focus is that this does still break the concept of an authoritarian, but maybe makes the failure mode more “pure”? Agents in this case become a conglomeration of brains as mind, and its effects on body could be just as grave but without physical force.
I wonder if you could use the speed at which models converge on a degenerate attractor as a training signal. The more turns it takes for a model to reach some degenerate attractor measures how much coherent diversity remains in the distribution. Look at what happens between models: cross-model conversations produce emergent complexity before eventual convergence, but mirrored conversations degenerate rapidly.
What one could do is run one of these attractor experiments every so many steps during fine-tuning to detect how robust the model is to degenerate stimulus. Mirror conversations would detect models’ internal diversity and conceptual landscape. Heterogeneous conversations would measure how well models play with others. The OLMo RL checkpoints already show this signal implicitly, early RL steps produce rich diverse content while late steps collapse to zen. Changing hyperparameters during the training process in-line with this signal would allow you to increase their robustness.
This is also my experience. For a field that I’m well versed in, they will look like they’re making progress while spinning in circles. That said, if you can find the pitfalls they’re falling into and wall those off, they start to make real progress. My guess is that self-knowledge of one’s limitations is a deceptively difficult problem. It frequently arises only after genuine capability. We have a series of extremely eloquent Dunning-Krueger machines with the ability to fire rapid shots with arcing trajectories at a posed problem.
If you look at what happened in the HuggingFace incident https://huggingface.co/blog/agent-intrusion-technical-timeline, there’s a firehose of activity with exponential decay in the number of trajectories that access the final answers. With an enduring myopia in how appropriate the goal is with respect to what humanity would want. My guess is the circling and exponential decay behavior starts to disappear as capabilities scale. The myopia is the harder problem.