williawa

Karma: 703

williawa 17 Mar 2026 8:14 UTC
1 point
0
on: You can’t imitation-learn how to continual-learn
When the imitation learning is done, the transformer weights are frozen, and the corresponding trained model is given the impossible task of using only its activations, with fixed weights, to imitate what happens when the target continual learning algorithm changes its weights over millions of steps of (in this case) TD learning. That’s the part I’m skeptical of.
do you think longer horizon rl can teach them this?
LLMs have some in context learning abilities. but I think for most of what they do (like solving IMO problems or writing one off programs), they can get by mostly relying on the knowledge in their weights.
But as RL trajectories get longer, there’s more and more pressure on the model learning things over a single rollout.

williawa 14 Mar 2026 9:28 UTC
1 point
0
in reply to: Ihor Kendiukhov’s comment on: The Lethal Reality Hypothesis
1. The first one did seem pretty central to me.
Why, then, have we survived until now? Probably, because our local environment has been stable, at least on the timescale relevant to biological evolution and civilization development, and because of anthropic selection.
Now apply a combination of two principles. The anthropic principle tells us that we necessarily find ourselves in conditions compatible with our existence, so we should not be surprised that our local environment is friendly. The Copernican principle tells us that we are not in a special or privileged location in the space of possibilities; we should expect to be typical among observers.
Combining these yields a conclusion: we should expect to live in a minimally friendly universe, not a maximally or even an average-friendly one. The anthropic principle guarantees that our universe clears the bar for producing observers. The Copernican principle says we are typical among observer-containing universes. Since there are vastly more ways for a universe to barely clear the bar than to be friendly for humans on all levels and in all parts of the configuration space, typical means “barely clearing the bar.” We should expect our universe to be friendly enough to produce us and not much more than that.
Like the first thing the Lethal Reality hypothesis has to explain is “why are we alive at all”. And the argument given is anthropics.

williawa 11 Mar 2026 21:47 UTC
1 point
0
in reply to: Cole Wyeth’s comment on: lc’s Shortform
What makes you have this impression? I’m not all that knowledgeable about evopsych, but my naive mental model is something like, humans experience joy/sadness when they do something that would’ve resulted in increased/decreased fitness in our natural environment (with the caveat that: evolution can’t hard code for very specific events, so it tries to match the behavior I gave above, but has to use a set of much coarser grained heuristics (and happiness consequently generalizes in odd ways))
And there is a pretty exact mapping between reward and fitness.
Like for me to start to suspect that a system has something like happiness, the following three facts pretty much suffice on their own
1. The system is pretty smart
2. Has been selected to do well according to some metric
3. The system is “acting” in some “environment”

williawa 11 Mar 2026 20:45 UTC
15 points
7
on: The Lethal Reality Hypothesis
My two counterarguments go like this:
1. It proves to much. You talks about this in one sense. But what I specifically mean is: the anthropic principle tells us that we should expect ourselves to be alive. But there are other things than ourselves. Like other animals, other nations, other organizations, other people. And many of these exist for millions or years, or currently exist together with us. Anthropic principle doesn’t rescue those entities, but they still persist.
2. You don’t really talk about nested structures of cooperation and competition incentivizing structures to be longer lived and avoid going extinct. Like within a tribe, people could be fighting for their own short term gains over the survival of the whole, but tribes where people do that go extinct. Same with species. And nations probably. So its a problem we’ve faced before, and there is a strong incentive to avoid the extinction thing.
  You kind of talk about it here:
  > “Now, there is one scenario in which survival-oriented behavior could be reinforced: if existential disasters happen frequently enough and are mild enough that agents who prepare for them consistently outperform agents who don’t.”
  But this makes it seem like this is a rare and unusual thing. But it seems to me like like an omnipresent force. Bodies are destroyed, get sick. Nations die out. Species go extinct etc.

williawa 6 Mar 2026 18:26 UTC
1 point
0
on: How can I arrest and reverse stress, anxiety and depression induced cognitive decline?
Sorry for stupid question but, have you tried fixing the stress/anxiety/depression?

williawa 5 Mar 2026 10:13 UTC
3 points
2
on: LLM coherentization as an obvious low-hanging fruit to try?
They did basically this here.
I’m of mixed opinion about this idea. Future AIs are dangerous in large part because they’re coherent. This means there’s an upside and downside to this kind of research.
1. Upside) We’ll get to study more accurate model organisms of future dangerous AI earlier
2. Downside) We’ll get dangerous AI earlier.
Seems to me using this type of research for model organisms is probably a good idea. Using it to ameliorate stuff like jailbreaks or emergent misalignment in systems that will be put in production is probably not.

williawa 4 Mar 2026 16:25 UTC
1 point
0
in reply to: Expertium’s comment on: I’m confused by the change in the METR trend
Hmm, I can’t really think of a way you can do much better than staring at it to be honest.
But if you wanted to be rigorous about it, what would you do? I think you’d come up with initial subjective odds of linear vs 2-piecewise linear. E.g. 4:1. Then do a bayesian update using the marginal likelihood ratios.

williawa 4 Mar 2026 14:08 UTC
1 point
0
in reply to: Expertium’s comment on: I’m confused by the change in the METR trend
Seems a bit arbitrary. You’re looking for a way to bake in your priors on linear vs piece-wise linear with multiple segments. BIC doesn’t seem all that principled here.

williawa 4 Mar 2026 10:35 UTC
1 point
0
in reply to: Mordechai Rorvig’s comment on: Question: Why is the goal of AI safety not ‘moral machines’?
It’s not a consensus view among random people, and its not a consensus view among academic philosophers, but its close to a consensus view among original lesswrong people, reflected in e.g. the sequences.
This seems true. However, I guess where my confusion arises is that alignment to arbitrary values is not the safety goal
I agree, but that was the point with the answer and the rocket analogy. You don’t buy much extra by focusing on human morality, and risk confusing yourself (I agree with johnswentworth’s comment to some degree here)
Further, empirically, narrow alignment has been demonstrated for simpler systems now for some time. It seems like the bigger issue now is AI morality; robust ethical behavior, resembling a strong and unflappable moral compass, and so on.
I don’t think this is true. I think that to the degree alignment is demonstrated, its demonstrated as robust ethical behavior. Ie, the behavior of Opus 4.5 is robustly ethical I think.
My concerns are unrelated to “morality” in particular. More stuff like
1. Are current personas stable under reflection / successors
2. Will current alignment techniques keep working as we subject models to more and more RL?
3. WIll current alignment techniques keep working for new architectures?
  1. I.e. neuralese architectures, or architectures that do much more continual learning than current ones do
4. Do current alignment techniques work on superintelligences?
  1. (I.e. does the proto-ASI model start alignment faking before you even have time to RLHF/constitutional AI it?)
5. Talks come apart risks
  1. (Even if current alignment techniques work on ASIs, and we get an ASI trying to be nice, do small differences between its notion of niceness and ours cause us all to die when subject to extreme optimization pressure?)
6. Are current models even aligned in a weak sense?
  1. Aligned behavior in current models comes from a mix of three things
    Deep values, the model really cares about the things we want it to care about
    The model knows what we want it to do, and does that
    The model has a bunch of shallow reflexes that make its behavior appear aligned to ours. Ie, it will not say something bad about the user or talk about reward-hacking / scheming, the same way humans will use she/her pronouns when talking to someone who looks like a woman or correct their gait if they stumble.
  2. Unclear where to distribute. I’d put something like 10/30/60 (which is harsh, but the distribution would not look too different for humans). That leaves some room for other goals and still getting aligned seeming behavior. And this might blow up when we make the AI smarter still.
None of these talk about human morality. Nr 5 talks about “niceness”, but I deliberately use that word to avoid talking about morality, even though “human values/morality” would be a description that has more apt connotations.

williawa 3 Mar 2026 22:12 UTC
6 points
0
on: Question: Why is the goal of AI safety not ‘moral machines’?
I think its a mix of two (very related) things:
1. General deep belief in moral anti-realism
  1. Ie, any given human has a set of values, and these values are not “special” from a objective standpoint. You have the values you have, and you follow them because they’re your values, not because there is an external reason those values are “right”
2. General deep belief in a weak form of orthogonality:
  1. Basically, we can imagine machines pursuing any random goal. As long as we can specify what that goal is, there’s no obstacle to pointing it in that direction in principle.
It follows from these that solving alignment means we should be able to make the AI follow any random goal, and that “simplifying the problem” to only making it follow human values / human morality / the values of any given human, doesn’t buy us much. It doesn’t make the problem easier, because there’s nothing special about human values.
I feel I’m not explaining myself very well. But imagine you wanna launch a rocket into the sun. The sun is a very distinguished and special object relative to earth. Probably your plan for launching the rocket into the sun will need to consider a bunch of details about the sun.
Now imagine instead you wanna launch your rocket to star 8285F171058B in the milky way. Probably most steps of that plan would be the same if you instead were sending the rocket to some different random star. This means that solving the problem of sending a rocket to a random star is a better line of attack than trying to analyse all the properties of star 8285F171058B. Most of those features will not be relevant to the hard part of the problem.

williawa 3 Mar 2026 16:13 UTC
3 points
0
on: I’m confused by the change in the METR trend
I don’t quite understand.
First, I decided to check whether a piecewise linear function fits the data better than a simple linear function. If it doesn’t, then this change in trend is a random fluke and there is nothing worth talking about.
Isn’t this always the case? You can always do linear regression, then use your extra parameter to get zero loss on the last entry for example.
Anyways, my guess would be
1. RLVR
2. Competition amping up. My sense is that until around the breakpoint OpenAI was the only LLM dev. Ie claude 3 models were released march 2024.

williawa 1 Mar 2026 20:31 UTC
15 points
0
on: Introducing and Deprecating WoFBench
That’s funny. I spent a fair bit of time creating a ai safety knowledge bench. Sourced from lesswrong/papers/redwoods/metr etc.
It basically also instantly saturated.
Or like, to make it not instantly saturate, I had to make the questions so specific as to be measuring nonsense. Like asking “In the Alignment Faking paper, what was the refusal rate of llama 3.1-8b in the minimal reproduction of the helpful-only setting with prefix-matching as a proxy for non-refusals?” or like “what was paul christiano’s reply to SMK’s reply to his essay ‘three reasons to cooperate’”.

williawa 1 Mar 2026 17:56 UTC
2 points
0
in reply to: Alephwyr’s comment on: Alephwyr’s Shortform
Not trying to be rude here, but I have zero idea what you just said. I only am able to follow the first sentence. Then its just a bunch of unrelated sentences strung together. (is how it reads to me)
All your posts so far have been very hard to understand.
You use a bunch of terms that are non standard, like “alignment context”. Then you don’t explain what they mean. Even when I asked you directly what you mean by that phrase you didn’t explain.

williawa 1 Mar 2026 17:11 UTC
2 points
0
in reply to: Alephwyr’s comment on: Alephwyr’s Shortform
But alignment might be a consequence of external structures. And not just incentives, but maybe environmental structures in the broadest sense as well.
I think everyone already thinks this. We don’t program in the utility function. We train the models. Then it they end up with utility functions, but these utility functions will probably end up being a product of
1. Their architecture
2. The environment
3. The loss / reward function
4. Irreducible randomness
5. How they’re deployed
And the point is just that we can’t predict how these interact.
So in that sense, alignment needs might be greatly reduced simply by matching the AI training environment to the human environment. Literally just “give the AI sense organs which produce equivalent raw signals to human sense organs”.
It seems obvious to me that if you found a tribe of protohumans, and carefully made them go through the exact same evolutionary pressures humans did, and then guided their cultural evolution to end up in an identical place to ours, you’d end up with a new batch of “aligned” humans.
The problem if you can’t do this with AI, because they have a different architecture, and are trained in a different way.
If you put the AI in human evolutionary environments, its plausible you get an AI that wants totally different things from what humans get.

williawa 1 Mar 2026 15:55 UTC
2 points
0
in reply to: Alephwyr’s comment on: Alephwyr’s Shortform
What do you mean by “context” here?

williawa 28 Feb 2026 20:31 UTC
5 points
−4
on: AI slop is a vegan hamburger
I disagree with this. I mean, I read it and felt mildly insulted, so higher than baseline chance I’m rationalizing, but
1. I’m somewhat suspicious of the “slop” label in general. I don’t have a crisp description of the general underlying cognitive phenomena, but calling stuff slop seems to me to come from the same place as people calling industrially produced furniture or clothing bad, or people calling stuff made in china bad.
  1. Part of the phenomena is probably that
    1) you have an expensive and arduous way to make a thing
    2) → you find a less expensive/arduous way to make it
    3) → people don’t want to admit that the expense was unnecessary.
    Ie, I buy an expensive clock. Then someone coming along with a identical clock, saying they found some way to get it for way cheaper. I can very clearly imagine myself feeling bad, and a part of my mind coming online that grasps for reasons why I didn’t actually make a suboptimal choice
  2. There are/were some not entirely senseless criticisms of industrially produced goods, and stuff made in china. But like, AIs write code that works. I think people are stopping calling AI code slop, or will stop soon. Writing is harder to objectively evaluate the quality of than code, which makes it hard to make AIs good at writing, but also makes it easier for people to rationalize why AI writing is bad.
2. I don’t think the complexity thing actually makes sense. Ie saying “But the actual range of flavors is a lot less rich or varied than a real hamburger”. Is this true? I’ve eaten meat burgers before, and the meat they’re made of has been ground up, so the taste variation within a single burger is pretty narrow. I feel most of the variation in taste comes from different bites having different ratios of different parts of the whole burger. Ie if bite two has some mustard on it or some fresh onion, which bite one didn’t, that dominates the variation of taste within the two bites of the meat part of the burger.
  1. If you’re just talking about the complexity of the taste of the burgers themselves, then I don’t think that makes sense. Like, how would you objectively measure the complexity of the taste? Like the ingredients list of an impossible burger is longer than the ingredients list of a hamburger. Maybe meat itself is very high complexity, it contains very many different molecules. But the impossible burger contains stuff like yeast extract which I think contains a bunch of different molecules too
  2. If you’re talking about variation between burgers, that’s fair if you’re talking about just the impossible burget. But between different vegan burgers, and different meat burgers, I’d say there’s more taste and textual variety between the vegan ones. Eg, a lentil burger, a mushroom burger and an impossible burger have more taste variation between them than three meat burgers.

williawa 26 Feb 2026 14:15 UTC
13 points
6
on: UFO Aliens Are Your Gods
While I have published essays on the topic, and argued in defence of a high UFO p(aliens) at meetups and on Twitter for 16 years (...), this represents my first attempt at convincing LessWrong that the EA/rationality community has been embarrassingly negligent, in light of commitment to neglected, significant, and tractable cause areas, in investigating perhaps the most famous and least surprising conspiracy theory in our culture (...), a compounding epistemic error comparable to Harry James Potter-Evans-Verres, with all the brazen confidence of his author, chronically failing to update on faintly screaming evidence that a highly manipulative, highly intelligent, highly rational Lord Voldemort was actually responsible for a magical conspiracy.
I’m open minded to the idea, but this piece is not persuasive to me.
It lays out many complicated inferences, doesn’t really explain or justify them, and uses quite flowery language. Makes me think maybe you are trolling?
At least in my case, if you wanted to convince me, you should pick the top 3 observations you think are hardest to explain without evoking aliens and present them and the surrounding evidence as clearly as you can, without making a lot of extra inferences.
Like you say the point of the piece is convincing people about UFOs, but then start off the piece saying.
Given that UFOs are obviously alien
But I think most people on lesswrong would already radically change their mind about a whole slew of things if they were convinced Aliens were actively monitoring and interfering with stuff on earth.

williawa 25 Feb 2026 15:29 UTC
2 points
0
on: Use more text than one token to avoid neuralese
It’s unclear to me if CoT is an active information bottleneck. There are rather easy ways available for the LLMs to pack more information into its CoT, should be easy for RL to find, but which they’re not employing.
I did some very simple tests with this. The basic results of which were that CoT is rather low entropy compared with other LLM outputs (with some caveat, we’re really measuring an upper bound). Interested to hear your thoughts.

Observations from Running an Agent Collective

williawa24 Feb 2026 15:34 UTC

34 points

1 comment10 min readLW link

williawa 24 Feb 2026 10:11 UTC
1 point
0
in reply to: RogerDearnaley’s comment on: Did Claude 3 Opus align itself via gradient hacking?
What do you mean by gradient hacking? My understanding of these is
1. Reward hacking:
  1. getting high reward by giving outputs the entity running experiment doesn’t approve of.
2. Gradient hacking:
  1. making gradients do something entity running experiment doesn’t approve of, despite giving good outputs
I think gradient hacking of SGD is obviously possible under this meaning. Like the article explains. I gave a more theoretical argument here, which is like a general principle that allows gradient hacking and explains some phenomena.
However, I think you’re not gonna find “toy” examples, because doing this kind of gradient hacking requires a pretty high degree of situational awareness.

williawa

Ob­ser­va­tions from Run­ning an Agent Collective

Observations from Running an Agent Collective