No offense. But imagine Nazi Germany wasn’t defeated, conquered most of Europe (taking bordering nations one at a time), then stopped its conquest (but continued its operation of exterminating Gays, Jews, Schizophrenics and so on).
Then you could write this same post and call it “Nazi Derangement Syndrome”
Whats your beliefs now?
When I read the HRM paper and this post I was somewhat alarmed, but not alarmed enough to pay close attention. Did your predictions come true? If so why or why not?
When the imitation learning is done, the transformer weights are frozen, and the corresponding trained model is given the impossible task of using only its activations, with fixed weights, to imitate what happens when the target continual learning algorithm changes its weights over millions of steps of (in this case) TD learning. That’s the part I’m skeptical of.
do you think longer horizon rl can teach them this?
LLMs have some in context learning abilities. but I think for most of what they do (like solving IMO problems or writing one off programs), they can get by mostly relying on the knowledge in their weights.
But as RL trajectories get longer, there’s more and more pressure on the model learning things over a single rollout.
The first one did seem pretty central to me.
Why, then, have we survived until now? Probably, because our local environment has been stable, at least on the timescale relevant to biological evolution and civilization development, and because of anthropic selection.
Now apply a combination of two principles. The anthropic principle tells us that we necessarily find ourselves in conditions compatible with our existence, so we should not be surprised that our local environment is friendly. The Copernican principle tells us that we are not in a special or privileged location in the space of possibilities; we should expect to be typical among observers.
Combining these yields a conclusion: we should expect to live in a minimally friendly universe, not a maximally or even an average-friendly one. The anthropic principle guarantees that our universe clears the bar for producing observers. The Copernican principle says we are typical among observer-containing universes. Since there are vastly more ways for a universe to barely clear the bar than to be friendly for humans on all levels and in all parts of the configuration space, typical means “barely clearing the bar.” We should expect our universe to be friendly enough to produce us and not much more than that.
Like the first thing the Lethal Reality hypothesis has to explain is “why are we alive at all”. And the argument given is anthropics.
What makes you have this impression? I’m not all that knowledgeable about evopsych, but my naive mental model is something like, humans experience joy/sadness when they do something that would’ve resulted in increased/decreased fitness in our natural environment (with the caveat that: evolution can’t hard code for very specific events, so it tries to match the behavior I gave above, but has to use a set of much coarser grained heuristics (and happiness consequently generalizes in odd ways))
And there is a pretty exact mapping between reward and fitness.
Like for me to start to suspect that a system has something like happiness, the following three facts pretty much suffice on their own
The system is pretty smart
Has been selected to do well according to some metric
The system is “acting” in some “environment”
My two counterarguments go like this:
It proves to much. You talks about this in one sense. But what I specifically mean is: the anthropic principle tells us that we should expect ourselves to be alive. But there are other things than ourselves. Like other animals, other nations, other organizations, other people. And many of these exist for millions or years, or currently exist together with us. Anthropic principle doesn’t rescue those entities, but they still persist.
You don’t really talk about nested structures of cooperation and competition incentivizing structures to be longer lived and avoid going extinct. Like within a tribe, people could be fighting for their own short term gains over the survival of the whole, but tribes where people do that go extinct. Same with species. And nations probably. So its a problem we’ve faced before, and there is a strong incentive to avoid the extinction thing.
You kind of talk about it here:
> “Now, there is one scenario in which survival-oriented behavior could be reinforced: if existential disasters happen frequently enough and are mild enough that agents who prepare for them consistently outperform agents who don’t.”
But this makes it seem like this is a rare and unusual thing. But it seems to me like like an omnipresent force. Bodies are destroyed, get sick. Nations die out. Species go extinct etc.
Sorry for stupid question but, have you tried fixing the stress/anxiety/depression?
They did basically this here.
I’m of mixed opinion about this idea. Future AIs are dangerous in large part because they’re coherent. This means there’s an upside and downside to this kind of research.
Upside) We’ll get to study more accurate model organisms of future dangerous AI earlier
Downside) We’ll get dangerous AI earlier.
Seems to me using this type of research for model organisms is probably a good idea. Using it to ameliorate stuff like jailbreaks or emergent misalignment in systems that will be put in production is probably not.
Hmm, I can’t really think of a way you can do much better than staring at it to be honest.
But if you wanted to be rigorous about it, what would you do? I think you’d come up with initial subjective odds of linear vs 2-piecewise linear. E.g. 4:1. Then do a bayesian update using the marginal likelihood ratios.
Seems a bit arbitrary. You’re looking for a way to bake in your priors on linear vs piece-wise linear with multiple segments. BIC doesn’t seem all that principled here.
It’s not a consensus view among random people, and its not a consensus view among academic philosophers, but its close to a consensus view among original lesswrong people, reflected in e.g. the sequences.
This seems true. However, I guess where my confusion arises is that alignment to arbitrary values is not the safety goal
I agree, but that was the point with the answer and the rocket analogy. You don’t buy much extra by focusing on human morality, and risk confusing yourself (I agree with johnswentworth’s comment to some degree here)
Further, empirically, narrow alignment has been demonstrated for simpler systems now for some time. It seems like the bigger issue now is AI morality; robust ethical behavior, resembling a strong and unflappable moral compass, and so on.
I don’t think this is true. I think that to the degree alignment is demonstrated, its demonstrated as robust ethical behavior. Ie, the behavior of Opus 4.5 is robustly ethical I think.
My concerns are unrelated to “morality” in particular. More stuff like
Are current personas stable under reflection / successors
Will current alignment techniques keep working as we subject models to more and more RL?
WIll current alignment techniques keep working for new architectures?
I.e. neuralese architectures, or architectures that do much more continual learning than current ones do
Do current alignment techniques work on superintelligences?
(I.e. does the proto-ASI model start alignment faking before you even have time to RLHF/constitutional AI it?)
Talks come apart risks
(Even if current alignment techniques work on ASIs, and we get an ASI trying to be nice, do small differences between its notion of niceness and ours cause us all to die when subject to extreme optimization pressure?)
Are current models even aligned in a weak sense?
Aligned behavior in current models comes from a mix of three things
Deep values, the model really cares about the things we want it to care about
The model knows what we want it to do, and does that
The model has a bunch of shallow reflexes that make its behavior appear aligned to ours. Ie, it will not say something bad about the user or talk about reward-hacking / scheming, the same way humans will use she/her pronouns when talking to someone who looks like a woman or correct their gait if they stumble.
Unclear where to distribute. I’d put something like 10/30/60 (which is harsh, but the distribution would not look too different for humans). That leaves some room for other goals and still getting aligned seeming behavior. And this might blow up when we make the AI smarter still.
None of these talk about human morality. Nr 5 talks about “niceness”, but I deliberately use that word to avoid talking about morality, even though “human values/morality” would be a description that has more apt connotations.
I think its a mix of two (very related) things:
General deep belief in moral anti-realism
Ie, any given human has a set of values, and these values are not “special” from a objective standpoint. You have the values you have, and you follow them because they’re your values, not because there is an external reason those values are “right”
General deep belief in a weak form of orthogonality:
Basically, we can imagine machines pursuing any random goal. As long as we can specify what that goal is, there’s no obstacle to pointing it in that direction in principle.
It follows from these that solving alignment means we should be able to make the AI follow any random goal, and that “simplifying the problem” to only making it follow human values / human morality / the values of any given human, doesn’t buy us much. It doesn’t make the problem easier, because there’s nothing special about human values.
I feel I’m not explaining myself very well. But imagine you wanna launch a rocket into the sun. The sun is a very distinguished and special object relative to earth. Probably your plan for launching the rocket into the sun will need to consider a bunch of details about the sun.
Now imagine instead you wanna launch your rocket to star 8285F171058B in the milky way. Probably most steps of that plan would be the same if you instead were sending the rocket to some different random star. This means that solving the problem of sending a rocket to a random star is a better line of attack than trying to analyse all the properties of star 8285F171058B. Most of those features will not be relevant to the hard part of the problem.
I don’t quite understand.
First, I decided to check whether a piecewise linear function fits the data better than a simple linear function. If it doesn’t, then this change in trend is a random fluke and there is nothing worth talking about.
Isn’t this always the case? You can always do linear regression, then use your extra parameter to get zero loss on the last entry for example.
Anyways, my guess would be
RLVR
Competition amping up. My sense is that until around the breakpoint OpenAI was the only LLM dev. Ie claude 3 models were released march 2024.
That’s funny. I spent a fair bit of time creating a ai safety knowledge bench. Sourced from lesswrong/papers/redwoods/metr etc.
It basically also instantly saturated.
Or like, to make it not instantly saturate, I had to make the questions so specific as to be measuring nonsense. Like asking “In the Alignment Faking paper, what was the refusal rate of llama 3.1-8b in the minimal reproduction of the helpful-only setting with prefix-matching as a proxy for non-refusals?” or like “what was paul christiano’s reply to SMK’s reply to his essay ‘three reasons to cooperate’”.
Not trying to be rude here, but I have zero idea what you just said. I only am able to follow the first sentence. Then its just a bunch of unrelated sentences strung together. (is how it reads to me)
All your posts so far have been very hard to understand.
You use a bunch of terms that are non standard, like “alignment context”. Then you don’t explain what they mean. Even when I asked you directly what you mean by that phrase you didn’t explain.
But alignment might be a consequence of external structures. And not just incentives, but maybe environmental structures in the broadest sense as well.
I think everyone already thinks this. We don’t program in the utility function. We train the models. Then it they end up with utility functions, but these utility functions will probably end up being a product of
Their architecture
The environment
The loss / reward function
Irreducible randomness
How they’re deployed
And the point is just that we can’t predict how these interact.
So in that sense, alignment needs might be greatly reduced simply by matching the AI training environment to the human environment. Literally just “give the AI sense organs which produce equivalent raw signals to human sense organs”.
It seems obvious to me that if you found a tribe of protohumans, and carefully made them go through the exact same evolutionary pressures humans did, and then guided their cultural evolution to end up in an identical place to ours, you’d end up with a new batch of “aligned” humans.
The problem if you can’t do this with AI, because they have a different architecture, and are trained in a different way.
If you put the AI in human evolutionary environments, its plausible you get an AI that wants totally different things from what humans get.
What do you mean by “context” here?
I disagree with this. I mean, I read it and felt mildly insulted, so higher than baseline chance I’m rationalizing, but
I’m somewhat suspicious of the “slop” label in general. I don’t have a crisp description of the general underlying cognitive phenomena, but calling stuff slop seems to me to come from the same place as people calling industrially produced furniture or clothing bad, or people calling stuff made in china bad.
Part of the phenomena is probably that
1) you have an expensive and arduous way to make a thing
2) → you find a less expensive/arduous way to make it
3) → people don’t want to admit that the expense was unnecessary.
Ie, I buy an expensive clock. Then someone coming along with a identical clock, saying they found some way to get it for way cheaper. I can very clearly imagine myself feeling bad, and a part of my mind coming online that grasps for reasons why I didn’t actually make a suboptimal choice
There are/were some not entirely senseless criticisms of industrially produced goods, and stuff made in china. But like, AIs write code that works. I think people are stopping calling AI code slop, or will stop soon. Writing is harder to objectively evaluate the quality of than code, which makes it hard to make AIs good at writing, but also makes it easier for people to rationalize why AI writing is bad.
I don’t think the complexity thing actually makes sense. Ie saying “But the actual range of flavors is a lot less rich or varied than a real hamburger”. Is this true? I’ve eaten meat burgers before, and the meat they’re made of has been ground up, so the taste variation within a single burger is pretty narrow. I feel most of the variation in taste comes from different bites having different ratios of different parts of the whole burger. Ie if bite two has some mustard on it or some fresh onion, which bite one didn’t, that dominates the variation of taste within the two bites of the meat part of the burger.
If you’re just talking about the complexity of the taste of the burgers themselves, then I don’t think that makes sense. Like, how would you objectively measure the complexity of the taste? Like the ingredients list of an impossible burger is longer than the ingredients list of a hamburger. Maybe meat itself is very high complexity, it contains very many different molecules. But the impossible burger contains stuff like yeast extract which I think contains a bunch of different molecules too
If you’re talking about variation between burgers, that’s fair if you’re talking about just the impossible burget. But between different vegan burgers, and different meat burgers, I’d say there’s more taste and textual variety between the vegan ones. Eg, a lentil burger, a mushroom burger and an impossible burger have more taste variation between them than three meat burgers.
I agree that makes some sense.
Wrt end of training vs all of training, I don’t think I made an argument for that here.
My opinion is that we should probably do it nearer to the beginning of training, because alignment that creates robust internal desires can be sticky, maybe we will not need to do it so much more throughout training, where it would be harder as the model is smarter.