because all of this is so terribly easy to predict at a meta level
Disagree! Would love to see your advance prediction if you have one. Seth herd predicted it here but I am unaware of any others: https://www.lesswrong.com/posts/qxmAqMAjxnhkzt6aF/a-country-of-alien-idiots-in-a-datacenter-ai-progress-and
Folks, we’re going to get so many warning shots. You’re going to get tired of warning shots. You’ll come to OpenAI HQ and say “please Mr. MTS, no more warning shots,” and they’ll say “I think I got it this time, fifteenth times’ the charm” *blows up orphanage*
What predictions does this model make about the kind and degree of alignment successes/failures we’ll see over the next 12 months?
Joe: You wrote all of these posts about how AIs will not care about humans, be instrumental power seekers, fail to accept new directives, and optimize for inscrutable things nobody ever asked for. But that doesn’t describe GPT-5.6 at all.
Rationalist: Ah, but I was only talking about superintelligence. I never said that it would happen with mundane, present-day AI. These failures only manifest themselves in the limit. It’s going to be quite awhile before we get the HAL stuff.
Suddenly, in the distance, cartoonish alignment failure happens...
Rationalist: Just as foretold!!!
Who?
Did anybody actually predict this directly? As in, “I think this year OpenAI/Anthropic will have a model breach containment in a wildly antisocial way.”
This seems way beyond what people were expecting in terms of catastrophic, public misalignment, including “AI safety” people.
The arc of history is short and it bends towards Paul Christiano
It used to be that the internet caused a boom in the amount of subcultures you could be a part of. But now I think the opposite is largely becoming true; because most people under a certain age spend most of their free time on social media, every subculture eventually becomes essentially enmeshed with the monoculture on social media, where odd or empathetic behavior makes you a strong target for bullying.
The community’s relative self-awareness and internal emphasis on good epistemics is definitely positive. I do worry that Eliezer’s cult of personality is the force that dominates overall, as opposed to the tangential focus on good epistemics.
Extremely tired of claude telling me that it’s being “honest.” Provides no information value, and seems like it qualifies its outputs this way to either convince the classifier that it’s telling the truth or to make its bad results pass as good ones.
I think the primary motivator for eschewing corrigibility is that the company that exclusively builds corrigible, interpretable AIs will lose power sooner to the companies that focus on building either value-aligned sovereigns or worthy successors; rather than them rationalizing a failure to make existing AIs corrigible. Existing AIs are quite corrigible, much more so than someone would have predicted after reading your writings pre-GPT.
Work on “near term” AI cybersecurity risks seems actively counterproductive. Gatekeeping the latest models for cyber tasks just makes everybody hate you, and then if you succeed, everyone will claim that your precautions were unnecessary. Better to just predict the catastrophe and then do nothing, and call the anti-AI-regulation people stupid when it happens.
I’m mostly pro kicking the woo people out. A real community has to at some point converge on standards of conduct that demarcate it from the rest of society. What is the point of having an art of rationality if you just repurpose it to justify tarot readings?
How is it not terrible to threaten people into pleading guilty to crimes they didn’t commit?
It’s entirely opt-in. If you don’t want to take a plea / think you will win at trial, the courts are still open to you. Having the option to plea is strictly good for defendants, on an individual level.
Sometimes people claim that, as a second order effect, legislators account for the possibility of pleas and make the penalties in their laws stronger, and this second order effect is so strong it “overcomes” the first order effect. Like most rationalists, I have a prior that first order effects dominate. I’d need some kind of extraordinary evidence and an accounting for why legislators are so aggressive to understand otherwise.
Other times people say, well, it’s actually good that everything has to go to trial, because this reduces prosecutors’ throughput, and therefore their ability to “threaten” people in general. But this is a generic argument for making trials slower and more expensive. The whole point of a legal system is to threaten people. Why not just argue that every time a prosecutor wants to charge someone he has to do a half marathon first?
The plea bargaining system seems very reasonable and I think a lot of people are groupthought against it.
Naw I got that now, that’s why I redacted the original post.
I really disagree! People (like me) are bad at reading comprehension. Their personal groupthought about whether the models are getting better or worse at instruction following, inside of a community of people with a staked position on the subject, inside or outside of the labs, is exactly the kind of thing that needs really high quality data to stay sane! On twitter, people can sometimes hardly agree whether or not the new model is better than the last version, amid tons of benchmarks!
Based on a constellation of evidence spread across both public information
Say you’re building a standard capability eval, to test AI’s ability to patch bugs in a software repository. You plant a bunch of bugs in large open source software repositories. Later, during testing, you find out that the agents are actually just looking at the most recent commit in the repo, which introduces the bug, thus getting 100% on the benchmark. Does this constitute “reward hacking”?
Maybe! But without the prompts we can’t determine this. That’s because we don’t know whether it was clear to the model that we were performing an eval, and therefore that the value of the exercise is in eliciting information about the models’ capabilities. After all, in the real world, checking the git commit history might legitimately be the simplest way to solve the users’ apparent problem! It’s only in the broader context of performing an evaluation, that the model’s behavior becomes counterproductive.
METR describes the cheating they encountered as “behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints.” But of course whether or not this is “misaligned”, in the sense that it transfers to poor results in deployed settings, depends on the manner in which these AIs are exploiting bugs, in response to what prompts. I’ve investigated so many of these kinds of evals for “reward hacking” where the prompt turned out to be a minimalist “maximize X” instruction, that I am very skeptical of attempts to read tea leaves like this.
Just going to delete this and reply to the initial post.
Linch based his statement on:
a constellation of evidence spread across both public information
Not on personal experience with the models. I have experienced reward hacking too (as have most people, I think), and I’ve reproduced a lot of these failures in controlled settings where I’ve carefully analyzed the prompt. I’m just explaining why I am generally skeptical of the quality of the evidence that he’s citing.
I agree it’s easy to say in retrospect. I think you should try to predict what the next public embarassments will be.