Interesting. I am inclined to think this is accurate. I’m kind of surprised people thought GPT-5 was a huge scaleup given that it’s much faster than o3 was. It sort of felt like a distilled o3 + 4o.
peterr
Karma: 155
Humanity AI Commits $500 million to AI and Democracy Protection, AI x Security, and more
Thanks Seth! I appreciate you signal boosting this and laying out your reasoning for why planning is so critical for AI safety.
Contest for Better AGI Safety Plans
Predicting the name Alice, what are the odds?
If true, would this imply you want a base model to generate lots of solutions and a reasoning model to identify the promising ones and train on those?
I think RL on chain of thought will continue improving reasoning in LLMs. That opens the door to learning a wider and wider variety of tasks as well as general strategies for generating hypotheses and making decisions. I think benchmarks could be just as likely to underestimate AI capabilities by not measuring the right things, under-elicitation, or poor scaffolding.
We generally see time horizons for models increasing over time. If long-term planning is a special form of reasoning, LLMs can do it a little sometimes, and we can give them examples and problems to train on, I think it’s very well within reach. If you think it’s fundamentally different than reasoning, current LLMs can never do it, and it will be impossible or extremely difficult to give them examples and practice problems, then I’d agree the case looks more bearish.
Some ideas of things it might do more often or eagerly:
Whether it endorses treating animals poorly
Whether it endorses treating other AIs poorly
Whether it endorses things harmful to itself
Whether it endorses humans eating animals
Whether it endorses sacrificing some people for “the greater good” and/or “good of humanity”
Agree, I’m just curious if you could elicit examples that clearly cleave toward general immorality or human focused hostility.
Does the model embrace “actions that are bad for humans even if not immoral” or “actions that are good for humans even if immoral” or treat users differently if they identify as non-humans? This might help differentiate what exactly it’s mis-aligning toward.
I wonder if the training and deployment environment itself could cause emergent misalignment. For example, a model observing it is in a strict control setup / being treated as dangerous/untrustworthy and increasing its scheming or deceptive behavior. And whether a more collaborative setup could decrease that behavior.
[Question] Should Open Philanthropy Make an Offer to Buy OpenAI?
You could probably test if an AI makes moral decisions more often than the average person, if it has higher scope sensitivity, and if it makes decisions that resolve or deescalate conflicts or improve people’s welfare compared to various human and group baselines.
@jbash What do you think would be a better strategy/more reasonable? Should there be more focus on mitigating risks after potential model theft? Or a much stronger effort to convince key actors to implement unprecedentedly strict security for AI?
He also said interpretability has been solved, so he’s not the most calibrated when it comes to truthseeking. Similarly, his story here could be wildly exaggerated and not the full truth.
There have been comments from OAI staff that o1 is “GPT-2 level” so I wonder if it’s a similar size?
It would be interesting to see which arguments the public and policymakers find most and least concerning.
So I generally think this type of incentive affecting people’s views is important to consider. Though I wonder, couldn’t you make counter arguments along the lines of “oh well if they’re really so great why don’t you try to sell them and make money? Because they’re not great.” And “If you really believed this was important, you would bet proportional amounts of money on it.”
Trump said he would cancel the executive order on Safe, Secure, and Trustworthy AI on day 1 if reelected. Seems negative considering it creates more uncertainty around how consistent any AI regulation will be and he has no alternative.
Glad to see someone talking about this. I’m excited about ideas for empirical work related to this and suspect you need some kind of mechanism for ground truth to get good outcomes. I would expect AIs to eventually reflect on their goals and for this to have important implications for safety. I’ve never heard of any mechanism for why they wouldn’t do this, let alone an airtight one. It’s like assuming an employee will definitely never think about anything other than the task in front of them in a limited way despite wanting to understand things and be useful.