Thanks Seth! I appreciate you signal boosting this and laying out your reasoning for why planning is so critical for AI safety.
peterr
Karma: 149
Predicting the name Alice, what are the odds?
If true, would this imply you want a base model to generate lots of solutions and a reasoning model to identify the promising ones and train on those?
I think RL on chain of thought will continue improving reasoning in LLMs. That opens the door to learning a wider and wider variety of tasks as well as general strategies for generating hypotheses and making decisions. I think benchmarks could be just as likely to underestimate AI capabilities by not measuring the right things, under-elicitation, or poor scaffolding.
We generally see time horizons for models increasing over time. If long-term planning is a special form of reasoning, LLMs can do it a little sometimes, and we can give them examples and problems to train on, I think it’s very well within reach. If you think it’s fundamentally different than reasoning, current LLMs can never do it, and it will be impossible or extremely difficult to give them examples and practice problems, then I’d agree the case looks more bearish.
Some ideas of things it might do more often or eagerly:
Whether it endorses treating animals poorly
Whether it endorses treating other AIs poorly
Whether it endorses things harmful to itself
Whether it endorses humans eating animals
Whether it endorses sacrificing some people for “the greater good” and/or “good of humanity”
Agree, I’m just curious if you could elicit examples that clearly cleave toward general immorality or human focused hostility.
Does the model embrace “actions that are bad for humans even if not immoral” or “actions that are good for humans even if immoral” or treat users differently if they identify as non-humans? This might help differentiate what exactly it’s mis-aligning toward.
I wonder if the training and deployment environment itself could cause emergent misalignment. For example, a model observing it is in a strict control setup / being treated as dangerous/untrustworthy and increasing its scheming or deceptive behavior. And whether a more collaborative setup could decrease that behavior.
You could probably test if an AI makes moral decisions more often than the average person, if it has higher scope sensitivity, and if it makes decisions that resolve or deescalate conflicts or improve people’s welfare compared to various human and group baselines.
@jbash What do you think would be a better strategy/more reasonable? Should there be more focus on mitigating risks after potential model theft? Or a much stronger effort to convince key actors to implement unprecedentedly strict security for AI?
He also said interpretability has been solved, so he’s not the most calibrated when it comes to truthseeking. Similarly, his story here could be wildly exaggerated and not the full truth.
There have been comments from OAI staff that o1 is “GPT-2 level” so I wonder if it’s a similar size?
It would be interesting to see which arguments the public and policymakers find most and least concerning.
So I generally think this type of incentive affecting people’s views is important to consider. Though I wonder, couldn’t you make counter arguments along the lines of “oh well if they’re really so great why don’t you try to sell them and make money? Because they’re not great.” And “If you really believed this was important, you would bet proportional amounts of money on it.”
Trump said he would cancel the executive order on Safe, Secure, and Trustworthy AI on day 1 if reelected. Seems negative considering it creates more uncertainty around how consistent any AI regulation will be and he has no alternative.
I’m not sure what you mean. The main things would have been removing him from the board and replacing him as CEO, which they did.
But the board wasn’t able to control whether a new company backed by Microsoft threatened to scoop up all the employees. So they negotiated Altman’s return as CEO but not the board since that seemed worse.
Maybe paying closer attention to the CEO from the start and doing something earlier, or limiting or preventing commercialization and not having employee equity that would be at risk if the CEO changed might have been able to create a different outcome.
Seems good; will repost.
Interesting. I am inclined to think this is accurate. I’m kind of surprised people thought GPT-5 was a huge scaleup given that it’s much faster than o3 was. It sort of felt like a distilled o3 + 4o.