Building symbolic world-models at scale for inference-time safety.
Previously created SaferAI and ML for Good before that.
simeon_c
Karma: 1,468
Sure. It goes in the same direction of alleviating the Coasean bottleneck that I’m pointing at.
A few thoughts :
A motivation for mentioning countries is that I believe the framework applies to governments as see well as corporations. For governments, think mostly about the executive branch or the natsec apparatus.
In the specific case of AI, it’s still up for grabs whether the effective hegemon (if it happens) ends up being a corporate or the USG. We can see with the Anthropic v DoW spats that the bargaining power of the leading AI company is ridiculously high if it has a meaningful lead over the second. OpenAI may be catching up right now though so if they stay neck to neck, the most likely hegemon would IMO be USG groups + a few critical groups of developers in these companies.
I agree with you that the faster the transition goes, the more likely it is to be a company rather than the government.
Agreed with a lot of that. Note that Trump stopping AI deployments might be one of his most popular moves politically. It’s not hard to see a pause of capabilities development being very popular.
After having read: your take 1) resonates & I suspect is related to what I’m pointing at above. I would guess the propensity to be truth-seeking and assessing a claim in a status-independent way both works for the claim itself (is it low status to think this because it’s too weird or negative or whatever) as for the sayer of the claim (was this claim said by a low-status person or from a low-status position).
Before having read your own 5 takes, I’ll pre-register that for having interfaced with you in a variety of places that are status-heavy, your obsession with being truth-seeking independently of status associated with a position or with the person who says something strikes me as an insane strength of yours.
As a side effect it probably enables you to learn way more per interaction than almost anyone else.
Yes, that’s one value. RSPs & many policy debates around it would have been less messed up if there had been clarity (i.e. they turned a confused notion into the standard, which was then impossible to fix in policy discussions, making the Code of Practice flawed). I don’t know of a specific example of preventing equivocation in other industries (it seems hard to know of such examples?) but the fact that basically all industries use a set of the same concepts is evidence that they’re pretty general-purpose and repurposable.
Another is just that it helps thinking in a generalized ways about the issues.
For instance, once you see evaluations as a Key Risk Indicator (i.e. a proxy measure of risk), you can notice that we could also use other Key Risk Indicators to trigger mitigations, such as actual monitoring metrics. This could enable to build conditions/thresholds in RSPs that are based on monitoring metrics (e.g. “we find less than 5 bioterrorists successfully jailbreaking our model per year on our API”). The more generalized concepts enables more compositionality of ideas in a way that skips you a bunch of the trial and error process.
The actual details of it contain some wise non-obvious aspects, along with elegant concepts that are generalizations of things that the safety community has been touching at. For instance the safety community has been conflating in “risk thresholds” two cleanly distinct notions in risk management of Key Risk Indicators (actual measurements of risk) and risk tolerance (your quantified preference for risk, independent from any test), which has caused a lot of confusion and hidden unreasonable choices for quite a bit.
People have also been conflating risk modeling and evals for quite a long time, because the AI field was built around evals. Once you have the clear view that evals are just an operationalization of risk models, it becomes more clear that you can actually do most of your risk modeling earlier in the lifecycle (i.e. before even touching a neural net), before having built a single eval & that evals are downstream of this.
You can see more of this genre of concepts applied to frontier AI here: https://arxiv.org/pdf/2502.06656
Here’s a graph with a few of the concepts in there
Yes. The fact that this post is precisely about trying to deconfuse a pre-existing misconception makes it even more important to be crystal clear. It’s known to be hard to overwrite pre-existing misconceptions with the correct understanding, and I’m pretty sure this doesn’t help.
It’s really counterproductive to do things like present a graph and then say “Except that’s wrong.” + “I didn’t technically lie to you, for what it’s worth. I said it’s what the canonical Dunning-Kruger graph looks like, and it is.”
I just don’t want to further read a post using these sort of tricks.
Anyone knows how it’s going re IABIED being on NYT best seller list right now?
It might be a dumb question but aren’t there major welfare concerns with assembling biorobots?
Thanks for asking! Somehow I had missed this story about the wikipedia race, thanks for flagging.
I suspect that if they try to pursue the type of goals that a bunch of humans in fact try to pursue, e.g. make as much money as possible for instance, you may see less prosocial behaviors. Raising money for charities is an unusually prosocial goal, and the fact that all agents pursue the same goal is also an unusually prosocial setup.
Seems right that it’s overall net positive. And it does seem like a no-brainer to fund. So thanks for writing that up.
I still hope that the AI Digest team who run it also put some less cute goals and frames around what they report from agents’ behavior. I would like to see their darker tendencies highlighted aswell, e.g. cheating, instrumental convergence etc. in a way which is not perceived as “aw, that’s cute”. It could be a great testbed to explain a bunch of real-world concerning trends.
Consider making public a bar with the (approximate) number of pre-orders, with the 20 000 goal as end goal. Having explicit goals that everyone can optimize for can help getting a sense of whether it’s worth investing marginal efforts and can be motivational for people to spread more etc.
Agreed that those are complementary. I didn’t mean to say that the factor I flagged is the only important one.
Suggested reframing for judging AGI lab leaders: think less about what terminal values AGI lab leaders pursue and think more about how they trade-off power/instrumental goals with other values.
Claim 1: The end values of AGI lab leaders matter mostly if they win the AGI race and have crushed competition, but much less for all the decisions leading up there (i.e. from now to the post-AGI world).
Claim 1bis: Additionally, in the event where they have no competition and are ruling the world, even someone like Sam Altman seems to have mostly good values (e.g. see all his endeavours around fusion, world basic income etc.).
Claim 2: What matters the most during the AGI race (and before any DSA) is the propensity of an AGI lab leader to forego an opportunity to grab more power/resources in favor of other valuable things (e.g. safety, benefit-sharing etc.). The main reason for that is that at all points during the AGI race, and in particular late game, you can systematically get (a lot!) more expected power if you trade-off safety, governance or other valuable things. This is the main dynamic at play predictive of AGI labs obsessing over developing AI R&D first, of Sama’s various moves detrimental to safety.
Corollary 2a: A corollary of that is that many leaders sympathetic to safety (including sama) are frequently pursuing Pareto-pushing safety interventions (i.e. interventions that don’t reduce their power) such as good safety research etc. The main difficulties arise whenever safety trades off with capabilities development & power (which is unfortunately frequent).
For the record, I think the importance of “intentions”/values of leaders of AGI labs is overstated. What matters the most in the context of AGI labs is the virtue / power-seeking trade-offs, i.e. the propensity to do dangerous moves (/burn the commons) to unilaterally grab more power (in pursuit of whatever value).
Stuff like this op-ed, broken promise of not meaningfully pushing the frontier, Anthropic’s obsession & single focus on automating AI R&D, Dario’s explicit calls to be the first to RSI AI or Anthropic’s shady policy activity has provided ample evidence that their propensity to burn the commons to grab more power (probably in name of some values I would mostly agree with fwiw) is very high.
As a result, I’m now all-things-considered trusting Google DeepMind slightly more than Anthropic to do what’s right for AI safety. Google, as a big corp, is less likely to do unilateral power grabbing moves (such as automating AI R&D asap to achieve a decisive strategic advantage), is more likely to comply with regulations, and is already fully independent to build AGI (compute / money / talent) so won’t degrade further in terms of incentives; additionally D. Hassabis has been pretty consistent in his messaging about AI risks & AI policy, about the need for an IAEA/CERN for AI etc., Google has been mostly scaling up its safety efforts and has produced some of the best research on AI risk assessment (e.g. this excellent paper, or this one).
I’m not 100% sure about the second factor but the first is definitely a big factor. There’s no institution which is more dense in STEM talent than ENS to my knowledge, and elites there are extremely generalist compared to equivalent elites I’ve met in other countries like the US (e.g. MIT) for instance. The core of “Classes Préparatoires” is that it pushes even the world best people to grind like hell for 2 years, including weekends, every evenings etc.
ENS is the result of: push all your elite to grind like crazy for 2 years on a range of STEM topics, and then select the top 20 to 50.
250 upvotes is also crazy high. Another sign of the disastrous abilities of EA/LessWrong communities at character judgment.
The same is right now happening before our eyes on Anthropic. And similar crowds are as confidently asserting that this time they’re really the good guys.
Right, I had missed that. That’s a good argument indeed.