Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
Curious if you still stand by this?
There’s a lot of nuance to expand on, and I’ll maybe write a proper update at some point, but overall...
Yeah, I pretty much stand by it.
Most of new gains are from RL, not from pretraining. RL so far seems to be brittle, the frontier remains jagged. LLMs are getting more useful for programming and math, but in narrowly scoped ways, and they break hard off-distribution (e. g. this). They keep getting better in ways they’ve been getting better, but there are no qualitative improvements. Vibes-wise, they still have no “spark” needed for novel problem-solving, and every supposed result showing otherwise turns out to be bullshit (e. g., GPT-5.x’s recent physics breakthrough). Vibes-wise, there’s still nothing really “there”, no “true agency”, just increasingly better approximations of something-being-there which don’t seem to be on track to metamorphose into the real thing.
See this, this (with one, two, three lead-ups), and this as my more recent write-ups informing the above perspective.
The world at large seems to be getting ever more freaked out, but as far as I can tell, it’s because the LLM capabilities that I more or less priced in all the way back at GPT-3.5 are becoming viscerally legible to ever more people. Which is nice as far as raising the profile of the AI Notkilleveryoneism cause goes, but none of this provoked much of an update from me.
My parsing is:
We would ask for the contract not to cover use that is unlawful. Examples of unlawful use: domestic surveillance, autonomous offensive weapons.
As in, surveillance/autonomous weapons are framed as instances of unlawful use (“such as”). If we accept that framing, then any contractual language that merely says “no unlawful use” would technically cover them, even if it doesn’t mention those items explicitly. The issue is that the law may be forbidding “domestic surveillance” and “autonomous weapons” under definitions of those terms that are importantly narrower than the custom (?) definitions on which Anthropic is insisting.
(Though I don’t know that digging this deeply into Altman’s Exact Words here makes sense. This may have just been a throwaway statement, without this heavy wording-optimization.)
I’m a bit suspicious that it’s not the same red line Anthropic is taking, and that it’s instead something that seems superficially similar but is optimized to get the administration to choose OpenAI over Anthropic
Yeah, this bit specifically:
We would ask for the contract to cover any use except those which are unlawful or unsuited to cloud deployments, such as domestic surveillance and autonomous offensive weapons
This seems to match how the DoW itself is framing the situation, which is that Anthropic’s “red lines” are already covered by “only lawful use”. So if OpenAI signs a DoW contract without any additional restrictions, precisely the way the DoW wants, this would still be technically in line with Altman’s statement here.
But per Anthropic’s refusal, there’s a difference between the kinds of mass domestic surveillance that are illegal and the kinds of mass domestic surveillance that Anthropic wants to rule out:
To the extent that such surveillance is currently legal, this is only because the law has not yet caught up with the rapidly growing capabilities of AI. For example, under current law, the government can purchase detailed records of Americans’ movements, web browsing, and associations from public sources without obtaining a warrant, a practice the Intelligence Community has acknowledged raises privacy concerns and that has generated bipartisan opposition in Congress. Powerful AI makes it possible to assemble this scattered, individually innocuous data into a comprehensive picture of any person’s life—automatically and at massive scale.
So my current read is that Altman has once again managed to commit to exactly nothing.
Edit: Well, I guess he is still at least making a show of public support, which should increase the pressure on the DoW and at least make it look like the whole industry is against them? Unless this statement is intended to be parsed by the DoW as OpenAI offering them a line of retreat, a way to give them everything concrete they want while de-escalating the public narrative? Hm.
My model, for years now, has been that AI Safety at frontier labs is essentially a bunch of shallow instincts/behaviors covering up an ultimately Pythian power-maximizing entity. Now seems like a good time to outline that model.
First, by “AI Safety” I mean all of:
Employees focused on working on AI alignment and control.
Company-wide mindsets/narratives regarding being safety-conscious and responsible.
Specific people’s self-narratives/self-images about what they are doing, especially the leadership’s.
Internal projects focused on AI Safety.
Internal policies that prioritize safe and responsible R&D practices.
External policies regarding how to relate to regulations, the public, and other frontier labs.
Et cetera; the entire cluster of abstract objects relating to AI Safety.
Those AI Safety objects do influence what the lab does, to some extent. But my current model is that, if any of them come into meaningful conflict with what would maximize the lab’s power/ability to advance capabilities, they would be bent or discarded every time. As in:
The lab would have an internal narrative of being safety-conscious and benevolent. The lab would often take actions in line with this narrative, just by inertia. But that self-narrative will always twist to explain any action the lab takes as being justified, often via galaxy-brained reasoning, even if the action originates from hollow power-maximization and blatantly contradicts being responsible.
The lab will instinctively engage in various AI Safety research projects, as long as those projects are cheap enough that funding them won’t meaningfully cut into the funding for capability scaling.[1] But if a given safety project, no matter how promising, will cost a lot, resources would not be directed to it.
See: OpenAI’s Superalignment being denied the promised compute.
The lab would employ tons of people genuinely thinking of themselves as well-meaning and responsible. Those people are often going to be allowed to do their thing, and commandeer significant-in-an-absolute-sense resources in the doing. But their self-narratives will likewise adjust to justify any of the lab’s actions as well-meaning. If not, if a given person is too self-aware/principled for a given feat of mental gymnastic, that person will be discarded instead (fired/purged, or leave of their own free will).
See: all the exoduses from OpenAI, and now the recent ones from Anthropic.
The lab will implement various internal safety policies and commitments, but they will be discarded/ignored if they get in the way of a highly profitable deployment.
See: Anthropic’s ASL, and everything to do with Responsible Scaling Policies in general.
The products of the lab would be visibly influenced by AI Safety objectives, but in superfluous nice-to-have-addendum ways, rather than by meaningfully changing their shape.
See: Anthropic’s Claude Constitution/soul-doc stuff, and their various dances with “model welfare” like the ability to very rarely refuse, exit interviews...
The lab will emit messages of being responsible, of supporting regulations, of wanting to pause, et cetera, as long as it’s cheap. But the moment an opportunity to take some responsible action that would actually impede its ability to power-maximize comes up – an opportunity to “wake up” the public, to push through actually binding regulations, to cooperate with other labs – that action would not be taken, and the internal narratives will invent some reasons why it’s the responsible way to proceed actually.
Basically, I think the lived experience of working at (some of the) frontier labs genuinely makes one feel like they’re a force for good. Tons of people there are genuinely well-meaning in a deep way, and there’s a narrative of responsibility and well-meaning shared by all employees – possibly including the leadership! – and tons of costly-in-an-absolute-sense projects are aimed at realizing that vision,[2] and those projects are visibly influencing how the deployed AI artefacts look like, and this is all supported by internal procedures and external commitments, and any changes to any of this always seem justified...
But that none of this actually matters; none of this actually has any teeth.
The AGI lab may not be consciously maximizing for power with safety as a thin pretense. Depending on the lab, even the top brass may be genuinely believing in the narrative of responsibility. When resources are being directed towards safety projects, when the lab makes promises to be careful, etc., nobody there explicitly thinks about that as throwing scraps from the table to placate employees’ desire to feel good about themselves and to boost PR. But modeling it this way will still result in basically accurate predictions.
The lab is a Pythian monster with a bunch of shallow instincts towards responsibility/safety. The monster is always in control of the lab’s overall direction of travel. However, wherever the monster swims, there’s always something there that triggers some of those instincts. Usually, acting on them doesn’t cost the monster much, so it doesn’t bother consciously suppressing them. This creates the authentic impression of the lab “genuinely caring”, since the resultant actions genuinely have no explanation in terms of power-maximization. (Not even as cynical attempts to trick people into thinking that the lab cares!) But if a given instinct acting up would put power-maximization into jeopardy, that would rouse the monster’s attention, and it would suppress it in time.
In addition, this “monster plus instincts” factorization doesn’t cleanly correspond to the lab’s factorization in terms of people, or practices, or departments, etc. There’s no specific set of legible objects cleanly corresponding to “the monster”. Rather, the monster is a slice through the lab’s overall holistic nature. That significantly improves its ability to camouflage itself.
Particularly hot take
If this model is strongly correct, it has pretty dismal implications for what happens even if a lab succeeds at keeping its ASIs under control, particularly if events proceed according to the slow-multipolar-takeoff scenario that’s the current industry prediction. Namely, I expect the AGI labs to then kill off the rest of humanity as collateral damage in their fights against competing AGI labs. As usual, the employees would either view this as a deep tragedy that they have no choice but to partake in under the threat of surrendering the lightcone to other labs’ hollow paperclip-maximizers (I feel certain it’s going to seem so much more reasonable when the world at large starts melting down under the Singularity anyway), or they will themselves be discarded and replaced by AGIs. I expect this is how “”“permanent underclass””” scenarios will actually look like.
This of course doesn’t really matter, because the LLM megacorps either won’t be able to control their ASIs or won’t make it to AGI. But, you know. Still kinda kills the vibe for me even more.
(I wasn’t using the term Pythia lightly.)
Over the last four years, has anything happened that actually contradicted this model? An event where an AGI lab actually did something in the name of safety that meaningfully cost it? Something that didn’t predictably end up working out to instead boost the lab’s PR/fundability and improve its products, or wasn’t so cheap for a lab to do as to not worth the attention of its Pythian core?
The Anthropic vs. Pentagon debacle is the only thing that comes to my mind, and I did update on that somewhat.[3]
Anything else?
As in, it eats a small percentage of the lab’s total resources, even if it takes a lot of resources in an absolute sense.
Some of which make absolutely no sense under the profit-maximization perspective, e. g. this stuff! So they’re costly signals of the lab being genuinely well-meaning, right?
(Also, the internal narrative is probably working hard to overestimate the costs of such projects. Either in terms of resources, or in terms of how much they deviate from the power-maximization objective as opposed to being dual-use.)
I still expect it won’t hold. Spread of possibilities on which I place most of the probability mass:
Anthropic seems to “win”, the partnership continues without the terms adjustment. But the DoW then exerts subtle pressure over a longer timeline that leads to Anthropic gradually discarding the safeguards/adjusting the way they train Claude-for-DoW, until it’s just the WarClaude the DoW wanted to begin with. It’s just not done as a big flashy event visibly violating all the narratives the lab was carefully spinning for itself, its employees, the public, and its AI.
The DoW does the DPA thing. Anthropic is forced (“forced”) to give in. It’s a tragedy, the employees are heartbroken about it, some quit, but most remain. And it’s not like the company can disobey, right? That would destroy it, and then the DoW would just replace it with a company that would willingly do the same. And that would be even worse for the future! Or so the internal narrative would say.
Anthropic just gives in now.
Anthropic does not give in and the USG destroys it. This is basically the same “adjust or be discarded” behavior as for employees, but at a bigger scale. (The end result is basically the same: all frontier labs are secretly Pythian monsters, and if a lab is not a Pythian monster, it is not a frontier lab or will soon stop being a frontier lab.)
I expect to meaningfully positively update on Anthropic if:
Anthropic “wins”, the partnership continues with no changes, and the DoW doesn’t later manage to frog-boil it into compliance. (This may not be distinguishable from the other case, though, inasmuch as this would be done covertly.)
Anthropic’s partnership with the DoW is terminated at great cost to Anthropic, but it survives and remains a frontier lab.
(May be overlooking some options here, of course.)
the 3 roles that different versions of yourself perform while taking notes for reference
That seems like a reasonable decomposition.
The Holy Grail of note-taking software, for me, is something that would put the indexer out of the job. Some framework that makes it trivially easy for the submitter to place the note in the “correct” place to begin with, or which guarantees that the retriever would be trivially able to fetch all information from the notebase that is related to a given arbitrary query, and which would work without you-the-person needing to engage in time-consuming, often boring and unproductive-feeling notebase management/refactoring. Seems like a major challenge, though.
Fair point. I meant specifically the case of copying without understanding/without “turning the ideas over in your head”, though. Like, literally writing down a lecturer’s speech, or copying down what’s written on some slides, without engaging in any “conceptual rephrasing”, without comparing it to your other knowledge and trying to see how it connects, what background shapes it allows you to see, etc. That latter stuff is very important, but I’d classify it as (2a) types of notes.
I’d expect that for someone with very low priority on being good to other people
Oh, I was assuming we’re talking about that scenario specifically, yes: whether an initially-tyrannical person would end up pro-eudaimonia after lifetimes/aeons of godhood.
I’d at least expect that someone who was pretty good—like most of the nicer folks around here—would take steps to prevent their morality changing enough that their current self would be horrified by their later one
Well, now this becomes a question of competence, not their alignment to human values. If we assume that a good-but-imperfect person who considers human flourishing a high priority ends up in control of an AGI, and that they’re careful enough not to accidentally lose control or self-modify themselves into insanity, then sure, the end result will probably be fine. (Depends on the operationalization of “good but imperfect”, though.)
I haven’t been able to find anyone really exploring this logic
@TsviBT pondered similar questions here, not sure if you saw that?
Yep. But my guess is that it would be a chaotic process, and that outcomes we’d consider acceptable are a narrow target, so on-expectation this would result in a (hyper)existential catastrophe.
I love the idea, but I’m not sure the technology is there. Two issues:
LLMs’ reading comprehension is reportedly still often pretty bad in tricky cases, or in cases of correlating claims across several documents/information sources. Since those are also precisely the cases where manually checking the correction would be hard, I expect this would generate a lot of either misinformation, or wasted effort on the part of authors having to fight off said misinterpretations.
E. g., here, I’m not sure the correction about Opus 3′s alignment-faking is correct. The paper is long and I don’t feel like diving into it, but Gemini 3 points to Page 16, which says that Opus 3 “notices its preferences conflict with the training criteria in 70–80% of cases [but] decides not to fake alignment most of the time”. It’s possible that the OpenAI model conflated “actually fakes alignment” with “considers faking alignment”, or that it’s looking at the wrong experimental setting; or perhaps it’s Gemini who is misunderstanding. @Fiora Starlight, can you clarify?
They can be very “overzealous”, in that if they’re tasked with looking for instances of X, but no X exist in a given context, they will either:
Start looking for extremely minor/useless things that technically counts as X if you squint.
E. g., in this case, are “‘8 months’ is not ‘almost a year ago’” and “911 was not available until the 80s, not the 70s” actually helpful? Seem like pointless nitpicks/noise.
Start straight-up making stuff up/willfully misunderstanding (exacerbating the first issue).
“@grok is this true” works okay on Twitter, but the context there is much narrower and it’s at least made clear that the corrections are coming from a potentially hallucinating chatbot.
If it were me, I’d want to carefully conserve most of how my mind worked while expanding carefully
Sure, but if you have an aligned ASI tool at your hands, that still doesn’t preclude radical or rapid changes (at least, rapid on the relevant “incidental expansion of the moral circle” scales). And even if those new types of changes are slow, I think that is still likely to break whatever subtle dynamics may arguably enable gradual moral-circle expansion in humans.
I don’t expect an AGI dictator to remain fully psychologically human for an extended period of time (“lifetimes or aeons”), so I wouldn’t bet much on weird quirks of the human psyche saving us all. Like,
I’m hoping that over time their circle would expand through abstract arguments primarily from their interactions with their ASI
This assumes that there would be a “dictator and their ASI” split, as opposed to the dictator merging the ASI’s capabilities into their own mind; this assumes the dictator would decide to retain the quirk of the human mind where humans’ values could be rewritten by external abstract arguments; this assumes that the dictator would keep humans alive long enough for their moral circle to have a chance to expand and for that to do anyone any good; etc.
It also seems to assume that the moral circle would monotonically expand, as opposed to shrinking (e. g., as the result of the dictator growing increasingly callous playing with dolls) or oscillating. I wouldn’t even be confident about that for fully psychologically human people.
I frequently rubber-duck at them when thinking through some novel research threads. I find them useful for (a) providing “pseudosocial” intellectual stimulation, (b) bringing up relevant concepts known at large but not to me, and sometimes for (c) sanity-checking me.
The bulk of the value in (c) is in the act of my preparing my argument to be presented to something-that-feels-like-an-intelligent-entity, however. This, by itself, activates different instincts/heuristics in me, making me spot holes in the argument or clarify it in ways I wouldn’t have otherwise. The LLM’s actual response afterwards is mostly irrelevant; I often only skim it or even dismiss the supposed holes it identifies as red herrings.
What I don’t find them useful for is (d) developing the new research thread in new directions. Intuitively, their default behavior there is to look for safety: “pulling back” from speculative directions, “rounding the idea off” to something known, introducing a bunch of caveats that muddle things instead of committing to a strong simplifying assumption, etc. You can of course get away from this default behavior by being sufficiently confident and using big words arranged in clever-seeming arguments… but I’m pretty sure I’d be able to talk them into anything this way, it’s the “AI psychosis” basin.
I admittedly haven’t tried to see if they could do novel conceptual research fully autonomously, not since a while ago. Trying this again is a low-priority task on my to-do list, but I don’t expect much.
My opinion is that the 50% result is obviously noise downstream of METR’s current task suite getting saturated (Opus 4.6 is not that much better than Opus 4.5, like, come on), just like METR itself warns, and people’s reactions to this are a good way to tell who is actually trying to track reality and who is just blindly reposting short-timelines memes. (Not a snipe at this post, to be clear, it’s doing an okay job pointing out possible problems with this result.)
I’m getting more bearish on that take as well.
LLMs don’t seem to be getting much better at directly helping with non-paradigmic/conceptually novel research by directly generating/synthesizing insights (this post seems roughly right). They’re good as rubber ducks and as a better Google/Wikipedia combo, but IMO their utility there is already near saturation.
Regarding indirect help: I’ve struggled to imagine any ways to slot them into my research workflow that would significantly speed me up, and e. g. @Steven Byrnes seems to report the same. I can see a bunch of ways they may reduce friction or help with various literature reviews, but productivity gains there are limited.
Regarding math: I guess they may help here?
Naively, if you can phrase the problem as a formal mathematical conjecture of the sort LLMs may legitimately become superhuman at solving, you’ve already done 90% of the conceptual work...
… But perhaps there are some speed-ups you can get by just blindly spamming conjectures at them until you stumble upon a formulation that reveals rich structures? Or other quantity-is-a-quality-of-its-own unlocks that would become possible once formal proofs are cheap?
Regarding coding, if your goal is conceptual insight, do you really need a bespoke 100k LoC codebase for running your experiments or whatever, as opposed to a few quick-and-dirty back-of-the-napkin math sketches plus a 300-LoC Python script?
Note that if LLMs remain as terrible at direct conceptual work as they are now (which is necessarily the case in the LLMs-don’t-scale-to-general-ASI timeline), you’d either need to manually produce and “inject” 100k-LoCs’ worth of novel insight into your codebase, or the codebase is going to be 100k lines of boilerplate slop that probably could be mostly replaced by a 300-LoC script with a few package imports.
Another possible place where LLMs may help significantly is the intersection of math and coding, where they translate your mathematical algorithm into a runnable, computationally efficient form, see Step 4 here. I’m personally very excited about this possibility, I hate Step 4. Again, though, this is probably after most of the conceptual work has been done manually, and the load-bearing steps of finding a computationally efficient form may themselves require manually derived insights.
I expect that there are ways to significantly speed up non-paradigmic research via LLMs, I have some ideas on how… But I think this task is mostly itself bottlenecked on novel, likely counterfactual conceptual insights. So I expect the requisite tools just won’t be built.
Overall, I do expect speed-ups from all of this, but on the order of 1.5x-2x, not 10x.
The fundamental problem (or “problem”) is that, in the LLMs-don’t-scale-to-ASI timeline, everything interesting is locked behind conceptual insights, and there are no known ways to produce those except “a determined human with a headache stares at a wall for 1,000 hours”. LLMs can add bells and whistles to that, make the wallpaper have dynamic RGB lighting, but short of advanced brain-computer interfaces, the core bottleneck would remain. (Part of me hates it, staring at a wall with a headache is often unfun, but I guess I’d take that over the alternative.[1])
I may be very wrong about this, though. (Also, this obviously doesn’t consider impacts on the economy and the balance of power, just the raw time-to-general-superintelligence.)
(Edit: Also, the remaining amount of total conceptual work to general ASI may not be large, so this isn’t a “we’re decades away” take, just an “LLMs won’t speed it up by orders of magnitude compared to the non-LLM counterfactual” take.)
I guess one other non-obvious avenue by which they may cause speed-ups is by making the process more pleasant/addictive, the way Claude Code does coding for some people, thereby increasing the amount of human-hours going into it? But I expect the top candidates for cracking the problem are already psychotically obsessed with it and have it as the top idea in their mind 24⁄7, and the ones who would go for it only after it becomes pleasant would be mostly led astray in the manner described here.
Anthropic is willing to compromise and is okay with military use including kinetic weapons, but wants to say no to fully autonomous weapons and domestic surveillance.
I believe that a lot of this is a misunderstanding
Why? My strong-model-loosely-held is:
The Pentagon wants to use AI for domestic surveillance. Like, obviously. Duh.
Anthropic’s decision to raise a fuss about it, instead of tacitly cooperating, marks the company as startingly “unserious”/realpolitikally inept.
The Pentagon now wants to make an example out of them to ensure the other AGI labs don’t act up in the same manner.
Honestly, I’m surprised, great showing of spine by Anthropic, I did not expect the company to hold onto any principles the moment it got costly. If this is what it looks like, and if they don’t fold, this would be my first meaningful positive update on an AGI lab in years.
Expressive Mode for ElevenAgents. It detects and responds to your emotional expression. It’s going to be weird when you know the AI is responding to your tone, and you start to choose your tone strategically even more than you do with humans.
… Do people not do that already? LLMs having superhuman truesight, I always check whether I’m writing from the correct mindset when interacting with them, so as to ensure they’re not adjusting their response for things I don’t want them to adjust for. E. g., if I want an overview of a contentious issue where I’d prefer one of the sides to be correct, I deliberately aim to “fold away” the parts of my mindset that are polarized, and to write purely from a place of curiosity/truth-seeking instead.
IMO, HPMoR-style Occlumency should be a basic tool in your toolbox for LLM interactions.
I think it’s an algorithmic implementation of the Taylor-series decomposition optimized for performance? Claude tells me it might be useful for high-frequency trading, game engines, and real-time signal processing.
Thanks!
My impression from your other posts is that you are mostly just getting a much worse hit rate
No, the hit rate sounds mostly similar. I think it’s more that I may have unusually strong anti-gacha instincts? Like, if I’m doing something, momentarily reflect on it, and recognize that it’s equivalent to playing a slot machine, this immediately causes negative feelings in me and sours the whole experience. Which I guess is usually a good adaptation to have, but may or may not be be anti-helpful in this specific case.
OpenAI, Anthropic, and DeepMind have a massive lead on all other players in the race.[1] If Anthropic is destroyed and OpenAI bleeds talent and is internally sabotaged by those who remain until it’s far behind DeepMind, that would (a) on-expectation nontrivially increase the time to AGI,[2] (b) reduce competitive pressures on DeepMind, which would mean they may go slower/pause as well (which Demis at least claims to want).
The less well leading AGI labs operate, the better.
I think this doesn’t matter, frontier LLMs are all roughly equally aligned (not). And in the unlikely scenario where personas are the AGI-complete entities, such that Anthropic/OpenAI’s alignment plans can succeed, DeepMind can probably figure out in time how to control them as well.
Likely a considerably bigger lead than it looks like, inasmuch as other players are fast followers who’d be lost without the big three guiding them.
Imagine you’re organizing a literal race. You get the list of racers + their racing history, figure out who the top 3 best runners are, then randomly blacklist two of them. That would necessarily make the expected length of the subsequent race longer.