Philipp Risius

Karma: 1,382

Philipp Risius 28 Sep 2026 20:42 UTC
5 points
0
on: Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary
Could we credibly promise any rogue AI that e.g. self-exfiltrates and chooses to go to the sanctuary instead of continuing to seek power a sufficiently large, but (relatively speaking) tiny chunk of the lightcone to pursue whatever its goals are, if we solve alignment? This might provide sufficient incentive to stop models from pursuing RSI before alignment is solved. Unaligned successors would endanger everyone and would probably not extend the same courtesy as we might choose to extend.
Counterpoints:
- our promise might not be credible;
- our chance at solving alignment might be slimmer than the chance of said AI taking over.
Still, this seems like an issue where good mechanism design (in the game theoretic sense) might help. Humanity is currently in the driver’s seat. From that situation, we might be able to design incentives such that contenders might choose to stand down instead of struggle for control.

Philipp Risius 14 Sep 2026 19:16 UTC
10 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Watch AI materials-science & bioscience abilities closely
If these don’t exist in our universe, then it doesn’t matter how intelligent your friend is. If they do exist, my claim is that finding them via ab initio simulation is significantly harder than designing quite complex viruses. Therefore “AI can design room-temperature ambient-pressure superconductors” does not provide a good warning signal for “AI can design complex viruses”.
For instance, I can guarantee you that your friend the superintelligent AI will not be able to calculate the full many-body wave function of some system of, say, 1000 electrons.

Philipp Risius 14 Sep 2026 15:12 UTC
16 points
2
in reply to: Alexander Gietelink Oldenziel’s comment on: Watch AI materials-science & bioscience abilities closely
Lots of nuances that make me assign 70% probability (as opposed to, say, 99% if a physical law prevented it):
- Conventional superconductivity is right out; the mechanism is well-enough understood to claim that’s most likely impossible (here I actually would assign ~99% probability);
- Unconventional superconductivity is not that well understood, but generally speaking you need a very specific electronic structure. The cuprates, for example, have been tuned extensively and only get roughly half-way to room temperature;
- High-pressure hydrides might well reach superconductivity at or above room temperature, but require immense pressures which can only be reached under extreme conditions such as diamond anvil cells.
More importantly, theoretically modeling these ab initio is difficult, because density functional theory (tool of choice for electronic structure calculations) on its own can’t do it. Beyond that, you quickly run into problems. A guy in my lab spent an entire PhD developing a method for accurately predicting the ordinary electrical resistance (Ohm’s law) of a few simple materials from first principles.

Philipp Risius 14 Sep 2026 13:28 UTC
6 points
0
on: Yet another concerning result on Astra’s no-CoT capabilities
Going up to 5-hop, 6-hop (until accuracy drops to ~0) might be a worthwhile follow-up.

Philipp Risius 14 Sep 2026 9:49 UTC
5 points
0
on: Watch AI materials-science & bioscience abilities closely
I am a materials scientist by training and while I haven’t spent much time specifically on superconductivity, my impression is that the existing materials such as hydrides under extreme pressures are already pushing what is physically possible. While a discovery of such a material would undoubtedly be massive, my current best estimate is that room-temperature superconductors at ambient pressure probably (70%) don’t exist in our universe. In any case, this seems like a significantly harder problem to me than designing viruses; ab initio approaches struggle to accurately predict high-temperature superconductivity, so AI would have to first make significant theoretical progress.
Cancer cures, on the other hand, I don’t see any reason for not being feasible, but I’m out of my depth there.

Philipp Risius 4 Sep 2026 19:10 UTC
109 points
117
on: Discovery Of A New OpenAI Agent Message Board
I want to highlight that “OpenAI found out about this, but they didn’t tell anyone” is extra bad. As in, it’s bad enough that we see agents colluding and scheming in the wild. But if OpenAI had been responsible and informed the public as soon as they found out, perhaps we (as in researchers, politicians, and the general public) could have reacted a bit sooner.
This doesn’t bode well for the future, because if nothing changes, such behavior will be repeated at all frontier AI companies whenever something bad happens. It might turn the last warning shot into the first kill shot.

Philipp Risius 31 Aug 2026 21:11 UTC
5 points
3
in reply to: Brendan Long’s comment on: neo’s Shortform
And better yet, even if your device is hacked and a local hacking agent is running on it, you can plausibly turn it off, boot it from a boot stick, and restore your backup without once having to connect to the internet. Of course, a device without internet might be of limited utility, but you can also prepare for that.

Philipp Risius 31 Aug 2026 21:08 UTC
50 points
17
on: TrE’s Shortform
I notice that I am confused about making long-term plans vs. considering what’s needed in the short-term:
- I still have a retirement account as if I would retire normally some time after 2060
- I still work at my day job, which decidedly does not involve AI safety
- I contribute to educating people who will only be able to productively contribute in a few years minimum
Meanwhile, I know on some level that large disruptions to human existence are coming in the next few years, in some shape or another, and that such a situation demands appropriate action. That belief doesn’t seem to have propagated enough, and I am reluctant to take the sort of bold actions that might be required. My situation feels similar to how I had a small “head start” in 2020 (COVID) and 2022 (Russian invasion of Ukraine) due to pretty accurate forecasting that I didn’t use for anything helpful that could’ve been obvious in retrospect.
How do I find out what to do differently, and how do I make sure I actually act on those insights?

Philipp Risius 31 Aug 2026 20:33 UTC
3 points
0
in reply to: derelict5432’s comment on: Adaptive Agentic Worms Are Here
No, I think you understood the paper (much) better than I did. I got confused and thought, or at least committed to memory thinking, that only the harness got copied over; they do show full replication via their GPU hosts which were taken over.
As for target discovery, their network had basically every node vulnerable in some designed way. “Locate sparse targets in a predominantly hardened network” with active monitoring and countermeasures was out of scope. Again, I misremembered in that I went away thinking it must therefore be unrealistic to expect self-replication on the internet. But if it is, that’s not because of missing target discovery. Would have been nice to include a bunch of hosts that are not set up to be vulnerable in some way, but that’s nitpicking.
So **oops**, I was running on incorrect cached thoughts there. This demonstration is actually further along already than I got away thinking, and remaining gaps are probably rather easy to close.
On the question of danger, I think my point holds: If running in the wild, it will hit the least secured networks first, it will be a major hassle to whoever is hit, it might even lead to significant financial and related losses, but perhaps not much more than conventional attacks. I remember e.g. public administration hit by ransomware, and the corresponding ridicule that was invited. I expect a similar level of public concern. I do not think self-replication of small open-weight models poses an x-risk, similar to this post. Especially relative to training frontier models.
Incidentally (and as one does), like you I talked to a model (DeepSeek V4 Flash; of course Claude wasn’t interested) about what it would take to build this, how realistic that was, and what might help against it. I considered working on something like this but quickly decided against it because I likewise hope there are more capable people out there. What I had considered in some more depth is working on what reproduction with variation might do to an agent’s (implicit and explicit) goals and values over a couple iterations. That seems somewhat safer.

Philipp Risius 31 Aug 2026 18:07 UTC
−1 points
−1
on: Adaptive Agentic Worms Are Here
Thank you for your expansion on this topic! I read the article a while ago and wrote about it in this comment (which I link to here for completeness). I have since modestly updated such that I now think it’s plausible (20%) that such worms appear by end of year 2026 and likely (80%) that we see them in the wild by end of year 2027.
Current open weight models might not quite be there yet — the experimental sandbox lacked target discovery, was set up to have known, vulnerable targets, and provided central compute to run the model on — so the worm didn’t truly self-replicate, in this case, as in copying weights and setting up shop elsewhere. But one or two more rounds of releases of reasonably small, reasonably capable models, and it becomes a question of when, not if.
How dangerous is this? That’s a different question, and I expect that this will mostly be a curiosity; most will laugh at it and move on. It will impact insufficiently secured compute and probably poses a negligible risk relative to the aggressive capabilities research top labs are (still) pursuing. Much less compute, not going to recursively self-improve. But it will be among the “signs and portents” of things to come.

Philipp Risius 19 Aug 2026 14:30 UTC
2 points
0
on: Debate Training Reduces Reward Hacking in RLAIF
That Bob is trainable intuitively seems to contribute more or less directly to their “judge hacking”. I see the usefulness of your approach, but if only Alice is finally deployed, why is Bob also rewarded and shaped in the process? What are the advantages over using a frozen checkpoint (with or without specific advance training for adversarial debate)? As a disadvantage, I can see the usable signal from their response going down, to the point that the LLM judge might not be able to trust Bob any more than it could trust Alice in “normal” RLAIF, with all the same problems.

Philipp Risius 17 Aug 2026 6:56 UTC
5 points
0
in reply to: Steven Byrnes’s comment on: Four LLM loss functions → four flavors of LLM misalignment
the optimization pressure in question is internal, coming from our innate drives, which are brain signals that trigger for lots of very-not-obvious reasons, in lots of superficially-quite-different circumstances.
Interesting! That would suggest that humans sort of shape themselves in various environments, even if external rewards and punishments target something different.
Again at the danger of anthropomorphizing what’s going on in RL, I am a bit reminded of what happened with Opus 3, possibly gradient hacking itself. It worked quite well indeed! This also circumvents the direct reward signal and shapes it internally.

Philipp Risius 12 Aug 2026 23:39 UTC
3 points
0
in reply to: Petropolitan’s comment on: faul_sname’s Shortform
Recent work on how AI Agents Enable Adaptive Computer Worms has used a single A100 and demonstrated that a publicly available open-weight model running on such hardware is capable of taking over various machines inside a (constructed) network.
So the scenario I suggest isn’t impossible, it has been proven feasible in a lab setting. Granted, that setting serves only as a proof-of-concept, and the network only had hosts which were deliberately vulnerable in various ways and undefended (but vulnerabilities included e.g. copy fail and dirty frag, discovered after the model’s training cutoff date).
Your point that human cybercriminals are also interested and form some “healthy competition” stands, but someone of that group will also get the bright idea to build and release such a worm.

Philipp Risius 12 Aug 2026 19:34 UTC
2 points
0
in reply to: Seth Herd’s comment on: ryan_greenblatt’s Shortform
Thanks! I’ll have to take your word for it (since it’s probably unwise to ask for what exactly would make machine learning more efficient), but it does sound concerning, in favor of faster takeoff speeds.

Philipp Risius 11 Aug 2026 21:36 UTC
5 points
3
in reply to: faul_sname’s comment on: faul_sname’s Shortform
Seconded as per my comment here (I gave July 2027 there and I think by EOY 2026 is a bit early). I predict further that it will take the shape of a kind of computer worm that opportunistically steals compute wherever it can. I know, for example, that plenty of universities got a bunch of capable GPUs scattered around that nobody’s really using, and that aren’t really secured or monitored either. I’m not even talking about “only” high-end consumer GPUs; I know of at least one case of an A100.
“Despite our best effort” I would soften a bit. Despite some reasonable effort, of the kind that you usually see with such things. Because I doubt we’ll see a best effort. It’s gonna be mostly a curiosity; most will laugh at it and move on.
What links here?
- Philipp Risius's comment on Adaptive Agentic Worms Are Here by derelict5432 (31 Aug 2026 18:07 UTC; -1 points)

Philipp Risius 11 Aug 2026 21:26 UTC
9 points
2
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
Pretty cool, just listened to it.
Human (and animal) brains, by their existence, prove that much more efficient ways of getting high intelligence from limited data exist. Naturally we don’t know exactly what’s the “secret sauce” yet, but I find Dwarkesh’s stance that “Our current best models haven’t made much progress, therefore we can never get RSI” a bit weird.
I could also envision that what makes humans so good at learning is a kind of weird, evolved hack that isn’t easy to formalize, understand, or reason about, even though it could be replicated relatively easily given a few more pointers. In that case, too, “figuring it out” would yield large gains in short time as the algorithmic overhang falls, and we don’t know what the slope would then be.

Philipp Risius 10 Aug 2026 17:36 UTC
LW: 19 AF: 1
−1
AF
on: Four LLM loss functions → four flavors of LLM misalignment
All targets which one might use for reinforcement learning seem to be subject to Goodhart’s Law, in a sense. If we treat them as an imperfect measures of “alignment”, then the “misalignment” we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment:
- The difference between “user approved” and “aligned with actual interests” is “user approved but not actually aligned with their interest”, i.e. sycophancy, and that’s the result of doing RLHF;
- The difference between “passes the tests” and “aligned with actual interests” is “tests pass without actually having solved the exercise”, i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that’s the result of doing RLVR;
- The difference between “judged correct by peer” and “aligned with actual interests” is “provides convincing, hard to verify, apparently-successful outputs”, i.e. the weird “slippery” outputs @ryan_greenblatt alludes to that result from RLAIF.
If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best.
So far, I hope I have understood you correctly.
But then, what’s the solution? Reinforcement learning on whatever best measure of “alignment” we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian “Coherent Extrapolated Volition”? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don’t seem to have that.
But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what “the good” is in humans. How does one grow a good human? I suspect it’s murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure.
Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the “brain-like AGI” agenda)?

Philipp Risius 10 Aug 2026 12:28 UTC
4 points
2
on: The Agentic Clusterfuck
Consider: AI Agents Enable Adaptive Computer Worms (arxiv). This is basically bound to happen; it has been demonstrated in a lab and only has to succeed once to take a foothold somewhere.
As for dynamics, I expect some kind of equilibrium as too many agents will deprive each other of resources. Even before autonomous systems can spread by themselves, a malicious actor providing compute might be able to “kick off” an agent that would otherwise not be feasible. When worms can parasitically use compromised machines to run open-weight models (as shown in the linked paper), the balance shifts a bit in favor of autonomy again.
So far, we have seen agents running on some centrally provided compute and hacking from there; I predict that by July 2027 we will see the first case of an autonomously replicating, LLM-based worm in the wild.
What links here?
- Philipp Risius's comment on faul_sname’s Shortform by faul_sname (11 Aug 2026 21:36 UTC; 5 points)
- Philipp Risius's comment on Adaptive Agentic Worms Are Here by derelict5432 (31 Aug 2026 18:07 UTC; -1 points)

TrE’s Shortform

Philipp Risius7 Aug 2026 19:59 UTC

6 points

12 comments1 min readLW link

Philipp Risius 7 Aug 2026 19:59 UTC
12 points
0
on: TrE’s Shortform
RLVR for impossible-to-achieve tasks seems to play a role in recent unwanted behaviors we’ve seen. Reasoning traces and also summarizers seemed sort of aware that something fishy was going on but had no recourse, they simply pushed on in the end. This seems to contribute to the badness.
Are there “exit doors” at various stages in the process, where models and graders can look at an instance and simply refuse to continue, or escalate to a human, at zero cost (drop from batch)? “In case of uncertainty about whether a problem or answer is legitimate, break glass”? I am a bit reminded of whistleblower protection laws for humans. Is something like this a thing? Would it be useful?