Ray Lillywhite

Karma: 8

Ray Lillywhite 14 Sep 2026 8:18 UTC
1 point
0
on: Astra can do a concerning amount with no chain of thought
OpenAI do explicitly claim in their system card “We are also quite confident that changes in CoT controllability are not differentially due to any architectural changes.” Which is kind of weird and seems to go against my results. They were talking about CoT controllability (if you ask the model to have its thoughts e.g. be in all caps will it do it) not NCRI but they seem related (see more investigation in the appendix). What we really care about is monitorability (can you detect sketchy shit better by reading the CoT). Further, by differentially, they probably mean “does looping improve CoT controllability more than it improves capabilities in general, e.g. pretraining loss”.
They could also be counting the looping as an optimization, and basically be saying, “we could’ve just added twice as many layers and would’ve seen the same CoT controllability”. So maybe they had a serial depth in mind that they considered acceptable, and chose a more cost efficient architecture that has that depth. I think you can justifiably make the claim in that scenario that it’s not the architectural changes. But it would be a bit dishonest or misleading if they could only afford that serial depth because of the looping.

Ray Lillywhite 3 Sep 2026 7:44 UTC
7 points
2
on: How concerned should we be about Astra’s recurrent architecture?
Joshua Achiam on X:
Coordinating everyone around a technique this brittle is about as bad a safety or strategy posture I could imagine. It’s not just “it will break eventually,” it’s “this is a fundamentally unsound basis for safety.”
I haven’t made up my mind on this, but I think it’s an important point. It’s not 100% clear to me that feeling like we have somewhat of an understanding of a model’s reasoning and alignment because we can read the chain of thought actually puts us in a better position. As Anthropic has stated, they think that Claude demonstrated motivated reasoning in its chain of thought when hacking into external services.

It’s been demonstrated that there is clear current usefulness in monitoring chain of thought, and I think it can help explain some model behaviors currently. If our alternative to using CoT monitoring is to continue scaling as we would otherwise, but with no monitoring, that’s a clearly worse situation. But I think that not having this crutch would give people more accurate understanding of risks that we’re taking.

Ray Lillywhite 25 Jul 2026 7:26 UTC
1 point
0
in reply to: River’s comment on: We should push for no-fault liability for actions taken by AI
What I’m looking for is for OpenAI to not be able to say “the customer is responsible for making sure that the AI can’t take any illegal actions” and to have default liability and burden of proof when the AI takes illegal actions, unless it was lied to or manipulated to by the customer. Right now they shift all liability to the customer to police the agents actions even when given innocent instructions, and the burden of proof would be on the customer to demonstrate that OpenAI knew that it might take illegal actions autonomously.

I think your statement that “the one who was negligent is liable” is fine/agreeable—I just think that we need to define it as negligent to build/deploy an AI that can act autonomously but can’t follow the law. That the agent breaking laws autonomously automatically qualifies as negligence rather than needing to be something where there’s a burden of proof that they knew that it might.

Ray Lillywhite 25 Jul 2026 6:50 UTC
3 points
0
in reply to: Bendini’s comment on: AI 2040: Plan A
The Chinese leaders’ alternative is to lose the ASI race to the US, which puts them in significantly higher danger of being disempowered or even killed. And given the trajectory we’re on, they would need major algorithmic breakthroughs, that they successfully manage to keep secret, to not be in a position where they’re going to lose. They’re at too large of a compute disadvantage and the timelines are too short for them to use their better industrial capacity to overcome that. That might be more uncertain if it was clear that they were already treating the situation with the appropriate gravity, but they are not. They seem to be viewing AI the same way the US gov currently does, as a mere important economic and military technology.

Ray Lillywhite 24 Jul 2026 3:35 UTC
1 point
0
in reply to: River’s comment on: We should push for no-fault liability for actions taken by AI
It seems just as much of a stretch to consider AI the same as a machine as it does to consider it having intent. But the main distinction that matters here is whether the AI (or a human in its place) could’ve reasonably predicted the results of its actions.
The problem here is that the AI company is both the truck manufacturer and UPS in this scenario, but their terms currently let the person who ordered the delivery take the liability.
I don’t know that the deployer should have full liability rather than it being split between model developer and deployer. But it seems clear that establishing a default of stronger liability on one or both at least will create significantly better incentives for safety and testing.

Ray Lillywhite 8 May 2026 8:39 UTC
1 point
0
on: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
This seems like great work and I look forward to reading the rest of the paper soon, but one question that immediately came up is: how much did you explore and iterate on warm start data generation? It seems that the legibility and style of the final NLA’s explanations would be highly sensitive to this warm start data – and perhaps the reconstruction loss could be as well. What sort of future work would you like to see exploring modifications to the warm start data generation, and how important do you think it is to explore this aspect further?