StellaAthena

Karma: 722

StellaAthena 9 Sep 2026 2:46 UTC
1 point
0
in reply to: Rauno Arike’s comment on: How concerned should we be about Astra’s recurrent architecture?
My update was about the difficulty of scheming, not about the extent to which current models are able to get around oversight methods: I previously thought that in order to subvert oversight, models would have to do elaborate unverbalized thinking about monitoring systems, the company and context in which they’re being deployed, etc, while now, it seems plausible that they just need to come up with a strategy to spoof tool calls without verbalizing their thinking about that.
Ah I see. This isn’t surprising to me at all, so I guess I discounted it as a possible factor in your analysis.

StellaAthena 4 Sep 2026 21:28 UTC
3 points
1
on: How concerned should we be about Astra’s recurrent architecture?
One of the biggest updates I made from the HuggingFace incident is that the opaque reasoning depth required for subverting oversight isn’t as large as I had previously thought. A rogue LLM trying to break containment doesn’t have to come up with a full plan to get around the monitors and human researchers in a single forward pass; it just has to find a way to spoof tool calls and use the spoofed tool calls to gain additional unmonitored thinking time and/or cover its tracks.
I think this is the wrong conclusion to draw from the HF-OAI incident. I don’t see any reason to believe that the models subverted oversight at all, just that OpenAI did not monitor their agents at all. Tomek Korbak told me this on twitter explicitly. Reproducing the most relevant parts of the exchange:
Tomek: we are now monitoring RL runs and evals too!
Stella: Why weren’t you doing so before? OpenAI has been publishing research about CoT monitoring for well over a year.
Tomek: I think we (as an org, individual opinions differed) mistakenly considered our sandboxing good enough to prevent models during evals from causing harm in the outside world
[...]
Tomek: At the end of the day safety teams have finite bandwidth and need to heavily prioritize what control measures to implement, given imperfect knowledge. i agree with you that frontier labs should be monitoring all agentic workloads of sufficiently capable models.
He went on to explain that previously CoT monitoring was only implemented for things run through the Codex application, but that some (or most?) internal evals work and none of the RL training is run through Codex and so they didn’t have the infrastructure to do that monitoring because they didn’t think it was very important. In other conversations Tomek and others at OpenAI have said things that to me indicate the models involved in the incident weren’t really being monitored in any meaningful sense and that OpenAI ignored things I would consider red flags strong enough to shut everything down on multiple occasions.
I don’t want to derail this too much with my personal views of OpenAI from a safety POV, but I don’t think the update your describing here is justified by what (I currently think) happened.

StellaAthena 10 Aug 2026 22:12 UTC
7 points
3
on: R-lens: Making J-lens More Faithful on Early Layers
One reason we didn’t really follow up on the tuned lens was that it’s really hard to tell if you have the right way of measuring “what the model thinks the answer is after layer K.” The approach we took to this was based in epistemology and assumed that models followed several intuitively plausible epistemological guidelines, and measured the extent to which the tune lends improved those guidelines over the logit lens. That evaluation is at best correlational, but I feel uncertain about whether other metrics even capture anything worth optimizing. After all, you can improve all naive performance metrics by reporting “what the model thinks the answer is after layer K+2″ in place of “what the model thinks the answer is after layer K.”
I was frustrated that many of the metrics in the J-Lens paper were either layer invariant by construction or would obviously face this pitfall.

What do you think regarding what the right way to do this kind of evaluation is / how concerned you are with this issue. You gesture at this with the discussion of probe upper bounds, but take those “upper bounds” with such a little seriousness that you’re comfortable with the fact that the R-Lens outperforms it sometimes.

StellaAthena 27 Jul 2026 19:00 UTC
3 points
0
in reply to: StanislavKrym’s comment on: An OpenAI model left notes about how to evade containment; we need more details
I’ve tried to edit my message for clarity (the cultural comments were meant to apply to OAI, not Anthropic) but we know that whatever safeguards Anthropic has are insufficient because the Mythos model regularly broke out of their sandboxes and gained access to the open internet during development and testing. According to the system card, about 20% of attempts by the model to do so were successful!

StellaAthena 27 Jul 2026 12:26 UTC
5 points
0
in reply to: StellaAthena’s comment on: An OpenAI model left notes about how to evade containment; we need more details
their refusal to implement adequate safeguards
This received a citation needed tag and I’m not sure why. The fact that they haven’t implemented adequate safeguards is evident from the fact that their model escaped, attacked HF, and OpenAI didn’t even notice for a week.

StellaAthena 27 Jul 2026 12:06 UTC
6 points
3
in reply to: avturchin’s comment on: An OpenAI model left notes about how to evade containment; we need more details
Their model broke out of their sandboxes, committed offensive cyber operations against a billion dollar company, and OpenAI only noticed after a week passed. IMO it kind of doesn’t matter whether or not they’re “compromised” or “bad at security“ or what. You should assume that OpenAI does not have the ability to sandbox their models or to monitor their behavior meaningfully.

StellaAthena 26 Jul 2026 18:00 UTC
75 points
10
on: An OpenAI model left notes about how to evade containment; we need more details
There have been loss of control and models escaping sandboxing incidents at both OpenAI and Anthropic for over a year. They’ve publicly disclosed some of them (e.g., the latest system cards have stories about this from both companies) and some of them have been leaked within the community.
I know for a fact that OpenAI and Anthropic have been warned by internal and external experts that their security infrastructure for testing misaligned agentic coding agents is insufficient because I have personally told them that as have several former staff members. I had a debate with the head of security (?) at Anthropic at DEF CON in 2023 where I was pressing him on the fact that Anthropic wasn’t building air gapped networks.
It is absolutely within OpenAI and Anthropic’s ability to build an air gapped system for developing and testing these models. It seems likely to me that an internal GitHub clone and a moderately sized intranet would be sufficient to test the vast majority of agentic and web-enabled capabilities on such a platform. I think that their refusal to implement adequate safeguards is unjustifiable, but based on conversations with current and former safety and security researchers at OpenAI it seems like a company culture and lack of executive leadership buy-in problem that’s very hard to change without massive external pressure.
One issue that seems very worrisome today is that back in like 2023 an OpenAI security researcher was telling me about how they were unable to get OpenAI staff to stop using unreleased and inadequately tested models to develop internal infra, including internal monitoring tooling. I wish I remembered the person‘s name, I’d love to follow up.
Note: the final paragraph is an anecdote was told me with the expectation that I not disclose it to anyone else. Given recent events I view breaking that trust as akin to being a whistleblower. If the person who told me that is reading this, I’m sorry. Before last week I never disclosed it to anyone.

StellaAthena 24 Mar 2026 5:15 UTC
2 points
0
on: AICRAFT: DARPA-Funded AI Alignment Researchers — Applications Open
You receive a final report with documented results. Promising pilots are recommended to DARPA for a 3-month extended engagement, contingent on your availability.
What about data and code? Suppose a project looks like “here is a FT algorithm that will increase controllability” or “if you curate your dataset in this way the model won’t learn to do such and such.” There is a significant amount of research output that would be very valuable and isn’t found in a report.
What does the IP agreement look like? Does AE Studio own the IP? Do I? Does the USG? What hurdles would need to be passed to publish the results?

StellaAthena 18 Jan 2026 19:08 UTC
5 points
0
in reply to: Niki Dupuis’s comment on: ryan_greenblatt’s Shortform
Training a 6B model for 500B tokens costs about 20k USD. That number increases linearly with model size and # of tokens, and decreases with the amount of money you have. Work like this is super doable, especially at large labs.

StellaAthena 26 Jan 2024 1:20 UTC
5 points
4
in reply to: Haiku’s comment on: RAND report finds no effect of current LLMs on viability of bioterrorism attacks
This is one area where I hope the USG will be able to exert coercive force to bring companies to heel. Early access evals, access to base models, and access to training data seem like no-brainers from a regulatory POV.

StellaAthena 25 Jan 2024 21:45 UTC
4 points
3
in reply to: seed’s comment on: RAND report finds no effect of current LLMs on viability of bioterrorism attacks
I think you’re misrepresenting Gwern’s argument. He’s arguing that terrorists are not optimizing for killing the most people. He makes no claims about whether terrorists are scientifically incompetent.

StellaAthena 25 Jan 2024 21:38 UTC
9 points
0
in reply to: habryka’s comment on: RAND report finds no effect of current LLMs on viability of bioterrorism attacks
Thanks! I like your title more :)

StellaAthena 16 Jan 2024 2:32 UTC
8 points
−5
in reply to: kave’s comment on: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

It seems helpful to me if policy discussions can include phrases like “the evidence suggests that if the current ML systems were trying to deceive us, we wouldn’t be able to change them not to”.

I take this as evidence that TurnTrout’s fears about this paper are well-grounded. This claim is not meaningfully supported by the paper, but I expect many people to repeat it as if it is supported by the paper.

StellaAthena 31 Dec 2023 13:18 UTC
3 points
2
in reply to: habryka’s comment on: Integrity in AI Governance and Advocacy
We ended up talking about this in DMs, but to gist of it is:
Back in June Hoagy opened a thread in our “community research projects” channel and the work migrated there. Three of the five authors of the [eventual paper](https://arxiv.org/abs/2309.08600) chose to have EleutherAI affiliation (for any work we organize with volunteers, we tell them they’re welcome to use an EleutherAI affiliation on the paper if they like) and we now have an entire channel dedicated to future work. I believe Hoagy has two separate paper ideas currently in the works and over a half dozen people working on them.

StellaAthena 31 Dec 2023 3:59 UTC
4 points
0
in reply to: habryka’s comment on: Integrity in AI Governance and Advocacy
Ooops. It appeared that I deleted my comment (deeming it largely off-topic) right as you were replying. I’ll reproduce the comment below, and then reply to your question.

I separately had a very weird experience with them on the Long Term Future Fund where Conor Leahy applied for funding for Eleuther AI. We told him we didn’t want to fund Eleuther AI since it sure mostly seemed like capabilities-research but we would be pretty interested in funding AI Alignment research by some of the same people. He then confusingly went around to a lot of people around EleutherAI and told them that “Open Phil is not interested in funding pre-paradigmatic AI Alignment research and that that is the reason why they didn’t fund Eleuther AI”. This was doubly confusing and misleading because Open Phil had never evaluated a grant to Eleuther AI (Asya who works at Open Phil was involved in the grant evaluation as a fund member, but nothing else), and of course the reason he cited had nothing to do with the reason we actually gave. He seems to have kept saying this for a long time even after I think someone explicitly corrected the statement to

While this anecdote is largely orthogonal to the broader piece, I remembered that this existed randomly today and wanted to mention that Open Phil has recommended a 2.6 M/3 years grant to EleutherAI to pursue interpretability research. It was a really pleasant and very easy experience: Nora Belrose (head of interpretability) and I (head of everything) talked with them about some of our recent and on-going work such as Eliciting Latent Predictions from Transformers with the Tuned Lens, Eliciting Latent Knowledge from Quirky Language Models, and Sparse Autoencoders Find Highly Interpretable Features in Language Models very interesting and once they knew we had shared areas of interest it was a really easy experience.

I had no vibes along the lines of “oh we don’t like EleutherAI” or “we don’t fund pre-paradigmatic research.” It was a surprise to some people at Open Phil that we had areas of overlapping interest, but we spent like half an hour clarifying our research agenda and half an hour talking about what we wanted to do next and people were already excited.

StellaAthena 2 Nov 2023 22:55 UTC
9 points
15
in reply to: Chris_Leong’s comment on: Propaganda or Science: A Look at Open Source AI and Bioterrorism Risk

I agree that a control group is vital for good science. Nonetheless, I think that such an experiment is valuable and informative, even if it doesn’t meet the high standards required by many professional science disciplines. I believe in the necessity of acting under uncertainty. Even with its flaws, this study is sufficient evidence for us to want to enact temporary regulation at the same time as we work to provide more robust evaluations.

But… this study doesn’t provide evidence that LLMs increase bioweapon risk.

StellaAthena 9 Oct 2023 16:09 UTC
1 point
0
in reply to: ryan_greenblatt’s comment on: Introducing the Center for AI Policy (& we’re hiring!)
It doesn’t let the government institute prior restraint on speech.

StellaAthena 8 Sep 2023 20:37 UTC
4 points
3
in reply to: Thomas Larsen’s comment on: Introducing the Center for AI Policy (& we’re hiring!)

So far, I’m confident that our proposals will not impede the vast majority of AI developers, but if we end up receiving feedback that this isn’t true, we’ll either rethink our proposals or remove this claim from our advocacy efforts.

It seems to me like you’ve received this feedback already in this very thread. The fact that you’re going to edit the claim to basically say “this doesn’t effect most people because most people don’t work on LLMs” completely dodges the actual issue here, which is that there’s a large non-profit and independent open source LLM community that this would heavily impact.

I applaud your honestly in admitting one approach you might take is to “remove this claim from our advocacy efforts,” but am quite sad to see that you don’t seem to care about limiting the impact of your regulation to potentially dangerous models.

StellaAthena 30 Aug 2023 5:52 UTC
14 points
0
in reply to: Holly_Elmore’s comment on: Introducing the Center for AI Policy (& we’re hiring!)
Nora didn’t say that this proposal is harmful. Nora said that if Zach’s explanation for the disconnect between their rhetoric and their stated policy goals is correct (namely that they don’t really know what they’re talking about) then their existence is likely net-harmful.

That said, yes requiring everyone who wants to finetune LLaMA 2 get a license would be absurd and harmful. la3orn and gallabyres articulate some reasons why in this thread.

Another reason is that it’s impossible to enforce, and passing laws or regulations and then not enforcing them is really bad for credibility.

Another reason is that the history of AI is a history of people ignoring laws and ethics so long as it makes them money and they can afford to pay the fines. Unless this regulation comes with fines so harsh that they remove all possibility of making money off of models, OpenAI et al. won’t be getting licenses. They’ll just pay the fines while small scale and indie devs (who allegedly the OP is specifically hoping to not impact) screech their work to a halt and wait for the government to tell them it’s okay for them to continue to do their work.

Also, such a regulation seems like it would be illegal in the US. While the government does have wide latitude to regulate commercial activities that impact multiple states, this is rather specifically a proposal that would regulate all activity (even models that never get released!). I’m unaware of any precedent for such an action, can you name one?

StellaAthena 30 Aug 2023 3:40 UTC
51 points
28
on: Introducing the Center for AI Policy (& we’re hiring!)

CAIP is also advised by experts from other organizations and is supported by many volunteers.

Who are the experts that advise you? Are claims like “our proposals will not impede the vast majority of AI developers” vetted by the developers you’re looking to avoid impacting?