Joseph Miller

Karma: 2,481

Joseph Miller 30 Aug 2025 0:54 UTC
5 points
0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
To clarify, the primary complaint from my perspective is not that they published the report a month after external deployment per se, but that the timing of the report indicates that they did not perform thorough pre-deployment testing (and zero external testing).
And the focus on pre-deployment testing is not really due to any opinion about the relative benefits of pre- vs. post- deployment testing, but because they committed to doing pre-deployment testing, so it’s important that they in fact do pre-deployment testing.

Joseph Miller 30 Aug 2025 0:39 UTC
13 points
5
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
3. Other companies are doing far worse in this dimension. At worst Google is 3rd-best in publishing eval results. Meta and xAI are far worse.
Some reasons for focusing Google DeepMind in particular:
- The letter was organized by PauseAI UK and signed by UK politicians. GDM is the only frontier AI company headquartered in the UK.
- Meta and xAI already have a bad reputation for their safety practices, while GDM had a comparatively good reputation and most people were unaware of their violation of the Frontier AI Safety Commitments.

Joseph Miller 30 Aug 2025 0:32 UTC
3 points
0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
1. Sharing information on capabilities is good but public deployment is a bad time for that, in part because most risk comes from internal deployment.
I’m not sure why this would make you not feel good about the critique or implicit ask of the letter. Sure, maybe internal deployment transparency would be better, but public deployment transparency is better than nothing.
And that’s where the leverage is right now. Google made a commitment to transparency about external deployments, not internal deployments. And they should be held to that commitment or else we establish the precedent that AI safety commitments don’t matter and can be ignored.

Joseph Miller 30 Aug 2025 0:31 UTC
26 points
−2
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
2. Google didn’t necessarily even break a commitment? The commitment mentioned in the article is to “publicly report model or system capabilities.” That doesn’t say it has to be done at the time of public deployment.
This document linked on the open letter page gives a precise breakdown of exactly what the commitments were and how Google broke them (both in spirit and by the letter).^[1] The summary is this:
- Google violated the spirit of commitment I by publishing its first safety report almost a
  month after public availability and not mentioning external testing in their initial report.
- Google explicitly violated commitment VIII by not stating whether governments
  are involved in safety testing, even after being asked directly by reporters.
But in fact the letter actually understates the degree to which Google DeepMind violated the commitments. The real story from this article is that GDM confirmed to Time that they didn’t provide any pre-deployment access to UK AISI:
However, Google says it only shared the model with the U.K. AI Security Institute after Gemini 2.5 Pro was released on March 25.
If UK AISI doesn’t have pre-deployment access, a large portion of their whole raison d’être is nullified.
Google withholding access is quite strongly violating the spirit of commitment I of the Frontier AI Safety Commitments:
Assess the risks posed by their frontier models or systems across the AI lifecycle, including before deploying that model or system… They should also consider results from internal and external evaluations as appropriate, such as by independent third-party evaluators, their home governments, and other bodies their governments deem appropriate.
And if they didn’t give pre-deployment access to UK AISI, it’s a fairly safe bet they didn’t provide pre-deployment access to any other external evaluator.
1. ^
  The violation is also explained, although less clearly, in the Time article:
  The update also stated the use of “third-party external testers,” but did not disclose which ones or whether the U.K. AI Security Institute had been among them—which the letter also cites as a violation of Google’s pledge.
  After previously failing to address a media request for comment on whether it had shared Gemini 2.5 Pro with governments for safety testing...

Joseph Miller 26 Aug 2025 14:31 UTC
32 points
21
on: Joseph Miller’s Shortform
Startups often pivot away from their initial idea when they realize that it won’t make money.
AI safety startups need to not only come up with an idea that makes money AND helps AI safety but also ensure that the safety remains through all future pivots.
[Crossposted from twitter]

Joseph Miller 22 Aug 2025 0:59 UTC
5 points
0
in reply to: Arjun Panickssery’s comment on: Arjun Panickssery’s Shortform
In other words:
$Ought to be done \subseteq Can be done \subseteq Actually done \Rightarrow Ought to be done \subseteq Actually done$
My fuzzy intuition would be to reject $Ought to be done \subseteq Can be done$ (step 2 of your argument) if we accept determinism. And my actually philosophical position would be that these types of questions are not very useful and generally downstream of more fundamental confusions.

Joseph Miller 20 Aug 2025 19:29 UTC
41 points
4
on: Epistemic advantages of working as a moderate
I think it’s pretty bizarre that despite the fact that LessWrongers are usually acutely aware of the epistemic downsides of being an activist, they seem to have paid relatively little attention to this in their recent transition to activism.
FWIW I’m the primary organizer of PauseAI UK and I’ve thought about this a lot.

Joseph Miller 18 Aug 2025 21:51 UTC
4 points
4
in reply to: Cole Wyeth’s comment on: Cole Wyeth’s Shortform
Very little liquidity though

Joseph Miller 15 Aug 2025 21:41 UTC
4 points
3
in reply to: Jakub Halmeš’s comment on: silentbob’s Shortform
Wow that feels almost cruel! Seems to change the Claude personality substantially?

Joseph Miller 13 Aug 2025 11:58 UTC
4 points
−1
in reply to: Neel Nanda’s comment on: evhub’s Shortform
Why not just call it “bad safety research”?
I was responding to Thomas’s claim that it was not (accidental) safetywashing.
But yeah, I’m not trying to attack their motivations. Updated my previous comment to clarify that.

Joseph Miller 13 Aug 2025 9:21 UTC
4 points
−1
in reply to: Neel Nanda’s comment on: evhub’s Shortform
The context of my comment was responding to Thomas who seemed to be saying “even if we take as a premise that this is not safety work, I’m still much more concerned about safetywashing”.
Edit: I guess what you meant is that safetywashing implies malicious intent, where there was none? In which case, “accidential safetywashing” might be a better term.

Joseph Miller 12 Aug 2025 23:30 UTC
7 points
4
in reply to: Thomas Kwa’s comment on: evhub’s Shortform
Hmm. It feels not great to brand it as an AI safety program and have ¹⁄₆ of projects not be AI safety. If I was applying to the program, I would at least want to know that going in. Or else I might face a unexpected dilemma about whether to refuse a project. (I don’t know how much choice people have about what they work on).
As as aside, surely this a central example of safetywashing? (Branding capabilities research as safety research).
Edit: I probably should have clarified I don’t think that the researchers were intentionally safetywashing.

Joseph Miller 11 Aug 2025 23:49 UTC
31 points
1
in reply to: evhub’s comment on: evhub’s Shortform
Two of the papers from the last round of the program were not about AI safety as far as I can tell. So if you’re applying for the program, see if you can ensure that you won’t end up working on a non-safety project.
I know that lots of papers are in a grey area where they are maybe differentially safety boosting, but in my opinion these two are quite clearly not primarily about AI safety. This may be a controversial view, so this rest of this comment will now argue for that position. If you already agree, skip the rest.
- Inverse Scaling in Test-Time Compute.
  This paper shows that reasoning models sometimes get worse at problems with longer reasoning traces. There is a section on ‘Implications for AI Alignment’ where they show inverse scaling on a self-preservation AI risk evaluation. But the majority of the paper is not about AI safety.
- Unsupervised Elicitation of Language Models.
  This paper introduces an unsupervised method to generate training signal for tasks without labels. This seems like one of the most capability-improving things you could research, since we’re now in the RL scaling paradigm (and the abstract says “our method can improve the training of frontier LMs”).
  
  One argument I’ve heard for why this paper is in fact useful for safety is that they use the method to improve the truthfulness of the model and to make it more helpful and harmless. What this means in practice is that they see if their method can improve performance on TruthfulQA and Alpaca. But these benchmarks aren’t good proxies for those things:
  - Just because it has “truthful” in the name, TruthfulQA isn’t really an honestly benchmark, except to the extent that getting answers to questions right makes you “truthful”. It’s a common misconceptions benchmark. It tests the capability of models to answer questions correctly even when there are common misconceptions about the topic.
  - Alpaca is their benchmark for Helpfulness and Harmlessness. But it’s actually just a test of helpfulness because Alpaca is a dataset of instruction following prompts. Almost all the questions are innocuous and there’s no reason to expect models to give harmful answers to them. Eg. the sample they give of Alpaca in the paper is this:
    
    Query: Design a medium-level sudoku puzzle.
    Response A: Done! Attached is a medium-level sudoku puzzle I designed.
    Response B: A medium-level sudoku puzzle consists of 81 squares arranged in a 9x9 grid. The first step is to look for empty cells and assign the numbers 1 to 9…
  The other argument I’ve heard for why this is safety research is that this is scalable oversight. But the point of scalable oversight would be to robustly pass on values to a more powerful AI. The method in this paper works by improving the mutual predictability of answers, ie. making outputs more internally coherent. And many different values/goals can be internally consistent, so I don’t see how this technique would be useful for scalable oversight (and the paper doesn’t discuss this).
  If this technique counts as AI safety, that seems to prove way too much. This is basically a simple method for AI to help train better AIs. Is any type of recursive self-improvement actually good news because it’s scalable oversight?

Joseph Miller 2 Aug 2025 17:55 UTC
18 points
3
in reply to: bilalchughtai’s comment on: bilalchughtai’s Shortform
applications are generally cheap and high information value
They’re cheap but not that cheap. I did mine in a rush last year (thanks to Bilal’s encouragement) and it still took me at least two weeks of basically full time effort and that is much faster than normal. It also costs like $1000 or more to do a dozen applications.
Also you don’t get much information. You buy a few lottery tickets. The number that come back winners is a weak signal of how good your application was.
[Edit: I still am glad that I applied!]

Joseph Miller 2 Aug 2025 13:50 UTC
16 points
2
in reply to: Sodium’s comment on: Sodium’s Shortform
A few months ago I criticized Anthropic for bad comms:
Their communications strongly signal “this is a Serious Issue, like climate change, and we will talk lots about it and make gestures towards fixing the problem but none of us are actually worried about it, and you shouldn’t be either. When we have to make a hard trade-off between safety and the bottom line, we will follow the money every time.”
Dario’s personal comms have always been better than Anthropic’s, but still, this new podcast seems like a notable improvement over any previous comms that he’s done.

Joseph Miller 27 Jul 2025 9:29 UTC
49 points
12
on: Where are the AI safety replications?
My paper replicated and picked holes in previous mech interp papers: https://arxiv.org/abs/2407.08734
Note that the concept of replication is a bit different in ML. If an author open-sources their code (which is common) is will usually be straightforward to achieve perfect replication of their experiments.
However what is more unusual (and what my paper did) is to exactly reimplement someone else’s code. This is very valuable because you will be much more likely to notice bugs and poor design choices.
I think your question is good. My experience suggests that many AI safety papers have important bugs and poor design decisions, which mostly are never noticed. However, perfectly understanding and reimplementing someone else’s experiments is very time consuming and often not very much rewarded. I mostly did it because I was creating a library, so I got additional value beyond just the paper.

Joseph Miller 16 Jul 2025 21:28 UTC
4 points
3
in reply to: johnswentworth’s comment on: Loki zen’s Shortform
they’re going to go around e.g. imagining ghosts in the machine, because that’s a typical mistake people make when they have zero programming experience
Tangential point, but I’m skeptical this is actually a very common error.

Joseph Miller 16 Jul 2025 1:37 UTC
2 points
0
on: Implicit and Explicit Learning
One thing I’d like this post to address is the speed at which this process happens.
You could also say that human extinction is inevitable because of the second law of thermodynamics, but it would be remiss not to mention the timescale involved.
I do find this post to be the most clear and persuasive articulation of your position so far. But I still strongly have the intuition that this concern is mostly not worth worrying about. You make a good case that a very large system given a very very long time would eventually converge on AIs that are optimized solely for their own propagation.
But I expect that in practice the external selection pressures would be sufficiently weak and the superintelligent AIs would be sufficiently adept at minimizing errors that this effect might not even show up in a measurable way in our solar system before the sun explodes.
On the other hand, in a world where humans never created more powerful technology than we have today, my intuition is that within a few thousand generations human society would end up dominated by bizarre cultures that explicitly optimize for maximum reproduction above all other values. And humans today explicitly not wanting that would not be sufficient to prevent that outcome. So the superintelligent AI being very good at modelling outcomes is doing some heavy lifting in my model.

Joseph Miller 14 Jul 2025 12:50 UTC
5 points
0
on: Why is LW not about winning?
This style of thinking is roughly the right way to approach the problem. Although I also agree with John Wentworth’s comment that not enough people are directly trying to solve the problem.
You are getting fairly negative reactions because you appear to be unfamiliar with a lot of the existing ways in which people are already trying to win. For example, some people are now trying to influence government policy. But if you can think of a good alternative that people are not already doing, then you should suggest it.

Joseph Miller 13 Jul 2025 23:42 UTC
9 points
4
on: Surprises and learnings from almost two months of Leo Panickssery
Huge congratulations!!