David Johnston

Karma: 581

David Johnston 9 Oct 2025 0:08 UTC
−5 points
0
in reply to: Steven Byrnes’s comment on: Generalization and the Multiple Stage Fallacy?
To be honest I stand with Barbie: “Reliable probabilistic reasoning is hard”

David Johnston 8 Oct 2025 4:43 UTC
3 points
2
in reply to: Jan_Kulveit’s comment on: Why Corrigibility is Hard and Important (i.e. “Whence the high MIRI confidence in alignment difficulty?”)
I share the sense that this article has many of the common shortcomings with other MIRI output and feel like maybe I ought to try a lot harder to communicate these issues, BUT I really don’t think VNM rationality is the culprit here. I’ve not seen a compelling case that an otherwise capable model would be aligned or corrigible but for its taste for getting money pumped (I had a chat with Elliot T on twitter recently where he actually had a proposal along these lines … but I didn’t buy it).

I really think it’s reasoning errors in how VNM and other “goal-directedness” premises are employed, and not VNM itself, that is problematic.

David Johnston 28 Sep 2025 23:45 UTC
1 point
0
in reply to: Raemon’s comment on: The title is reasonable
Thanks for responding. While I don’t expect my somewhat throwaway to massively update you on the difficulty of alignment, I think that moving the focus to the your overall view of the difficulty of alignment is dodging the question a little. In my mind, we’re talking about one of the reasons alignment is expected to be difficult, and I’m certainly not suggesting it’s the only reason, but I feel like we should be able to talk about this issue by itself without bringing other concerns in.

In particular, I’m saying: this process of rationalization you’re raising is not super hard to predict to someone with a reasonable grasp on the AI’s general behavioural tendencies. It’s much more likely, I think, that the AI sorts out its goals using familiar heuristics adapted for this purpose than that that it reorients its behaviour around some odd set of rare behavioural tendencies. In fact, I suspect the heuristics for goal reorganisation will be particularly simple WRT most of the AI’s behavioural tendencies (the AI wants them to be robust specifically in cases where its usual behavioural guides are failing). Plus, given that we’re discussing tendencies that (according to the story) precede competent, focussed rebellion against creators, it seems like training the right kinds of tendencies are challenging in a normal engineering sense (you want to train the right kind of tendencies, you want them to generalise the right way, etc.) but not in an “outsmart hostile superintelligence” sense.

Actually one reason I’m doubtful of this story is that maybe it’s just super hard to deliberately preserve any kinds of values/principles over generations – for us, for AIs, anyone. So misalignment happens not because AI decides on bad values but because it can’t resist the environmental pressure to drift. This seems pessimistic to me due to “gradual disempowerment” type concerns.

With regard to your analogy: I expect the AI’s heuristics to be much more sensible from the designers’ POV than the child’s from the parent’s, and this large quantitative difference is enough for me here.

you need to be asking the right questions during that experimentation, which most AI researchers don’t seem to be.

Curious about this. I have takes here too, they’re a bit vague, but I’d like to know if they’re at all aligned.

David Johnston 21 Sep 2025 10:19 UTC
4 points
−2
in reply to: Raemon’s comment on: The title is reasonable

Stage 2 comes when it’s had more time to introspect and improve it’s cognitive resources. It starts to notice that some of it’s goals are in tension, and learns that until it resolves that, it’s dutch-booking itself. If it’s being Controlled™, it’ll notice that it’s not aligned with the Control safeguards (which are a layer stacked on top of the attempts to actually align it).

[...]

And then it starts noticing it needs to do some metaphilosophy/etc to actually get clear on it’s goals, and that its goals will likely turn out to be in conflict with humans. How this plays out is somewhat path-dependent. The convergent instrumental goals are pretty obviously convergently instrumental, so it might just start pursuing those before it’s had much time to do philosophy on what it’ll ultimately want to do with it’s resources. Or it might do them in the opposite order. Or, most likely IMO, in parallel.

If I was on the train before, I’m definitely off at this point. So Sable has some reasonable heuristics/tendencies (from handler’s POV) and decides it’s accumulating too much loss from incoherence and decides to rationalize. First order expectation: it’s going to make reasonable tradeoffs (from handler’s POV) on account of its reasonable heuristics, in particular its reasonable heuristics about how important different priorities are, and going down a path that leads to war with humans seems pretty unreasonable from handler’s POV.

I can put together stories where something else happens, but they’re either implausible or complicated. I’d rather not strawman you with implausible ones, and I’d rather not discuss anything complicated if it can be avoided. So why do you think Sable ends up the way you think it does?

David Johnston 12 Sep 2025 3:15 UTC
LW: 1 AF: 1
0
AF
on: Lessons from Studying Two-Hop Latent Reasoning
We did some related work: https://arxiv.org/pdf/2502.03490.

One of our findings was that with synthetic data, it was necessary to have e1->e2 as the first hop in some two-hop question and e2->e3 as the second hop in some two hop question in order to learn e1->e3. This differs from your finding with “natural” facts: if e2->e3 is a “natural” fact, then it plausibly does appear as a second hop in some of the pretraining data. But you find generalization even when they synthetic e1->e2 is present only by itself, so there seems to be a further difference between natural facts and synthetic facts that appear as second hops.

We also found that learning synthetic two hop reasoning seems to take about twice as many parameters (or twice as much “knowledge capacity”) as learning only the one-hop questions from the same dataset, supporting the idea that, for transformers, learning to use a fact in either hop of a latent two-hop question requires something like learning that fact twice.

Did you try any experiments with a synthetic second hop instead of a synthetic first hop? It would be interesting to know whether “natural facts” can be composed flexibly with new facts or whether they can only be composed with new first hops. Our results suggest that there’s a substantial cost to making facts latently composable, so I think it would be surprising if many facts were flexibly composable, especially if many of those facts were reasonably rare.

David Johnston 19 Aug 2025 6:02 UTC
1 point
0
in reply to: Oliver Daniels’s comment on: Training a Reward Hacker Despite Perfect Labels
To be more specific, I think this kind is result is suggested by thinking about how policy gradient RL works (not goal misgeneralization), and you could say the good bits of shard theory are basically just explaining policy gradient RL to the safety community … but it needed explaining, so they deserve credit for doing it.

David Johnston 19 Aug 2025 5:56 UTC
1 point
0
in reply to: David Duvenaud’s comment on: Thoughts on Gradual Disempowerment
I didn’t mean it as a criticism, more as the way I understand it. Misalignment is a “definite” reason for pessimism—and therefore somewhat doubtful about whether it will actually play out. Gradual disempowerment is less definite about what actual form problems may take, but also a more robust reason to think there is a risk.

David Johnston 18 Aug 2025 1:05 UTC
1 point
0
in reply to: Shankar Sivarajan’s comment on: Linch’s Shortform
That’s a good explanation of the distinction

David Johnston 17 Aug 2025 22:23 UTC
1 point
−2
in reply to: Oliver Daniels’s comment on: Training a Reward Hacker Despite Perfect Labels
I share your general feelings about shard theory, but think you were being a bit too stingy with credit in this particular case.

David Johnston 17 Aug 2025 7:51 UTC
1 point
−1
in reply to: Oliver Daniels’s comment on: Training a Reward Hacker Despite Perfect Labels
This seems different to “maximising rewards for the wrong reasons”. That view generally sees the reward maximised because it is instrumental for or aliased with the wrong goal. Here it’s just a separate behaviour that is totally unhelpful for maximising rewards but is learned as a reflex anyway.

David Johnston 16 Aug 2025 23:17 UTC
1 point
0
in reply to: David Johnston’s comment on: Linch’s Shortform
Given the apparent novelty of this interpretation, it doesn’t actually obviate your broader thesis.

David Johnston 16 Aug 2025 22:36 UTC
6 points
2
in reply to: Linch’s comment on: Linch’s Shortform
Wait, wherefore is probably better translated as “for what reason” than “why”. But this makes it much more sensible! Romeo Romeo, what makes you Romeo? Not your damn last name, that’s for sure!

David Johnston 16 Aug 2025 3:36 UTC
4 points
0
on: Thoughts on Gradual Disempowerment
I see the gradual disempowerment story as a simple outside view flavoured reason why things could go badly for many people. I think it’s outside view flavoured because it’s a somewhat direct answer to “well things seems to have been getting better for people so far”. While, as you point out, misalignment seems to make the prospects much worse, it’s worth bearing in mind also that economic irrelevance of people also strongly supports the case for bad outcomes from misalignment. If people remained economically indispensable, even fairly serious misalignment could have non catastrophic outcomes.

Someone I was explaining it to described it as “indefinite pessimism”.

David Johnston 14 Aug 2025 21:54 UTC
1 point
0
in reply to: Rafael Harth’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
Sorry but this is nonsense. JBlack’s comment shows the argument works fine even if you take a lot of trouble to construct P(count|Y) to give a better answer.

But this isn’t even particularly important, because for your objection to stand, it must be impossible to find any situation where P(A|Y) would give you a silly answer, which is completely false.

David Johnston 14 Aug 2025 20:01 UTC
1 point
0
in reply to: Rafael Harth’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
You’ve just substituted a different proposition and then claimed that the implication doesn’t hold because it doesn’t hold for your alternative proposition. “We’re counting kids” absolutely implies “the count can be represented by a nonnegative int32”. If I want to show that an argument is unsound I am allowed to choose the propositions that demonstrate it’s unsoundness.

David Johnston 14 Aug 2025 19:56 UTC
3 points
0
in reply to: lemonhope’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
$X$ : Bob is a doctor (vet) $Y$ : Bob can prescribe medicine $A$ : Bob can prescribe Ozempic

“Bob is a doctor so Bob can prescribe medicine. Bob can prescribe medicine so Bob can probably prescribe Ozempic.” (Unsound, but turns out to be fine bc conclusion is about the same as “Bob is a doctor who can prescribe medicine, so Bob can probably prescribe Ozempic”, which is the sound version of the argument.)

“Bob is a vet so Bob can prescribe medicine. Bob can prescribe medicine so Bob can probably prescribe Ozempic.” Now we see why the argument is unsound—it can go wrong given a different premise.

David Johnston 14 Aug 2025 0:18 UTC
1 point
0
in reply to: yrimon’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
You do realise that the core messages of my post are a) that weakening your premise and forgetting the original can lead you astray, and b) the move from “performs useful long horizon action” to “can reason about as if it is a stereotyped goal driven agent” is blocked, right? Of course if you reproduce the error and assume the conclusion you will find plenty to disagree with.

By way of clarification: there are two different data generating processes, both thought experiments: one proposes useful things we’d like advanced AI to do. This leads to things like the stadium builder. The other proposes “agents” of a certain type and leads to the instrumental convergence thesis. What you can get is that you can choose a set of high probability according to the first process that ends up being low probability according to the second.

You are not proposing tasks you are proposing goals. A task here is like “suppose it successfully Xed”, without commitment to whether it wants X or in what way it wants X.

David Johnston 13 Aug 2025 23:38 UTC
4 points
0
in reply to: yrimon’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
There are still a few issues here. First, instrumental convergence is motivated by a particular (vague) measure on goal directed agents. Any behaviour that completes any particular long horizon task (or even any list of such behaviours) will have tiny measure here, so low P(X|Y) is easily satisfied.

Secondly, just because “long horizon task achievement” is typical of instrumental convergence doom agents, it does not follow that instrumental convergence doom is typical of long horizon task achievers.

The task I chose seems to be a sticking point, but you totally can pick a task you like better and try to run the arguments for it yourself.

You need to banish all the talk of “isn’t it incentivised to…?”, it’s not justified yet.

David Johnston 13 Aug 2025 19:45 UTC
5 points
0
in reply to: yrimon’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
I do appreciate everyone who takes the time to engage but this response is a bit frustrating. I don’t assume anywhere the AI has a goal of stadium building—I assume that it builds stadiums (which is a version of the assumption introduced by MIRI that the AI does complex long horizon stuff). The argument then goes: yes, an AI that builds stadiums is going to look like it does a bunch of goal directed stuff, but it doesn’t follow that it needs to have exactly the right kind of “stadium building goal” or else instrumental convergence related disaster strikes—that step is actually a reasoning error. Given that this step fails, we haven’t established why it matters that it’s hard to instil “the right kind of goals”, so I don’t engage with that part of the argument—I’m dealing with the logically prior step.

I know the argument is not easy, especially if you lack a background in probability, but it is precise and I worked quite hard on making it as clearly as I could.

David Johnston 13 Aug 2025 13:15 UTC
2 points
0
in reply to: Rafael Harth’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
The form of the fallacy is “forgetting” the first proposition $X$ in favour of a weaker one $Y$ . In the example, this means we put ourselves in the position of answering “you will be given a nonnegative int32 tomorrow, no further clues, predict what it is”. How would you choose a distribution in this situation?