Adrià Garriga-alonso

• Adrià Garriga-alonso 11 May 2026 22:14 UTC

  2 points

  0

  on: Maybe I was too harsh on deep learn­ing the­ory (three days ago)

  it was shown by Lee et al.

  Also contemporaneously Alexander G. de G. Matthews et al.! And, while less famous, that paper was better in one way: it took the limit of the width of all layers simultaneously, instead of one by one. That is, Lee et al was a statement about:

  lim(width->infty) [ b_2 + W_2 nonlinearity( lim(width → infty) [W_1x + b_1])]
  
  whereas Matthews et al was a statement about:
  
  lim(width->infty)[ b_2 + W_2 nonlinearity(W_1x + b_1)]
  
  which is more complicated

  What links here?

  • Maybe I was too harsh on deep learn­ing the­ory (three days ago) by LawrenceC (30 Apr 2026 6:57 UTC; 106 points)

• Adrià Garriga-alonso 2 Apr 2026 18:34 UTC

  2 points

  0

  on: The In­ner Ring by C. S. Lewis

  And then, if you are drawn in, next week it will be something a little further from the rules, and next year something further still, but all in the jolliest, friendliest spirit

  This is incredibly similar to one interesting passage from Tolstoy’s The Death of Ivan Illyich, another famous moralizing story:

  but all this [affairs, immoral things] was done with such a tone of good breeding that no hard names could be applied to it. It all came under the heading of the French saying: Il faut que jeunesse se passe. It was all done with clean hands, in clean linen, with French phrases, and above all among people of the best society and consequently with the approval of people of rank.

• Adrià Garriga-alonso 11 Mar 2026 19:26 UTC

  5 points

  0

  in reply to: Mati_Roy’s comment on: Less Dead

  You can probably take out a loan to pay Nectome and MAiD, then pay back the loan with life insurance a couple of weeks later once you’re already preserved.

How much su­per­po­si­tion is there?

• Adrià Garriga-alonso 27 Jan 2026 15:57 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: michaelcohen’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  We can monitor that and mitigate it when we get there, using the previous generation of AIs.

• Adrià Garriga-alonso 27 Dec 2025 21:18 UTC

  2 points

  0

  in reply to: g’s comment on: True End­ing: Sacri­fi­cial Fire (7/​8)

  It goes from 1.05 to 1.2.

• Adrià Garriga-alonso 15 Dec 2025 2:56 UTC

  2 points

  0

  in reply to: shawnghu’s comment on: An­thropic’s JumpReLU train­ing method is re­ally good

  Thank you! I have fixed the link now.

• Adrià Garriga-alonso 6 Dec 2025 22:37 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: denkenberger’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  Yeah, true. It’s gone so well for so long that I forgot. I didn’t spend a lot of time thinking about this list.

• Adrià Garriga-alonso 6 Dec 2025 9:10 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: otto.barten’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  No, I think the blue-team will keep having the latest and best LLMs and be able to stop such attempts from randos. These AGIs won’t be so much magically superintelligent that they can take all the unethical actions needed to take over the world, without other AGIs stopping them.

• Adrià Garriga-alonso 2 Dec 2025 5:29 UTC

  2 points

  0

  in reply to: habryka’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  On Chrome on a Mac you can just C-f in the PDF, it just OCRs automatically. I didn’t have this problem.

• Adrià Garriga-alonso 2 Dec 2025 5:27 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  But it feels like you’d need to demonstrate this with some construction that’s actually adversarially robust, which seems difficult.

  I agree it’s kind of difficult.

  Have you seen Nicholas Carlini’s Game of Life series? It starts by building up logical gates up to a microprocessor that factors 15 in to 3 x 5.

  Depending on the adversarial robustness model (e.g. every second the adversary can make 1 square behave the opposite of lawfully), it might be possible to make robust logic gates and circuits. In fact the existing circuits are a little robust already—though not at the tune of 1 square per tick, that’s too much power for the adversary.

• Adrià Garriga-alonso 30 Nov 2025 18:05 UTC

  5 points

  −5

  in reply to: Liron’s comment on: Un­less its gov­er­nance changes, An­thropic is untrustworthy

  I very strongly agree with this and think it should be the top objection people first see when scrolling down. In a low-P(doom) world, Anthropic has done lots of good. (They proved that you can have the best and the most aligned model, and also the leadership is more trustworthy than OpenAI, who would otherwise lead). This is my current view.

  In a high-P(doom) world, none of that matters because they’ve raised the bar for capabilities when we really should be pausing AI. This was my previous view.

  I’m grudgingly impressed with Anthropic leadership for getting this right when I did not (not that anyone other than me cares what I believed, having ~zero power).

• Adrià Garriga-alonso 30 Nov 2025 16:51 UTC

  2 points

  0

  in reply to: otto.barten’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  What do you mean “faithfully execute”?

  If you mean that it’s executing its own goal, then I dispute that that will be random, the goal will be to be helpful and good.

  Is this different from the standard instrumental convergence algorithm?

• Adrià Garriga-alonso 30 Nov 2025 1:15 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Neel Nanda’s comment on: Cir­cuit dis­cov­ery through chain of thought us­ing policy gradients

  I am concerned that long chains of RL are just sufficiently fucked functions with enough butterfly effects that this wouldn’t be well approximated by this process.

  This is a concern. Two possible replies:

  • If it’s truly a chaotic system then there’s no good way to estimate the expectation.

  • In reality, it could be that the effects of neurons are not very chaotic, but this estimate of the gradient is very chaotic. Previous work actually shows that policy gradients are much less chaotic than the ‘reparameterization trick’ (in the case where the transition is continuous, differentiating through it). It could be that finite differences (resampling many rollouts with/​without the neuron activated) actually estimates effects better with less variance. We’ll see.

Cir­cuit dis­cov­ery through chain of thought us­ing policy gradients

• Adrià Garriga-alonso 29 Nov 2025 6:16 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Tim Hua’s comment on: Align­ment re­mains a hard, un­solved problem

  IMO at any level of sampling?

  Vacuously true. The actual question is: how much do you need to sample? My guess is it’s too much, but we’d see the base model scaling better than the RL’d model just like in this paper.

  Fortunately, DeepSeek’s Mathv2 just dropped which is an open-source model that gets IMO gold. We can do the experiment: is it similarly not improving with sampling compared to its own base model? My guess is yes, the same will happen.

• Adrià Garriga-alonso 28 Nov 2025 23:15 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: baurse’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  It’s not impossible that we are in an alignment-by-default world. But, I claim that our current insight isn’t enough to distinguish such a world from the gradual disempowerment/​going out with a whimper world.

  Well, yeah, I agree with that! You might notice this item in my candidate “What’s next” list:

  • Prevent economic incentives from destroying all value. Markets have been remarkably aligned so far but I fear their future effects. (Intelligence Curse, Gradual Disempowerment. Remarkably European takes.)

  This is not classical alignment emphasizing scheming etc. Rather it’s “we get more slop, AIs outcompete humans and so the humans that don’t own AIs have no recourse.” So I don’t think that undermines my point at all.

  Approximately never, I assume, because doing so isn’t useful.

  You should not assume such things. Humans invented scheming to take over, it might be the very reason we are intelligent.

  Until then, I claim we have strong reasons to believe that we just don’t know yet.

  We don’t know but we never really know and must act under uncertainty. I put forward that we can make a good guess.

• Adrià Garriga-alonso 28 Nov 2025 23:08 UTC

  2 points

  −13

  AF

  in reply to: evhub’s comment on: Align­ment re­mains a hard, un­solved problem

  Thank you for the reply. I want to engage without making it tiresome. The problem is that there are many things I disagree with in the worldview, the disagreement isn’t reducible to 1-5 double cruxes, but here are some candidates for the biggest cruxes for me. If any of these are wrong it’s bad news for my current view:

  • N->N+1 alignment:

    • will let humans align N+1 in cases where we can still check, with less and less effort.

    • is stable instead of diverging (the values of N+1 don’t drift arbitrarily far apart)

  • The N->N+1 improvement will continue to give out linear-ish improvements in perceived intelligence. We might get one larger jump or two, but it won’t continuously accelerate.

    • (a good analogy for this is loudness perception being logarithmic in sound pressure. Actual intelligence is logarithmic in METR time-horizon graph.)

  • Aligned-persona-seeming models won’t give out false AI safety research results, without making it visible on a CoT or latent reasoning.

    • (It’s perhaps possible to refrain from doing your best (sandbagging) but that doesn’t have nearly as bad effects, so it doesn’t count for this.)

  And here’s another prediction where I really stick my neck out, which isn’t load-bearing to the view, but still increases my confidence, so defeating it is important:

  • we can to a significant extent train with RL against model internals (probes) and textual evaluations from other models, without ill effects.

    • That is, we ask the N model to evaluate N+1, giving test-time compute to N, and train against that. (we continuously also finetune N to predict N+1′s relevant behavior better).

    • we also train linear probes and update them during the RL.

    • Effectively I’m claiming these things are good enough or they’re self-reinforcing when model is already ~aligned, so that effectively Goodhart’s Law is a poor description of reality.

  ---

  I still disagree with several of the points, but for time reasons I request that readers not update against Evan’s points if he just doesn’t reply to these.

  • disagree that increasing capabilities are exponential in a capability sense. It’s true that METR’s time horizon plot increases exponentially, but this still corresponds to a linear intuitive intelligence. (Like loudness (logarithmic) and sound pressure (exponential); we handle huge ranges well.) Each model that has come out has exponentially larger time horizon but is not (intuitively empirically) exponentially smarter.

  • “we still extensively rely on direct human oversight and review to catch alignment issues” That’s a fair point and should decrease confidence in my view, though I expected it. For properly testing sandwiching we’ll probably have to wait till models are superhuman, or use weak models + less weak models and test it out. Unfortunately perhaps the weak models are still too weak. But we’ve reached the point where you can maybe just use the actual Opus 3 as the weak model?

  • If we have a misaligned model doing research, we have lots of time to examine it with the previous model. I also do expect to see sabotage in the CoT or in deception probes

  • I updated way down on Goodharting on model internals due to Cundy and Gleave.

  Again, readers please don’t update down on these due to lack of a response.

• Adrià Garriga-alonso 28 Nov 2025 22:48 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: evhub’s comment on: Align­ment re­mains a hard, un­solved problem

  I’m honestly very curious what Ethan is up to now, both you and Thomas Kwa implied that he’s not doing alignment anymore. I’ll have to reach out...

• Adrià Garriga-alonso 28 Nov 2025 22:47 UTC

  LW: 3 AF: 1

  −1

  AF

  in reply to: Eli Tyre’s comment on: Align­ment re­mains a hard, un­solved problem

  In fact, base model seem to be better than RL models at reasoning, when you take best of N (with the same N for both the RL’d and base model). Check out my post summarizing the research on the matter:

  Yue, Chen et al. have a different hypothesis: what if the base model already knows all the reasoning trajectories, and all RL does is increase the frequency of reasoning or the frequency of the trajectory that is likely to work? To test this, Yue, Chen et al. use pass@K: let’s give the LLM a total of K attempts to answer the question, and if any of them succeed, mark the question as answering correctly. They report the proportion of correct questions in the data set.

  If the RL model genuinely learns new reasoning skills, over many questions the pass@K performance of RL will remain higher than the performance of the base model. As we increase K, the base model answers more and more of the easy questions, so its performance improves. But the RL model’s performance also answers more and more difficult questions. The performance of both increases in tandem with larger K.

  What actually happened is neither of these two things. For large enough K, the base model does better than the RL model. (!!!)