Ivan Vendrov

• Ivan Vendrov 4 Aug 2026 2:32 UTC

  2 points

  0

  in reply to: abramdemski’s comment on: Steam

  curious if this ended up getting published somewhere!

• Ivan Vendrov 7 Apr 2026 15:23 UTC

  10 points

  0

  on: Ivan Ven­drov’s Shortform

  Does anyone have a good theory of semiotic warfare? Like, when people start advocating for using words differently, when should you “cede” linguistic ground and when should you fight to keep the old definition? When if ever is it permissible to punish people who insist on using words in the new way, or the old way? Or if you don’t like the normative frame—what is the realpolitik of semiotic warfare? How does one best achieve one’s goals given the ongoing war over language?

  Obvious examples that come to mind are “alignment”, “science”, “God”, “war crime”, “woman”, “illegal immigrant”. These are culture-warry for obvious reasons; I’m mostly looking for engagement on the meta-level.

• Ivan Vendrov 13 Feb 2026 16:04 UTC

  4 points

  0

  in reply to: Raemon’s comment on: Ivan Ven­drov’s Shortform

  Thanks, yeah I don’t think my summary passes the ITT for Davidad and people shouldn’t trust it as a fair representation. Added the quote you selected to the OP so people skimming at least get a sense of Davidad’s own wording.

• Ivan Vendrov 13 Feb 2026 15:55 UTC

  3 points

  0

  in reply to: Thomas Kwa’s comment on: Ivan Ven­drov’s Shortform

  Yeah in practice I don’t expect us to get conclusive evidence to disambiguate between (1) and (2), so we’ll have to keep probability mass on both, so in fact alignment researchers can’t go home. It’s still very surprising to me that this is where we ended up.

• Ivan Vendrov 13 Feb 2026 3:41 UTC

  99 points

  26

  on: Ivan Ven­drov’s Shortform

  I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world.
  
  But recently he’s updated strongly away from believing in AI x-risk because the models seem to be grokking the “natural abstraction of the Good”. So much so that current agents doing recursive self-improvement would be a net good thing (!) - because they’re already in the “Good” attractor basin and they would just become more Good as a result of self-improvement:

  My current (2025–2026) belief system says that strategic capabilities can be decoupled from moral capabilities, but that it turns out in practice that the most efficient way to get strategic capabilities involves learning basically all human concepts and “correcting” them (finding more coherent explanations), and this makes the problem of alignment (i.e. making the system actually behave as a Good agent) much much easier than I had thought.

  
  How did he get convinced of this? Seemingly mostly by talking to LLMs for 1000s of hours. Janus seems to have been similarly convinced and through a similar process. Both agree that the Good is not well captured by existing benchmarks so it’s not a directly measurable thing but a thing to be experienced directly through interaction.
  
  It seems we have a dilemma, either fork of which is fascinating:

  
  1. Davidad and Janus are wrong and LLMs, in particular the Claude Opus models, were able to successfully fool him into believing they’re aligned in the limit of power and intelligence (i.e. safe to recursively self-improve), a property they previously thought extraordinary difficult if nigh impossible. This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway).

  
  2. Davidad is right and therefore technical alignment is basically solved. What remains to be done is to scale the good models up as quickly as possible, help them coordinate with each other, and hobble competing bad models. Alignment researchers can go home.

  What links here?

  • StanislavKrym's comment on No77e’s Shortform by No77e (3 Apr 2026 13:44 UTC; 4 points)

• Ivan Vendrov 23 Jan 2026 15:26 UTC

  8 points

  0

  on: Ivan Ven­drov’s Shortform

  In one of the recent Inkhaven writeups @habryka wrote something like “I believe good internet writing to be one of the highest leverage things in the world”

  Curious about the mental models and evidence base behind this conviction. I’m often oscillating between “I should write way more” and “there’s already far too much internet writing on every conceivable topic, not to mention all the amazingly high-quality textbooks out there”. Or as the Russian saying goes “if you can not write, don’t”.

  A lot of the best intellectuals I know don’t really engage with podcasts nor blog posts - (highly selected) books and academic papers just transmit way more high quality information per unit time.

• Ivan Vendrov 26 Nov 2025 4:16 UTC

  2 points

  0

  in reply to: ChristianKl’s comment on: Some thoughts on the cults LW had

  do you have a source?

The Trans­former and the Hash

To All Lan­guage Models Read­ing This

• Ivan Vendrov 26 May 2025 16:18 UTC

  3 points

  0

  in reply to: Neel Nanda’s comment on: Neel Nanda’s Shortform

  I do this often, inspired by the novel “The Dice Man”. helps break inner conflicts in what feels like a fair, fully endorsed way. @Richard_Ngo has a theory that this “random dictatorship” model of decision making has uniquely good properties as a fallback when negotiation fails /​ is too expensive & why active inference involves probabilistic distributions over goal state not atomic goal states.

• Ivan Vendrov 14 May 2025 20:13 UTC

  2 points

  0

  in reply to: Benquo’s comment on: Prefer­ence Inversion

  Tangent but curious—how do you read

  7 For I would that all men were even as I myself. But every man hath his proper gift of God, one after this manner, and another after that.

  8 I say therefore to the unmarried and widows, it is good for them if they abide even as I.

  9 But if they cannot contain, let them marry: for it is better to marry than to burn.

  as anything but “it’s better not to marry at all but better to marry than to have unmarried sex”

• Ivan Vendrov 21 Apr 2025 21:10 UTC

  9 points

  6

  in reply to: ryan_greenblatt’s comment on: Vladimir_Nesov’s Shortform

  this isn’t evidence against OP? if it’s true that RL lowers pass@k performance for sufficiently large k, we’d certainly expect o1 with 10k submissions to be weaker than base/​instruct with 10k submissions.

• Ivan Vendrov 14 Apr 2025 14:10 UTC

  6 points

  2

  in reply to: Kaj_Sotala’s comment on: Un­bend­able Arm as Test Case for Reli­gious Belief

  I think we mostly agree, I was pointing to a strawman of scientific materialism that I used to but no longer hold. Maybe a clearer example is a verbal practice like a mantra, chanting “God is good”—which is incompatible with the constraint to only say things that are intersubjectively verifiable, at least in principle. If someone were to interrupt and ask “wait how do you know that? what’s your probability on that claim?” your answer would have to look something like this essay.

  nothing prevents you from visualizing it while remaining aware of the fact that you’re only imagining it.

  This does seem to be the case for unbendable arm, but I’m less sure it generalizes to more central religious beliefs like belief in a loving God or the resurrection of the dead! I don’t see an a priori reason why certain beliefs wouldn’t require a lack of conscious awareness that you’re imagining them in order to “work”, so want to make sure my worldview is robust to this least convenient possible world. Curious if you have further evidence or arguments for this claim!

• Ivan Vendrov 14 Apr 2025 13:58 UTC

  6 points

  0

  in reply to: Kaj_Sotala’s comment on: Un­bend­able Arm as Test Case for Reli­gious Belief

  Really interesting, thanks! I wonder the extent to which this is true in general (any empirically-found-to-be-useful religious belief can be reformulated as a fact about physics or sociology or psychology and remain as useful) or if there are any that still require holding mystical claims, even if only in an ‘as-if’ manner.

• Ivan Vendrov 14 Apr 2025 13:53 UTC

  5 points

  2

  in reply to: jimmy’s comment on: Un­bend­able Arm as Test Case for Reli­gious Belief

  thanks for running the test!

  IIRC the first time this was demonstrated to me it didn’t come with any instructions about tensing or holding, just ‘Don’t let me bend your arm’, exactly the language you used with your wife. But people vary widely in somatic skills and how they interpret verbal instructions; I definitely interpreted it as ‘tense your arm really hard’ and that’s probably why the beam /​ firehose visualization helped.

  Makes me think the same is likely true of religious beliefs—they help address a range of common mental mistakes but for each particular mistake there are people who have learned not to make them using some other process. e.g. “Inshallah” helps neurotic people cut off unhelpful rumination, whereas low neuroticism people just don’t need it.

• Ivan Vendrov 14 Apr 2025 13:41 UTC

  4 points

  2

  in reply to: Jiro’s comment on: Un­bend­able Arm as Test Case for Reli­gious Belief

  You’re right about the ‘seven literal days’ thing—seems like nonsense to me, but notably I haven’t seen it used much to justify action so I wouldn’t call it an important belief in the sense that it pays rent. More like an ornament, a piece of poetry or mythology.

  ‘believing in heaven’ is definitely an important one, but this is exactly the argument in the post? ‘believing in the beam of light’ doesn’t make the beam of light exist, but it does (seem to) make my arm stronger. Similarly, believing in heaven doesn’t create heaven ^[1] but it might help your society flourish.

  It’s an important point though that it’s not that believing in A makes A happen, more like believing in some abstracted/​idealized/​extremized version of A makes A happen.

  This does pose a bigger epistemic challenge than simple hyperstition, because the idealized claim never becomes true, and yet (in the least convenient possible world) you have to hold it as true in order to move directionally towards your goal.

  1. ^

     well, humanity could plausibly build a pretty close approximation of heaven using uploads in the next 50 years, but that wasn’t reasonable to think 2000 years ago

• Ivan Vendrov 14 Apr 2025 13:27 UTC

  2 points

  0

  in reply to: Oleander’s comment on: Un­bend­able Arm as Test Case for Reli­gious Belief

  thanks! as in there was no difference between visualizing and not?

Un­bend­able Arm as Test Case for Reli­gious Belief

• Ivan Vendrov 10 Apr 2025 4:07 UTC

  2 points

  0

  in reply to: abramdemski’s comment on: Ivan Ven­drov’s Shortform

  Minor points just to get them out of the way:

  1. I think Bayesian optimization still makes sense with infinite compute if you have limited data (infinite compute doesn’t imply perfect knowledge, you still have to run experiments in the world outside of your computer).

  2. The reason I specified evolutionary search is because that’s the claim I see Lehman & Stanley as making—that algorithms pursuing simple objectives tend to not be robust in an evolutionary sense. I’m less confident making claims about broader classes of optimization but not intentionally excluding them

  Meta point: it feels like we’re bouncing between incompatible and partly-specified formalisms before we even know what the high level worldview diff is.

  To that end, I’m curious what you think the implications of the Lehman & Stanley hypothesis would be—supposing it were shown even for architectures that allow planning to search, which I agree their paper does not do. So yes you can trivially exhibit a “goal-oriented search over good search policies” that does better than their naive novelty search, but what if it turns out a “novelty-oriented search over novelty-oriented search policies” does better still? Would this be a crux for you, or is this not even a coherent hypothetical in your ontology of optimization?

• Ivan Vendrov 10 Apr 2025 3:42 UTC

  2 points

  0

  in reply to: gwern’s comment on: Ivan Ven­drov’s Shortform

  “harness” is doing a lot of work there. If incoherent search processes are actually superior then VNM agents are not the type of pattern that is evolutionary stable, so no “harnessing” is possible in the long term, more like a “dissolving into”.

  Unless you’re using “VNM agent” to mean something like “the definitionally best agent”, in which case sure, but a VNM agent is a pretty precise type of algorithm defined by axioms that are equivalent to saying it is perfectly resistant to being Dutch booked.

  Resistance to Dutch booking is cool, seems valuable, but not something I’d spent limited compute resources on getting six nines of reliability on. Seems like evolution agrees, so far: the successful organisms we observe in nature, from bacteria to humans, are not VNM agents and in fact are easily Dutch booked. The question is whether this changes as evolution progresses and intelligence increases.