Ivan Vendrov

Karma: 1,042

Ivan Vendrov 21 Apr 2025 21:10 UTC
9 points
6
in reply to: ryan_greenblatt’s comment on: Vladimir_Nesov’s Shortform
this isn’t evidence against OP? if it’s true that RL lowers pass@k performance for sufficiently large k, we’d certainly expect o1 with 10k submissions to be weaker than base/instruct with 10k submissions.

Ivan Vendrov 14 Apr 2025 14:10 UTC
6 points
2
in reply to: Kaj_Sotala’s comment on: Unbendable Arm as Test Case for Religious Belief
I think we mostly agree, I was pointing to a strawman of scientific materialism that I used to but no longer hold. Maybe a clearer example is a verbal practice like a mantra, chanting “God is good”—which is incompatible with the constraint to only say things that are intersubjectively verifiable, at least in principle. If someone were to interrupt and ask “wait how do you know that? what’s your probability on that claim?” your answer would have to look something like this essay.
nothing prevents you from visualizing it while remaining aware of the fact that you’re only imagining it.
This does seem to be the case for unbendable arm, but I’m less sure it generalizes to more central religious beliefs like belief in a loving God or the resurrection of the dead! I don’t see an a priori reason why certain beliefs wouldn’t require a lack of conscious awareness that you’re imagining them in order to “work”, so want to make sure my worldview is robust to this least convenient possible world. Curious if you have further evidence or arguments for this claim!

Ivan Vendrov 14 Apr 2025 13:58 UTC
6 points
0
in reply to: Kaj_Sotala’s comment on: Unbendable Arm as Test Case for Religious Belief
Really interesting, thanks! I wonder the extent to which this is true in general (any empirically-found-to-be-useful religious belief can be reformulated as a fact about physics or sociology or psychology and remain as useful) or if there are any that still require holding mystical claims, even if only in an ‘as-if’ manner.

Ivan Vendrov 14 Apr 2025 13:53 UTC
5 points
2
in reply to: jimmy’s comment on: Unbendable Arm as Test Case for Religious Belief
thanks for running the test!
IIRC the first time this was demonstrated to me it didn’t come with any instructions about tensing or holding, just ‘Don’t let me bend your arm’, exactly the language you used with your wife. But people vary widely in somatic skills and how they interpret verbal instructions; I definitely interpreted it as ‘tense your arm really hard’ and that’s probably why the beam / firehose visualization helped.
Makes me think the same is likely true of religious beliefs—they help address a range of common mental mistakes but for each particular mistake there are people who have learned not to make them using some other process. e.g. “Inshallah” helps neurotic people cut off unhelpful rumination, whereas low neuroticism people just don’t need it.

Ivan Vendrov 14 Apr 2025 13:41 UTC
4 points
2
in reply to: Jiro’s comment on: Unbendable Arm as Test Case for Religious Belief
You’re right about the ‘seven literal days’ thing—seems like nonsense to me, but notably I haven’t seen it used much to justify action so I wouldn’t call it an important belief in the sense that it pays rent. More like an ornament, a piece of poetry or mythology.
‘believing in heaven’ is definitely an important one, but this is exactly the argument in the post? ‘believing in the beam of light’ doesn’t make the beam of light exist, but it does (seem to) make my arm stronger. Similarly, believing in heaven doesn’t create heaven ^[1] but it might help your society flourish.
It’s an important point though that it’s not that believing in A makes A happen, more like believing in some abstracted/idealized/extremized version of A makes A happen.
This does pose a bigger epistemic challenge than simple hyperstition, because the idealized claim never becomes true, and yet (in the least convenient possible world) you have to hold it as true in order to move directionally towards your goal.
1. ^
  well, humanity could plausibly build a pretty close approximation of heaven using uploads in the next 50 years, but that wasn’t reasonable to think 2000 years ago

Ivan Vendrov 14 Apr 2025 13:27 UTC
2 points
0
in reply to: Oleander’s comment on: Unbendable Arm as Test Case for Religious Belief
thanks! as in there was no difference between visualizing and not?

Unbendable Arm as Test Case for Religious Belief

Ivan Vendrov14 Apr 2025 1:57 UTC

28 points

48 comments2 min readLW link

(nothinghuman.substack.com)

Ivan Vendrov 10 Apr 2025 4:07 UTC
2 points
0
in reply to: abramdemski’s comment on: Ivan Vendrov’s Shortform
Minor points just to get them out of the way:
1. I think Bayesian optimization still makes sense with infinite compute if you have limited data (infinite compute doesn’t imply perfect knowledge, you still have to run experiments in the world outside of your computer).
2. The reason I specified evolutionary search is because that’s the claim I see Lehman & Stanley as making—that algorithms pursuing simple objectives tend to not be robust in an evolutionary sense. I’m less confident making claims about broader classes of optimization but not intentionally excluding them
Meta point: it feels like we’re bouncing between incompatible and partly-specified formalisms before we even know what the high level worldview diff is.
To that end, I’m curious what you think the implications of the Lehman & Stanley hypothesis would be—supposing it were shown even for architectures that allow planning to search, which I agree their paper does not do. So yes you can trivially exhibit a “goal-oriented search over good search policies” that does better than their naive novelty search, but what if it turns out a “novelty-oriented search over novelty-oriented search policies” does better still? Would this be a crux for you, or is this not even a coherent hypothetical in your ontology of optimization?

Ivan Vendrov 10 Apr 2025 3:42 UTC
2 points
0
in reply to: gwern’s comment on: Ivan Vendrov’s Shortform
“harness” is doing a lot of work there. If incoherent search processes are actually superior then VNM agents are not the type of pattern that is evolutionary stable, so no “harnessing” is possible in the long term, more like a “dissolving into”.
Unless you’re using “VNM agent” to mean something like “the definitionally best agent”, in which case sure, but a VNM agent is a pretty precise type of algorithm defined by axioms that are equivalent to saying it is perfectly resistant to being Dutch booked.
Resistance to Dutch booking is cool, seems valuable, but not something I’d spent limited compute resources on getting six nines of reliability on. Seems like evolution agrees, so far: the successful organisms we observe in nature, from bacteria to humans, are not VNM agents and in fact are easily Dutch booked. The question is whether this changes as evolution progresses and intelligence increases.

Ivan Vendrov 8 Apr 2025 16:07 UTC
2 points
0
in reply to: abramdemski’s comment on: Ivan Vendrov’s Shortform
I agree Bayesian optimization should win out given infinite compute, but what makes you confident that evolutionary search under computational resource scarcity selects for anything like an explicit Bayesian optimizer or long term planner? (I say “explicit” because the Bayesian formalism has enough free parameters that you can post-hoc recast ~any successful algorithm as an approximation to a Bayesian ideal)

Ivan Vendrov’s Shortform

Ivan Vendrov8 Apr 2025 14:16 UTC

6 points

9 comments LW link

Ivan Vendrov 8 Apr 2025 14:16 UTC
2 points
−2
on: Ivan Vendrov’s Shortform
Are instrumental convergence & Omohundro drives just plain false? If Lehman and Stanley are right in “Novelty Search and the Problem with Objectives” (https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/lehmanNoveltySearch11.pdf) later popularized in their book “Why Greatness Cannot Be Planned”, VNM-coherent agents that pursue goal stability will reliably be outcompeted by incoherent search processes pursuing novelty.

Ivan Vendrov 7 Feb 2025 21:27 UTC
8 points
2
on: Racing Towards Fusion and AI
Great, thought-provoking post. The AI research community certainly felt much more cooperative before it got an injection of startup/monopoly/winner-take-all thinking. Google Brain publishing the Transformer paper being a great example.
I wonder how much this truly is narrative, as opposed to AI being genuinely more winner-take-all than fusion in the economic sense. Certainly the hardware layer has proven quite winner-take-all so far with NVDA taking a huge fraction of the profit; same with adtech, the most profitable application of (last-generation) AI, where network effects and first mover advantages have led to the dominance of a couple of companies.
Global foundation model development efforts being pooled into an international consortium like ITER or CERN seems quite good to me in terms of defusing race dynamics. Perhaps we will get there in a few years if private capital loses interest in funding 100B+ training runs.

Ivan Vendrov 22 Jan 2025 5:24 UTC
0 points
−1
in reply to: Ashin’s comment on: Passages I Highlighted in The Letters of J.R.R.Tolkien
I think writing one of the best selling books of your century is extraordinary evidence you’ve understood something deep about human nature, which is more than most random rationalist bloggers can claim. but yes doesn’t imply you have a coherent philosophy or benevolent political program

Ivan Vendrov 30 Dec 2024 0:02 UTC
3 points
0
in reply to: Stag’s comment on: Shallow review of technical AI safety, 2024
cuts off some nuance, I would call this the projection of the collective intelligence agenda onto the AI safety frame of “eliminate the risk of very bad things happening” which I think is an incomplete way of looking at how to impact the future
in particular I tend to spend more time thinking about future worlds that are more like the current one in that they are messy and confusing and have very terrible and very good things happening simultaneously and a lot of the impact of collective intelligence tech (for good or ill) will determine the parameters of that world

Ivan Vendrov 29 Dec 2024 13:13 UTC
6 points
2
on: Shallow review of technical AI safety, 2024
Thanks, this is a really helpful broad survey of the field. Would be useful to see a one-screen-size summary, perhaps a table with the orthodox alignment problems as one axis?
I’ll add that the collective intelligence work I’m doing is not really “technical AI safety” but is directly targeted at orthodox problems 11. Someone else will deploy unsafe superintelligence first and 13. Fair, sane pivotal processes, and targeting all alignment difficulty worlds not just the optimistic one (in particular, I think human coordination becomes more not less important in the pessimistic world). I write more of how I think about pivotal processes in general in AI Safety Endgame Stories but it’s broadly along the lines of von Neumann’s
For progress there is no cure. Any attempt to find automatically safe channels for the present explosive variety of progress must lead to frustration. The only safety possible is relative, and it lies in an intelligent exercise of day-to-day judgment.

Ivan Vendrov 26 Nov 2024 16:53 UTC
11 points
4
in reply to: Lao Mein’s comment on: Passages I Highlighted in The Letters of J.R.R.Tolkien
I find that surprising, do you care to elaborate? I don’t think his worldview is complete, but he cares deeply about a lot of things I value too, which modern society seems not to value. I would certainly be glad to have him in my moral parliament.

Ivan Vendrov 26 Nov 2024 16:15 UTC
8 points
3
in reply to: cousin_it’s comment on: Passages I Highlighted in The Letters of J.R.R.Tolkien
Feels connected to his distrust of “quick, bright, standardized, mental processes”, and the obsession with language. It’s like his mind is relentlessly orienting to the territory, refusing to accept anyone else’s map. Which makes it harder to be a student but easier to discover something new. Reminds me of Geoff Hinton’s advice to not read the literature before engaging with the problem yourself.