Chris van Merwijk

Karma: 770

Chris van Merwijk 27 Aug 2025 3:01 UTC
3 points
2
in reply to: Noosphere89’s comment on: Epistemic advantages of working as a moderate
But the problem is that we likely don’t have time to flesh out all the details or do all the relevant experiments before it might be too late, and governments need to understand that based on arguments that therefore cannot possibly rely on everything being fleshed out.

Of course I want people to gather as much important empirical evidence and concrete detailed theory as possible asap.

Also, the pre-everything-worked-out-in-detail arguments also need to inform which experiments are done, and so that is why people who have actually listened to those pre-detailed arguments end up on average doing much more relevant empirical work IMO.

Chris van Merwijk 27 Aug 2025 2:55 UTC
1 point
0
in reply to: Richard_Ngo’s comment on: Epistemic advantages of working as a moderate
I don’t think it’s *contradicting* it but I vaguely thought maybe it’s in tension with:
“Big changes within ~~companies~~ Government AI x-risk policy are typically bottlenecked much more by coalitional politics than knowledge of technical details.
Because lack of knowledge of technical details by A ends up getting B to reject and oppose A.

Mostly I wasn’t trying to push against you though, and more trying to download part of your model on how important you think this is, out of curiosity, given your experience at OA.

Chris van Merwijk 26 Aug 2025 6:40 UTC
14 points
20
in reply to: Richard_Ngo’s comment on: Epistemic advantages of working as a moderate
Do you not think it’s a problem that big-picture decisions can be blocked by a kind of overly-strong demand for rigor from people who are used to mostly think about technical details?

I sometimes notice something roughly like the following dynamic:
1. Person A is trying to make a big-picture claim (e.g. that ASI could lead to extinction) that cannot be argued for purely in terms of robust technical details (since we don’t have ASI yet to run experiments, and don’t have a theory yet),
2. Person B is more used to think about technical details that allow you to make robust but way more limited conclusions.
3. B finds some detail in A’s argument that is unjustified or isn’t exactly right, or even just might be wrong.
4. A thinks the detail really won’t change the conclusion, and thinks this just misses the point, but doesn’t want to spend time, because getting all the details exactly right would take maybe a decade.
5. B concludes A doesn’t know what they’re talking about and continues ignoring the big picture question completely and keeps focusing on more limited questions.
6. The issue ends up ignored.

It seems to me that this dynamic is part of the coalitional politics and how the high-level arguments are received?

Chris van Merwijk 23 Aug 2025 6:52 UTC
10 points
0
on: Underdog bias rules everything around me
Interesting, I wonder what you think of this:

My sense is that there is a underlying psychological dynamic going on that people tend to judge their current state relative to some “absolute victory” state:
* Two countries who are in total war with each other are both in a precarious state. Each of them judges the current state relative to “I have completely won and subjugated the opponent”. Relative to this total victory state they feel very precarious.
* Islam is doing quite well globally. Islam “has” 1/4th the global population, it is increasing in members the fastest out of the major religions. Yet I suspect the reason for at least some Muslims that this feels very dis-satisfactory is because according to them Islam “should” have ~100% followers worldwide. (of course there are many other factors going on here). Relative to this total victory state, the current situation of merely 1/4th of the global population seems extremely weak.
* People in the AI x-risk community want ~everyone or ~every-serious-person to take AI x-risk seriously, rather than mock or ignore it. People who think it’s all a big dangerous distraction want ~no-one to take it seriously and it to never get any serious news coverage. Relative to these diametrically opposed total victory conditions, both sides feel the other side has “too much” influence, and their own side seems too weak.

What do you think of this idea, that the sense of being an underdog is (often) downstream of a prior feeling of your side being weak/threatened relative to a total-victory condition? And that this causes a distorted picture in which your side is actually objectively weak relative to the other side.

Chris van Merwijk 19 Jul 2025 6:36 UTC
3 points
0
on: A night-watchman ASI as a first step toward a great future
Just a quick comment after skimming: This seems broadly similar to what Eric Drexler called “Security Services” in his “Comprehensive AI Services” technical report he wrote at FHI.

Chris van Merwijk 16 Jul 2025 6:19 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Why I’m not into the Free Energy Principle
So I think all of this sounds mostly reasonable (and probably based on a bunch of implicit world-model about the brain I don’t have), especially the longest paragraph makes me update.

I think whether I agree with this view really depends heavily on quantitatively how well these brain-in-a-dish systems perform which I don’t know so I’ll look into it more first.

Chris van Merwijk 15 Jul 2025 17:04 UTC
5 points
0
in reply to: Steven Byrnes’s comment on: Why I’m not into the Free Energy Principle
Oh, I didn’t expect you to deny the evidence, interesting. Before I look into it more to try to verify/falsify (which I may or may not do), suppose that it turns out this general method does in fact, i.e. it learns to play pong, or at least in some other experiment it learns using this exact mechanism, would that be a crux? I.e. would that make you significantly update towards active inference being a useful and correct theory of the (neo-)cortex?
EDIT: the paper in your last link seems to be a purely semantic criticism of the paper’s usage of words like “sentience” and “intelligence”. They do not provide any analysis at all of the actual experiment performed.

Chris van Merwijk 15 Jul 2025 5:02 UTC
2 points
0
in reply to: Roman Leventov’s comment on: Why I’m not into the Free Energy Principle
Im curious what @Steven Byrnes has to say about the Kagan et al, Oct 2022 paper that @Roman Leventov mentioned.
My summary (I’m uncertain as its a bit unclearly written I find) is that they
1. Put a bunch of undifferentiated and unconnected human/mice cortical neurons in a petridish with electrodes connected to a computer, playing the Pong game.
2. Some of the electrodes activated a high voltage when the ball was at that relative position (place encoding).
3. Some electrodes measured electrical signals that then caused the bar/racket to move.
4. Whenever the racket missed the ball, 4 seconds of uniform random noise was input instead of the place encoding.
And this caused the cells to learn to play pong correctly, i.e. not miss the ball.
Isn’t this concrete evidence in favor of active inference? This seems like evidence that the cortex is doing active inference at a very micro level? Because the neural circuit in the petridish is not merely making predictions about what sensory observations it will see, but actually taking actions to minimize prediction error later. We could try to look at the implementation details of how this happens and then it might turn out to work by a feedback control system, but I don’t know the implementation details. My best guess for how to model this behaviour would be essentially that the whole neural net is optimizing itself so as to minimize its average prediction error over time, and where prediction error is actually a hardcoded variable in the neural net (I don’t know what, maybe some number of proteins of a certain type inside the soma, or perhaps just the firing rate, I don’t know enough about neuroscience).

I’m not sure about any of this. But it really does seem like the neural network ends up taking actions to minimize its later prediction error, without any kind of RL. It basically seems like the outputs of the neurons are all jointly optimized to minimze average prediction error over time within the Pong environment. And that is exactly what active inference claims, as far as I understand (but I haven’t studied it much). And to be clear, afaict it is completely possible that (in the brain, not in this petridish) on top of this active inference system there is RL happening, so this doesn’t mean predictive processing + active inference (+ free energy principle maybe idk) is a unified theory of the brain, but maybe it still is a correct theory of the (neo-)cortex?

(Of course that doesn’t mean that the free energy principle is the right tool, but it might be the right tool for an active inference system even though it’s overkill for a thermostat. This is not my main question though).

Chris van Merwijk 14 Jul 2025 5:39 UTC
3 points
2
in reply to: Vaniver’s comment on: IABIED: Advertisement design competition
I’m worried that “This is not a metaphor” will not be taken correctly. Most people do not communicate in this sort of explicit literal way I think. I expect them to basically interpret it as a metaphor if that is their first instinct, and then just be confused or not even pay attention to “this is not a metaphor”

Chris van Merwijk 14 Jul 2025 5:36 UTC
6 points
3
on: IABIED: Advertisement design competition
Just a general concern regarding some of these proposals:
* It should be really very clear that this is a book
* Some of the proposals give me a vibe that might be interpreted as “this is a creative ad for a found-footage movie/game pretending to be serious”
* People’s priors with these kinds of posters are very strongly that it is entertainment. This needs to be actively prevented.
* Even if it rationally cannot be entertainment if you analyze the written words, it is I think much better if it actually feels at a gut level (before you read the text) closer to an infomercial than to the marketing poster of a movie. Like I’m worried about the big red letters with black background for example...

Chris van Merwijk 5 Jul 2025 7:10 UTC
7 points
2
in reply to: Daniel Kokotajlo’s comment on: I Think Eliezer Should Go on Glenn Beck
Do you still agree with this as of july 2025? It seems currently slightly more on track to be blue-coded or at least anti-Trump coded? I’m not American, but it seems to me that as of July 2025 the situation has changed significantly and anything that strengthens the pro AI=xrisk camp within the Republican camp is good.

Chris van Merwijk 5 Jul 2025 7:08 UTC
3 points
0
in reply to: Vaniver’s comment on: I Think Eliezer Should Go on Glenn Beck
I know this is a very late response, but my intuition is that going on very conservative shows is a good way for it NOT to end up polarized (better than just going on neutral shows), since it’s more likely to be polarized pro-liberal in the long run? Avoiding conservative shows seems like exactly the kind of attitude that will make it polarized.

Chris van Merwijk 5 Jul 2025 5:54 UTC
1 point
0
in reply to: Vaniver’s comment on: New Endorsements for “If Anyone Builds It, Everyone Dies”
Is there a way we can ask Jon Wolfsthal to ask Obama too?

Chris van Merwijk 10 Apr 2025 4:23 UTC
LW: 3 AF: 2
0
AF
on: The Pando Problem: Rethinking AI Individuality
This comment was written by Claude, based on my bullet points:

I’ve been thinking about the split-brain patient phenomenon as another angle on this AI individuality question.
Consider split-brain patients: despite having the corpus callosum severed, the two hemispheres don’t suddenly become independent agents with totally different goals. They still largely cooperate toward shared objectives. Each hemisphere makes predictions about what the other is doing and adjusts accordingly, even without direct communication.
Why does this happen? I think it’s because both hemispheres were trained together for their whole life, developing shared predictive models and cooperative behaviors. When the connection is cut, these established patterns don’t just disappear—each hemisphere fills in missing information with predictions based on years of shared experience.
Similarly, imagine training an AI model to solve some larger task, consisting of a bunch of subtasks. Just for practical reasons it will have to carve up the subtask to some extent and call instances of itself to solve the subtask. In order to perform the larger task well, there will be an incentive on the model for these instances to have internal predictive models, habits, drives of something like “I am part of a larger agent, performing a subtask”.
Even if we later placed multiple instances of such a model (or of different but similar models) in positions meant to be adversarial—perhaps as checks and balances on each other—they might still have deeply embedded patterns predicting cooperative behavior from similar models. Each instance might continue acting as if it were part of a larger cooperative system, maintaining coordination through these predictive patterns rather than through communication even though their “corpus callosum” is cut (in analogy with split brain patients).
I’m not sure how far this analogy goes, it’s just a thought.

Chris van Merwijk 10 Apr 2025 2:45 UTC
LW: 1 AF: 1
0
AF
on: The Pando Problem: Rethinking AI Individuality
A version of what ChatGPT wrote here prompted
What was the prompt?

Chris van Merwijk 24 Mar 2025 13:48 UTC
1 point
−1
on: Vacuum Decay: Expert Survey Results
Overall, compared to the previous question, there was more of a consensus, with 55% of people responding that there is a 0% chance that technologically induced vacuum decay is possible.
Since anywhere near 0% seems way overconfident to me at first sight, just a random highly speculative unsubstantiated thought: Could this be partly motivated reasoning, that they’re afraid of a backlash against physics funding or something?

Chris van Merwijk 24 Mar 2025 13:42 UTC
2 points
0
on: Vacuum Decay: Expert Survey Results
They stated justification was primarily that the Standard Model of particle physics predicts metastability
Just to be sure, does this mean
1. That the standard model predicts that metastability is possible? i.e. it is consistent with the standard model for there to be metastability; or
2. If the standard model is correct, and certain empirical observations are correct, then we must be in a metastable state. i.e. the standard model together with certain empirical observations implies our actual universe is metastable?

Chris van Merwijk 23 Mar 2025 16:00 UTC
LW: 1 AF: 1
0
AF
in reply to: Vanessa Kosoy’s comment on: Compositional language for hypotheses about computations
I may be confused somehow. Feel free to ignore. But:
* At first I thought you meant the input alphabet to be the colors, not the operations.
* Instead, am I correct that “the free operad generated by the input alphabet of the tree automaton” is an operad with just one color, and the “operations” are basically all the labeled trees where labels of the nodes are the elements of the alphabet, such that the number of children of a node is always equal to the arity of that label in the input alphabet?
* That would make sense, as the algebra would then I guess assign the state space of the tree automaton to the single color of the operad, and each arity n operation would be mapped to the mathematical function from Q^n to Q.
* That would make sense I think, but then why do you talk about a “colored” operad in: “we can now define a deterministic automaton over a (colored) operad $O$ to be an $O$ -algebra”?

Chris van Merwijk 23 Mar 2025 4:17 UTC
LW: 1 AF: 1
0
AF
on: Compositional language for hypotheses about computations
More precisely, they are algebras over the free operad generated by the input alphabet of the tree automaton
Wouldn’t this fail to preserve the arity of the input alphabet? i.e. you can have trees where a given symbol occurs multiple times, and with different amounts of children? That wouldn’t be allowed from the perspective of the tree automaton right?

Chris van Merwijk 20 Feb 2025 11:18 UTC
6 points
0
in reply to: Noosphere89’s comment on: How might we safely pass the buck to AI?
Noosphere, why are you responding for a second time to a false interpretation of what Eliezer was saying, directly after he clarified this isn’t what he meant?