valentin forch

Karma: 4

valentin forch 26 Feb 2026 21:47 UTC
1 point
0
in reply to: Steven Byrnes’s comment on: Why we should expect ruthless sociopath ASI
A)
Yes, being much faster / able to clone would give AHI a great advantage over us and in principle this could be the seed for an AGI society with way faster minds than ours (they’d still have to wait for their experiments to finish though). Anyway, I think we should not let this happen. We have no mental tools except the examples of current societies to reason about this kind of thing. As far as I can tell, it would be impossible for us to control. Best case would be that we get the pet status.
B)
The “plan” is not directly encoded in the world model—actions are. The plan is either derived based on the world model or another name for the policy conditioned on the current world model state. Simple example would be MCTS over world model states conditioned on some policy.
Of course we will have to figure out what those states (and actions) mean. Good thing is that at any given time the world model will have only few (<<100) active high-level states (figuring out which states are high-level and which are not is possible, e.g., by checking for robustness to input perturbations). How many things can you imagine at once? I think this would be easier than what Anthropic is currently doing in their MechInterp department.
I think it is an interesting question whether it would have been possible to decode from Stalin’s mind in a meaningful sense how many people died under his regime. But in general I think it is possible to decode from peoples’ minds whether they just hurt or manipulated somebody. [This made me realize that the world model of a sociopath will indeed look different from a “normal” person because the internal states describing the agent will look different. We could check those states in response to other people getting hurt, too. I guess we would not need these internal states for tasks that do not include emulating a persona, so we could discard them anyways.]
Under your threat model we only need to audit the policy for sociopathy. Once it passes the test, we can run it at full speed. The test may take a while, sure. However, not testing models for safety when doing level 4-5 automation would be insane, even if you think you’ve managed to align the model.
With access to the internet / scientific literature any future machine will know everything about the alignment and safety discourse and how people think a model should behave.

valentin forch 24 Feb 2026 21:24 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Why we should expect ruthless sociopath ASI
I am sorry, I am not sure I quite understand what you are getting at with the Bezos and Stalin examples. If you agree that having ruthless sociopathic AHI (Stalin?) is a big deal, why start with the more distant, uncertain, and hard to reason about ASI scenario?
Can you walk through a concrete example of what someone can do with a such a system? Ideally something that’s very impactful, e.g. so impactful that it could plausibly cause or prevent human extinction.
I can’t give an example that goes much beyond self driving. However, self-driving (and other autonomous robotics applications) is quite a big deal (Tesla is currently worth more than “capture the light cone of all future value in the universe”-OpenAI). All vehicles suddenly going rogue (caused by sociopath AHI) would probably end human civilization (not quite human extinction).
I wanted to suggest that once you’ve mapped such a world model, and the world model is everything an agent can use for planning, you can do virtual look ahead, i.e. roll-outs in the world model conditioned on your policy, and see whether the policy ends up killing 200.000.000 people, or significantly more than one would expect under what humans statistically do in their day-to-day, strange effects at a distance that where not part of the goal state / seed state, etc. - i.e., screen for scheming.
I don’t think you can do anything nontrivially useful in the world if you have a cortex-like algorithm but don’t attach it to any RL, as I explained in my last comment.
I agree (if RL overlaps with the things I listed under GOFAI), but if you can seperate the world model from the RL part at any given point, you can uproot most sociopathic tendencies. Screen for sociopathic behaviour / intents via virtual roll-outs (see above) if you like with honeypots and throw away the policy if it looks suspicious.
“Not an open book” is an understatement. It’s a massive unlabeled data structure. It would be a huge research project to understand it, at best. Then if you want to do anything useful with it, you presumably need to do continual learning, so the data structure keeps changing, so you need to keep pausing the system and doing huge research projects
I think reverse-engineering the human brain’s learning algorithms and scaling them to ASI is much harder than doing MI on a world model that is more compact than current frontier LLMs and which is aligned with the human mind.
But that requires the RL part, and having the AI go out into the world, and do things autonomously, with open-ended continual learning, and no human in the loop. These big-big-big-deal things do not seem compatible with what you seem to be describing.
On the contrary, I believe it is outright impossible to go into the real world without having the brain-like world model down first (or be very tolerant to collateral damage even on ones own side—doesn’t seem like a valid business strategy to me).*
Meanwhile you get outcompeted by the next firm down the street that’s running 100,000 of these things at full speed.
This implies that the only way to escape your scenario is to solve the alignment problem + be faster than everyone else because only than they will copy your recipe, no?
How about we instead go the political route, convince everybody that having AHI gives us all the nice things we need (except for immortality [How is this a good thing on a societal scale again?] and incomprehensible works of super-human intellects), and collectively put an embargo on ASI research?
___
*Could you point me in some direction for a description of how ASI can be achieved in principle? Specifically, what are the requirements on training data? I understand it would involve model-based RL under your view. But will it bootstrap itself from random init or use human experiental priors (e.g., artifacts of human culural evolution / things that cannot be encoded in DNA like spider detectors)? Can it be trained in simulations or will it need real-world interactions?

valentin forch 23 Feb 2026 20:40 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Why we should expect ruthless sociopath ASI
Thanks for taking time to respond.

I am not saying humans don’t use RL. I am trying to say that RL is not what makes us special compared to current SotA (LLM or RL) models. It is our perception. AlphaZero blows us away in closed, non-fuzzy domains. Our ability for abstraction, which I claim is mostly an extension of perception, is what makes us special. Finding a robust hierarchy of coarse grainings in perceptual chaos through self-supervised learning where RL is mostly there to maximize for interestingness. Some call it understanding.
By GOFAI I mean things like MCTS (with good pruning heuristics), TD learning (over a hierarchy of abstract states), production systems, model predictive control, etc. I claim that we don’t need to train a fancy policy network to build very useful stuff. Having a reasonable predictive model for car and environment dynamics with a sampling rate of, say, 50 Hz that does not break under adversarial perturbations over a time horizon of 30 seconds is extremely useful for example.

Yes, a brain-like world model is not an open book. But it should be easier to understand this thing than some world model + some policy network which might be in superposition. When you have a brain-like world model you can combine it for instance with a (potentially malicious) policy and sample future trajectories to evaluate the expected outcomes of the policy.
Yes, ASI will happen, somehow, sometime. I think your concerns are valid in principle and also apply for AHI involving unaligned end-to-end RL, which I think is also much more likely to occur in the next decade. LLMs as of today are not AHI and scaling them won’t get us to AHI. ASI scenarios however are extremely uncertain and there are many more known and unknown dangers to this which dilutes the expected utility of solving the problem you pose.
Do you think there is no grave danger in consequentalist RL if ASI (something super human, incomprehensible by default) is not involved?

valentin forch 22 Feb 2026 6:44 UTC
1 point
0
on: Why we should expect ruthless sociopath ASI
I share your belief that end-to-end RL is ruthless, but I’d be more interested in a version of your argument that does not invoke ASI. ASI implies levels of power that are very dangerous by default.
Human-level artificial intellects (let’s call them AHI) that interface naturally with the internet are more tangible and more likely under our current technological paradigm—and potentially still very dangerous (being kind-of-immortal let’s you play different games). However, if you allow for AHI, there may be a third way to get to it: brain-like perception + GOFAI. I think the human brain’s super-power does not lie in its advanced RL-algorithms to figure out our life policy. Day-to-day decisions are under control of the Basal Ganglia (hierarchical TD-learning), human “deep” planning is some form of very resource-constrained MCTS with good pruning heuristics (e.g., intuitions about interestingness), and all the other advanced stuff we may do is achieved by imitation of planning / decision algorithms and heuristics we deliberately learned, i.e. they are a product of cultural evolution.

So the planning algorithms are already there—we don’t need to reinvent them, and we don’t need to run them in unintelligible deep networks. The hardest part here is getting the hierarchical abstraction for representing our state space, but this is “just” an extension of perception. And we are currently clearly missing brain-like perception. Pairing human levels of robustness in perception with classic planning + the language interface of LLMs would be much more useful than whatever we have today for self-driving and robotics.

The nice thing about brain-like perception + GOFAI is that even if you get there via end-to-end RL, you can throw away the agentic part after you are done training and just keep the perception module that gives you the world model which you then query with GOFAI. The problem becomes representational alignment (make it intelligible) instead of agentic alignment (make it do what you want).