I don’t see why agents should be seen as optimisers rather than as achieving some minimal conditions they are satisfied with. The second view seems more consistent both with actual human behaviour and with the concept of bounded rationality as a whole. The minimal conditions seem intuitively related to ensuring existence/propagation (e.g. drinking “enough” water, acquiring “enough” shelter, etc..), but I don’t have a more complete way to put it than that for the moment.
I like this perspective a lot and I think it is indeed more informative than the optimizey perspective wrt agents-that-we-currently-observe-exist. But I don’t expect this perspective to be informative if we build something that is very consequentialist/optimize-y (e.g. ASI).
Imo the best formal grounding for this intuition of agents being exist-y/satisfice-y perspective is FEP. And I do think ASI will be an active inference agent, but that doesn’t really preclude the possibility that it’s also optimize-y; active inference agents behave more and more like EU maximizers under some conditions (namely low ambiguity), and I (tentatively) expect these conditions to be met for ASI.
Some of my uncertainties around this:
Maybe it’s possible to construct an agent that has high optimization power/capability, but is uncertain about what to optimize for. This would probably lead to it acting in less scary ways. Whether this is possible probably depends on the particular selection process the agent went through.
I’m not sure how to think about parts and wholes, in relation to this. For example, can you have an optimize-y thing, that is itself made up of exist-y/satisfice-y things? Can you have an exist-y/satisfice-y thing, that is itself made up of optimizey things? I’m not sure how these kinds of agents compose/interact across scales.
I’m not sure if there’s a meaningful “general intelligence” knob that a process can choose to crank up — even one that nominally exists for exactly that purpose (like a project that tries to build an ASI). This could be cruxy?
Yes, I think it’s cruxy. Could you elaborate on your uncertainty? Even if you’re just sketching out very feathery intuitions.
And I’m curious—if you think this knob doesn’t really meaningfully exist, what do you think current frontier labs are doing/selecting for, and what do you think they’re trying to do/select for? (Like, for example, do you think they’re trying to crank up the general intelligence knob, and that this is a futile task––really, they’re cranking up some different, adjacent knob?)
I’d expect two ASIs with different goals to have similar abstractions about the world, but different abstractions where those abstractions involve themselves/some level of recursive modeling, since those are how internal goals are represented; i.e. imo mutual info between them is low in this case.
Thanks for probing on this! I’m not sure I endorse that strong of a claim anymore. Refining into something I’d endorse more:
The two ASIs exist in the same world, and this world has underlying laws; these laws can be inferred at a sufficiently-high intelligence level, with sufficient data.
But not every single law will be inferred by the two ASIs, because of boundedness. The laws that are inferred will probably depend on their goals. There’s a problem of relevance, here.
But some laws are convergently useful to infer, no matter what goal you have, because there are a small set of bottlenecks to achieving your goals for a wide variety of goals (e.g. resource constraints).
Predicting those convergently useful domains well will lead agents to a similar set of abstractions
The last point seems to be the most important point, but I’m not sure why I buy it. But I do buy it.
I agree that FEP-shaped intuitions are very good for satisfice-ey agents. I’m unconvinced by the concrete mathematical modelling (notably not a fan of Bayesian generative models ) but I find the ideas conceptually useful if you abstract away the implementation.
My scepticism of general intelligence is closely related to your point that ASIs won’t infer every single law. Any given level of complexity in an organism can only acommodate a limited ontology. Of course, you can always “juice up” the agent and give it more resources so it learns a more textured world model. One pseudo-mathematical way to put this is that for every set of abstractions, there exists an abstraction that oblates all of them at once; for a fixed level of complexity however, there exist two sets of abstractions such that neither one clearly dominates.
Our crux might start at “some laws are convergently useful to infer”. One corollary of my last pseudo-mathematical claim is that any bounded agent has to “choose” between incomparable ontologies. The claim in my original post is that the revealed goals an agent is endowed with affects this choice. This amounts to advocating that a focus on the effect of selection pressures on learned abstractions will yield better predictions than a focus on finding “convergent” or “natural” abstractions.
quick addendum: my point feels spiritually related to the idea that “convergent evolution” is an incomplete concept without a specification of the attractor basin.
I expect one strong reason for different ASIs to develop similar abstractions regardless of goals is because they need to predict a bunch of other agents in the world (either humans or other ASIs) and so need to be able to represent the goals of other agents.
I think I phrased my previous comment poorly. What I meant is that if you have developed a set of abstractions relevant to achieving your goals and I want to predict you accurately, then I also need to develop abstractions that are are relevant to achieving your goals. Given a limited representational capacity, this creates a pressure for you to develop representations similar to those of others.
I like this perspective a lot and I think it is indeed more informative than the optimizey perspective wrt agents-that-we-currently-observe-exist. But I don’t expect this perspective to be informative if we build something that is very consequentialist/optimize-y (e.g. ASI).
Imo the best formal grounding for this intuition of agents being exist-y/satisfice-y perspective is FEP. And I do think ASI will be an active inference agent, but that doesn’t really preclude the possibility that it’s also optimize-y; active inference agents behave more and more like EU maximizers under some conditions (namely low ambiguity), and I (tentatively) expect these conditions to be met for ASI.
Some of my uncertainties around this:
Maybe it’s possible to construct an agent that has high optimization power/capability, but is uncertain about what to optimize for. This would probably lead to it acting in less scary ways. Whether this is possible probably depends on the particular selection process the agent went through.
I’m not sure how to think about parts and wholes, in relation to this. For example, can you have an optimize-y thing, that is itself made up of exist-y/satisfice-y things? Can you have an exist-y/satisfice-y thing, that is itself made up of optimizey things? I’m not sure how these kinds of agents compose/interact across scales.
Yes, I think it’s cruxy. Could you elaborate on your uncertainty? Even if you’re just sketching out very feathery intuitions.
And I’m curious—if you think this knob doesn’t really meaningfully exist, what do you think current frontier labs are doing/selecting for, and what do you think they’re trying to do/select for? (Like, for example, do you think they’re trying to crank up the general intelligence knob, and that this is a futile task––really, they’re cranking up some different, adjacent knob?)
Thanks for probing on this! I’m not sure I endorse that strong of a claim anymore. Refining into something I’d endorse more:
The two ASIs exist in the same world, and this world has underlying laws; these laws can be inferred at a sufficiently-high intelligence level, with sufficient data.
But not every single law will be inferred by the two ASIs, because of boundedness. The laws that are inferred will probably depend on their goals. There’s a problem of relevance, here.
But some laws are convergently useful to infer, no matter what goal you have, because there are a small set of bottlenecks to achieving your goals for a wide variety of goals (e.g. resource constraints).
Predicting those convergently useful domains well will lead agents to a similar set of abstractions
The last point seems to be the most important point, but I’m not sure why I buy it. But I do buy it.
I agree that FEP-shaped intuitions are very good for satisfice-ey agents. I’m unconvinced by the concrete mathematical modelling (notably not a fan of Bayesian generative models ) but I find the ideas conceptually useful if you abstract away the implementation.
My scepticism of general intelligence is closely related to your point that ASIs won’t infer every single law. Any given level of complexity in an organism can only acommodate a limited ontology. Of course, you can always “juice up” the agent and give it more resources so it learns a more textured world model. One pseudo-mathematical way to put this is that for every set of abstractions, there exists an abstraction that oblates all of them at once; for a fixed level of complexity however, there exist two sets of abstractions such that neither one clearly dominates.
Our crux might start at “some laws are convergently useful to infer”. One corollary of my last pseudo-mathematical claim is that any bounded agent has to “choose” between incomparable ontologies. The claim in my original post is that the revealed goals an agent is endowed with affects this choice. This amounts to advocating that a focus on the effect of selection pressures on learned abstractions will yield better predictions than a focus on finding “convergent” or “natural” abstractions.
quick addendum: my point feels spiritually related to the idea that “convergent evolution” is an incomplete concept without a specification of the attractor basin.
I expect one strong reason for different ASIs to develop similar abstractions regardless of goals is because they need to predict a bunch of other agents in the world (either humans or other ASIs) and so need to be able to represent the goals of other agents.
why are the goals of other agents more likely to have natural convergent representations of them than other things in the world?
I think I phrased my previous comment poorly. What I meant is that if you have developed a set of abstractions relevant to achieving your goals and I want to predict you accurately, then I also need to develop abstractions that are are relevant to achieving your goals. Given a limited representational capacity, this creates a pressure for you to develop representations similar to those of others.