Hm, I’m not sure I understand what’s confusing about this.
First, suppose you’re an approximate utility maximizer. There’s a difference between optimizing the expected utilityE[U(world,action)] and optimizing utility in the expected worldU(E(world),action). In general, in the former case, you’re not necessarily keeping the most-likely worlds in mind; you’re optimizing the worlds in which you can get the most payoff. Those may be specific terrible outcomes you want to avert, or specific high-leverage worlds in which you can win big (e. g., where your startup succeeds).
Choosing which worlds to keep in mind/optimize obviously impacts in which worlds you succeed. (Startup founders who start being risk-averse instead of aiming to win in the worlds in which they can win big lose – because they’re no longer “looking” at the worlds where they succeed, and aren’t shaping their actions to exploit their features.)
Second, human world-models are hierarchical, and your probability distribution over worlds is likely multimodal. So when you pick a set of worlds you care about, you likely pick several modes of this distribution (rather than specific fully specified worlds), characterized by various high-level properties (such as “AI progress continues apace” vs. “DL runs into a wall”). When thinking about one of the high-level-constrained worlds/the neighbourhood of a mode, you further zoom-in on modes corresponding to lower-level properties, and so on.
Which is why you’re not keeping a bunch of basically-the-same expected trajectories in your head, but meaningfully “different” trajectories.
This… all seems to be business-as-usual to me? I may be misunderstanding what you’re getting at.
I wrote this because I am increasingly noticing that the rules for “which worlds to keep in mind/optimize” are often quite different from “which worlds my spreadsheets say are the most likely worlds”. And that this is in conflict with my heuristics which would’ve said “optimize the world-models in your head for being the most accurate ones – the ones that will give you the most accurate answers to most questions” rather than something like “optimize the world-models in your head for being the most useful ones”.
(Though the true answer is some more complicated function combining both practical utility and map-territory correspondence.)
I’m not sure I understand what’s confusing about this.
I will note that what is confusing to one person need not be confusing to another person. In my experience it is a common state of affairs for one person in a conversation to be confused and the other not (whether it be because the latter person is pre-confused, post-confused, or simply because their path to understanding a phenomena didn’t pass through the state of their interlocutor).
It seems probable to me that I have found this subject more confusing than have others.
Hm, I’m not sure I understand what’s confusing about this.
First, suppose you’re an approximate utility maximizer. There’s a difference between optimizing the expected utility E[U(world,action)] and optimizing utility in the expected world U(E(world),action). In general, in the former case, you’re not necessarily keeping the most-likely worlds in mind; you’re optimizing the worlds in which you can get the most payoff. Those may be specific terrible outcomes you want to avert, or specific high-leverage worlds in which you can win big (e. g., where your startup succeeds).
Choosing which worlds to keep in mind/optimize obviously impacts in which worlds you succeed. (Startup founders who start being risk-averse instead of aiming to win in the worlds in which they can win big lose – because they’re no longer “looking” at the worlds where they succeed, and aren’t shaping their actions to exploit their features.)
Second, human world-models are hierarchical, and your probability distribution over worlds is likely multimodal. So when you pick a set of worlds you care about, you likely pick several modes of this distribution (rather than specific fully specified worlds), characterized by various high-level properties (such as “AI progress continues apace” vs. “DL runs into a wall”). When thinking about one of the high-level-constrained worlds/the neighbourhood of a mode, you further zoom-in on modes corresponding to lower-level properties, and so on.
Which is why you’re not keeping a bunch of basically-the-same expected trajectories in your head, but meaningfully “different” trajectories.
This… all seems to be business-as-usual to me? I may be misunderstanding what you’re getting at.
I wrote this because I am increasingly noticing that the rules for “which worlds to keep in mind/optimize” are often quite different from “which worlds my spreadsheets say are the most likely worlds”. And that this is in conflict with my heuristics which would’ve said “optimize the world-models in your head for being the most accurate ones – the ones that will give you the most accurate answers to most questions” rather than something like “optimize the world-models in your head for being the most useful ones”.
(Though the true answer is some more complicated function combining both practical utility and map-territory correspondence.)
I will note that what is confusing to one person need not be confusing to another person. In my experience it is a common state of affairs for one person in a conversation to be confused and the other not (whether it be because the latter person is pre-confused, post-confused, or simply because their path to understanding a phenomena didn’t pass through the state of their interlocutor).
It seems probable to me that I have found this subject more confusing than have others.