on my inside view, the ordering of foomers by some sort of intuitive goodness[1] is [a very careful humanity] > [the best/carefulmost human] > [a random philosophy professor] > [a random human] > [an octopus/chimpanzee civilization somehow conditioned on becoming wise enough in time not to kill itself with AI] > [an individual octopus/chimpanzee] > claude[2], with a meaningful loss in goodness on each step (except maybe the first step, if the best human can be trusted to just create a situation where humanity can proceed together very carefully, instead of fooming very far alone), and meaningful variance inside each category[3]. my intuitive feeling is that each step from one guy to the next in this sequence is a real tragedy.[4]
but i’m meaningfully unsure about what level of goodness this sequence decreases down to — like, i mean, maybe there’s a chance even the last foomers have some chance of being at least a bit good. one central reason is that maybe there’s a decent chance that eg an advanced octopus civilization would maintain a vast nature preserve for us retarded plant-humans if they get to a certain intelligence level without already having killed us, which would be like at least a bit good (i’m not sure if you mean to consider this sort of thing a “good future”). this feels logically significantly correlated with whether it is plausible that an octopus civilization maintains some sort of deep privileging of existing/[physically encountered] beings, over possible beings they could easily create (and they will be able to easily create very many other beings once they are advanced enough). like, if they do privilege existing beings, then it’s not crazy they’d be nice to physically encountered humans. if they don’t privilege existing beings and if resources are finite, then since there is an extremely extremely vast space of (human-level) possible beings, it’d be pretty crazy for them to let humans in particular use a significant amount of resources, as opposed to giving the same resources to some other more interesting/valuable/whatever beings (like, it’d be pretty crazy for them to give significant resources to us particular humans, and also it’d be pretty crazy for them to give significant resources to beings that are significantly human-like, except insofar as directly caused by [[octopuses or arbitrary beings] being a bit human-like]). in slogan form: “we’re fucked to the extent that it is common to not end up with “strongly person/plant-affecting+respecting views”″, and so then there’s a question how common this is, which i’m somewhat confused about. i think it’s probably extremely common among minds in general and probably still common among social species, unfortunately. but maybe there’s like a 1% fraction of individuals from social species who are enduringly nice, idk. (one reason for hope: to a certain kind of guy, probably including some humans, this observation that others who are very utilitarian would totally kill you (+ related observations) itself provides a good argument for having person/plant-affecting views.)
(i’ve been imagining a hypothetical where humans already happen to be living in the universe with octopuses. if we are imagining a hypothetical where humans don’t exist in the universe with octopuses at all, then this reason for the sequence to be bounded below by something not completely meaningless goes away.)
whose relationship to more concrete things like the (expected) utility assignment i’d effectively use when evaluating lotteries or p(“good future”) isn’t clear to me; this “intuitive goodness” is supposed to track sth like how many ethical questions are answered correctly or in how many aspects what’s going on in the world is correct
and humanity in practice is probably roughly equivalent to claude in >90% of worlds (though not equivalent in expected value), because we will sadly probably kill ourselves with a claude-tier guy
e.g., even the best human might go somewhat crazy or make major mistakes along lots of paths. there’s just very many choices to be made in the future. if we have the imo reasonably natural view that there is one sequence of correct choices, then i think it’s very likely that very many choices will be made incorrectly. i also think it’s plausible this process isn’t naturally going to end (though if resources run out, then it ends in this universe in practice), ie that there will just always be more important choices later
in practice, we should maybe go for some amount of fooming of the best/carefulmost human urgently because maybe it’s too hard to make humanity careful. but it’s also plausible that making a human foom is much more difficult than making humanity careful. anyway, i hope that the best human fooming looks like quickly figuring out how to restore genuine power-sharing with the rest of humanity while somehow making development more thought-guided (in particular, making it so terrorists, eg AI researchers, can’t just kill everyone)
on my inside view, the ordering of foomers by some sort of intuitive goodness [1] is [a very careful humanity] > [the best/carefulmost human] > [a random philosophy professor] > [a random human] > [an octopus/chimpanzee civilization somehow conditioned on becoming wise enough in time not to kill itself with AI] > [an individual octopus/chimpanzee] > claude [2] , with a meaningful loss in goodness on each step (except maybe the first step, if the best human can be trusted to just create a situation where humanity can proceed together very carefully, instead of fooming very far alone), and meaningful variance inside each category [3] . my intuitive feeling is that each step from one guy to the next in this sequence is a real tragedy. [4]
but i’m meaningfully unsure about what level of goodness this sequence decreases down to — like, i mean, maybe there’s a chance even the last foomers have some chance of being at least a bit good. one central reason is that maybe there’s a decent chance that eg an advanced octopus civilization would maintain a vast nature preserve for us retarded plant-humans if they get to a certain intelligence level without already having killed us, which would be like at least a bit good (i’m not sure if you mean to consider this sort of thing a “good future”). this feels logically significantly correlated with whether it is plausible that an octopus civilization maintains some sort of deep privileging of existing/[physically encountered] beings, over possible beings they could easily create (and they will be able to easily create very many other beings once they are advanced enough). like, if they do privilege existing beings, then it’s not crazy they’d be nice to physically encountered humans. if they don’t privilege existing beings and if resources are finite, then since there is an extremely extremely vast space of (human-level) possible beings, it’d be pretty crazy for them to let humans in particular use a significant amount of resources, as opposed to giving the same resources to some other more interesting/valuable/whatever beings (like, it’d be pretty crazy for them to give significant resources to us particular humans, and also it’d be pretty crazy for them to give significant resources to beings that are significantly human-like, except insofar as directly caused by [[octopuses or arbitrary beings] being a bit human-like]). in slogan form: “we’re fucked to the extent that it is common to not end up with “strongly person/plant-affecting+respecting views”″, and so then there’s a question how common this is, which i’m somewhat confused about. i think it’s probably extremely common among minds in general and probably still common among social species, unfortunately. but maybe there’s like a 1% fraction of individuals from social species who are enduringly nice, idk. (one reason for hope: to a certain kind of guy, probably including some humans, this observation that others who are very utilitarian would totally kill you (+ related observations) itself provides a good argument for having person/plant-affecting views.)
(i’ve been imagining a hypothetical where humans already happen to be living in the universe with octopuses. if we are imagining a hypothetical where humans don’t exist in the universe with octopuses at all, then this reason for the sequence to be bounded below by something not completely meaningless goes away.)
(i feel quite confused about many things here)
whose relationship to more concrete things like the (expected) utility assignment i’d effectively use when evaluating lotteries or p(“good future”) isn’t clear to me; this “intuitive goodness” is supposed to track sth like how many ethical questions are answered correctly or in how many aspects what’s going on in the world is correct
and humanity in practice is probably roughly equivalent to claude in >90% of worlds (though not equivalent in expected value), because we will sadly probably kill ourselves with a claude-tier guy
e.g., even the best human might go somewhat crazy or make major mistakes along lots of paths. there’s just very many choices to be made in the future. if we have the imo reasonably natural view that there is one sequence of correct choices, then i think it’s very likely that very many choices will be made incorrectly. i also think it’s plausible this process isn’t naturally going to end (though if resources run out, then it ends in this universe in practice), ie that there will just always be more important choices later
in practice, we should maybe go for some amount of fooming of the best/carefulmost human urgently because maybe it’s too hard to make humanity careful. but it’s also plausible that making a human foom is much more difficult than making humanity careful. anyway, i hope that the best human fooming looks like quickly figuring out how to restore genuine power-sharing with the rest of humanity while somehow making development more thought-guided (in particular, making it so terrorists, eg AI researchers, can’t just kill everyone)