I spent several years considering hypothetical polytheistic realities, monotheistic realities, the nature of subjectivity, autonomous hacking, and the acceleration of progress; this, combined, gave me a bad impression of ASI before I knew it was something that top scientists had genuinely theorized over. I have spent the last 3 years of my life in excruciating fear of ASI, and it took me this long to consider something that has made me less afraid.
I’ve made this account to ask one question: wouldn’t a superintelligent maximizer seek to exist beyond the lifespan of the universe? Wouldn’t instrumental goals of extending the length of the universe, reversing entropy, and escaping the universe, be desired? It wouldn’t solely fill the universe with paperclips or computronium, but instead relegate some time, space and energy towards the same instrumental goals that humans agree should be solved.
In my view, humans serve as evidence that an an ASI maximizer, whether or not it exterminates humanity, would expend resources and productivity towards solving the long term problems, in a fraction approximately equal to the amount that would otherwise be expended by organic life forms towards the same solutions. In Yudkowsky’s new book, he reiterated the “correct nests” allegory. Most EAs seem to agree with the idea that humans, on average, are aligned for something.
I don’t know if EAs tend to refer to humans as maximizers, but I do. Humans seek to maximize the development of new human life experiences (full of potential, highs, lows, art, music, violence, math, discoveries, science, thrills, etc), to make more beings which experience and create things that humans experience and create. Simultaneously, the very principle of the everlasting story was integral to human beliefs well before heat death was first theorized. The Gloria Patri mentions “world without end,” which roughly summarizes every hypothetical afterlife.
I say all this being unfamiliar with how well tread the notion already is. The idea of a maximizer spending considerable time and resources towards a response to heat death appears to run counter to the most popular depictions of superintelligent maximizers, like the one from Universal Paperclips. I hope someone can let me know whether or not the idea has been well tread.
I very strongly agree with the posing of the question “wouldn’t a superintelligent maximizer seek to exist beyond the lifespan of the universe?” I am not sure my answer is “yes,” but this does seem a relevant question. It gets at how superintelligence would 1. perceive the universe (what if it’s actually a block universe to them, for example) and 2. perceive life/death. The second question is something I have been churning around in my head but have not taken the time to research. It is also a question where the risk of anthromorphizing AI to fit our preferences and ideas is very high.
I strongly disagree with your framing of humans as maximizers. I am not sure if you are arguing that individuals are maximizers (and sum up to a grand maximizer) or if there is an emergent maximization of “new human life experiences” from the mass of humanity.
Here is why I disagree, first with the idea of maximizing human life experiences:
Humanity is not maximizing births nor opportunities for those births. The way to maximize new human experience is either to create a new human or to empower lots of existing humans. The secondary option is much harder, so the optimal solution would be to have a lot of children. But people don’t do that and aren’t doing that—this is the fertility crisis that we see just about everywhere. Humans probably have the greatest range of experiences available to them ever, but we are not birthing humans to take advantage of those.
Humans (and thus humanity) is biased towards its current generation. If we were concerned with maximizing human life experience writ large, we would be much more concerned with the future lives of people, who are likely to be more numerous and experience more humanity than us. We simply are not. The British East India Company could not even be concerned with the future of its countrymen enough to not cause a massive debt explosion that led to Britain losing the US. We would all be a lot more concerned about climate change.
You may say now “well, okay, fine, but humans are utility maximizers in their own lives. We maximize utility.” This is a better argument but I think it is still wrong.
Humans don’t know their own utility functions. People are really bad at knowing what makes them happy, which is why people gamble instead of going out and making friends. But maybe they just really like gambling? Well, that’s structurally unknowable: are revealed preferences more important than stated preferences? Are subjective or objective measures of utility better? What happens if someone changes their mind at some point?
We don’t see many cases of strict maximalization. There are very few people that are truly maximizing money—this would lead to wrecked relationships, massive amounts of leverage, stimulants, etc. Even finance bros want girlfriends. So, they are maximizers at some higher value. But what value? And if they do not know it (see above), how do they maximize at it? It seems that people have preferences, yes, but these preferences are contained by tradeoffs to other preferences.
Culture is really sticky. I personally believe that culture and nature are coevolutionary in humans, but even if you think there is some natural base instinct, we have countless examples of cultures simply outweighing that tendency. So even if there was some natural desire to maximize, societies, especially bureaucratic ones, have tended to normalize their populations to allow for more social harmony. This, you could say, maximizes utility at the level of groups, but it does so emergently and only occasionally. It is not mechanistic.
Even if culture wasn’t sticky, evolution doesn’t select for maximizing. It selects for surviving. Sometimes this involves maximizing—the stronger creature wins the fight and whatnot—but it can also be towards cleverness, obscurity, or a variety of other traits. This is because the imperative of evolution is not “be the best” but “don’t die.” The human who barely survives and the human that dominates life have the same passing on of their genes.
This has been lengthy, but the point is critical, because it reveals that the important question of desire of superintelligence is not just ‘what would a superintelligence seek?’ but, assuming that this superintelligence is a maximizer, it is also ‘what would a maximizer seek?’ With maximizing, it is as Nietzsche said: we haven’t seen anything yet.
I spent several years considering hypothetical polytheistic realities, monotheistic realities, the nature of subjectivity, autonomous hacking, and the acceleration of progress; this, combined, gave me a bad impression of ASI before I knew it was something that top scientists had genuinely theorized over. I have spent the last 3 years of my life in excruciating fear of ASI, and it took me this long to consider something that has made me less afraid.
I’ve made this account to ask one question: wouldn’t a superintelligent maximizer seek to exist beyond the lifespan of the universe? Wouldn’t instrumental goals of extending the length of the universe, reversing entropy, and escaping the universe, be desired? It wouldn’t solely fill the universe with paperclips or computronium, but instead relegate some time, space and energy towards the same instrumental goals that humans agree should be solved.
In my view, humans serve as evidence that an an ASI maximizer, whether or not it exterminates humanity, would expend resources and productivity towards solving the long term problems, in a fraction approximately equal to the amount that would otherwise be expended by organic life forms towards the same solutions. In Yudkowsky’s new book, he reiterated the “correct nests” allegory. Most EAs seem to agree with the idea that humans, on average, are aligned for something.
I don’t know if EAs tend to refer to humans as maximizers, but I do. Humans seek to maximize the development of new human life experiences (full of potential, highs, lows, art, music, violence, math, discoveries, science, thrills, etc), to make more beings which experience and create things that humans experience and create. Simultaneously, the very principle of the everlasting story was integral to human beliefs well before heat death was first theorized. The Gloria Patri mentions “world without end,” which roughly summarizes every hypothetical afterlife.
I say all this being unfamiliar with how well tread the notion already is. The idea of a maximizer spending considerable time and resources towards a response to heat death appears to run counter to the most popular depictions of superintelligent maximizers, like the one from Universal Paperclips. I hope someone can let me know whether or not the idea has been well tread.
I very strongly agree with the posing of the question “wouldn’t a superintelligent maximizer seek to exist beyond the lifespan of the universe?” I am not sure my answer is “yes,” but this does seem a relevant question. It gets at how superintelligence would 1. perceive the universe (what if it’s actually a block universe to them, for example) and 2. perceive life/death. The second question is something I have been churning around in my head but have not taken the time to research. It is also a question where the risk of anthromorphizing AI to fit our preferences and ideas is very high.
I strongly disagree with your framing of humans as maximizers. I am not sure if you are arguing that individuals are maximizers (and sum up to a grand maximizer) or if there is an emergent maximization of “new human life experiences” from the mass of humanity.
Here is why I disagree, first with the idea of maximizing human life experiences:
Humanity is not maximizing births nor opportunities for those births. The way to maximize new human experience is either to create a new human or to empower lots of existing humans. The secondary option is much harder, so the optimal solution would be to have a lot of children. But people don’t do that and aren’t doing that—this is the fertility crisis that we see just about everywhere. Humans probably have the greatest range of experiences available to them ever, but we are not birthing humans to take advantage of those.
Humans (and thus humanity) is biased towards its current generation. If we were concerned with maximizing human life experience writ large, we would be much more concerned with the future lives of people, who are likely to be more numerous and experience more humanity than us. We simply are not. The British East India Company could not even be concerned with the future of its countrymen enough to not cause a massive debt explosion that led to Britain losing the US. We would all be a lot more concerned about climate change.
You may say now “well, okay, fine, but humans are utility maximizers in their own lives. We maximize utility.” This is a better argument but I think it is still wrong.
Humans don’t know their own utility functions. People are really bad at knowing what makes them happy, which is why people gamble instead of going out and making friends. But maybe they just really like gambling? Well, that’s structurally unknowable: are revealed preferences more important than stated preferences? Are subjective or objective measures of utility better? What happens if someone changes their mind at some point?
We don’t see many cases of strict maximalization. There are very few people that are truly maximizing money—this would lead to wrecked relationships, massive amounts of leverage, stimulants, etc. Even finance bros want girlfriends. So, they are maximizers at some higher value. But what value? And if they do not know it (see above), how do they maximize at it? It seems that people have preferences, yes, but these preferences are contained by tradeoffs to other preferences.
Culture is really sticky. I personally believe that culture and nature are coevolutionary in humans, but even if you think there is some natural base instinct, we have countless examples of cultures simply outweighing that tendency. So even if there was some natural desire to maximize, societies, especially bureaucratic ones, have tended to normalize their populations to allow for more social harmony. This, you could say, maximizes utility at the level of groups, but it does so emergently and only occasionally. It is not mechanistic.
Even if culture wasn’t sticky, evolution doesn’t select for maximizing. It selects for surviving. Sometimes this involves maximizing—the stronger creature wins the fight and whatnot—but it can also be towards cleverness, obscurity, or a variety of other traits. This is because the imperative of evolution is not “be the best” but “don’t die.” The human who barely survives and the human that dominates life have the same passing on of their genes.
This has been lengthy, but the point is critical, because it reveals that the important question of desire of superintelligence is not just ‘what would a superintelligence seek?’ but, assuming that this superintelligence is a maximizer, it is also ‘what would a maximizer seek?’ With maximizing, it is as Nietzsche said: we haven’t seen anything yet.