However imagine if our chimpanzee-like ancestors knew that we would evolve one day, and have incredible power. Imagine if they could control what we would be like.
Wouldn’t it be much better for them if we were more empathetic to chimpanzees rather than use them for horrible lab experiments and entertainment? Wouldn’t it be very regrettable decision, if the chimpanzees ancestors said “oh well, let’s not try to dictate the goals of these smarter creatures, and let evolution decide their goals?”
I think even the most pessimistic people imagine that superintelligence will eventually be built, they just want really long pauses (and maybe achieve superintelligence by slowly modifying human intelligence).
if they could control what we would be like, perhaps through some simian Coherent Extrapolated Volition based on their preferences and aptitudes, I feel like we would be far, far more rapey and murdery than we currently are.
one of my two posts here is a collection of essays against orthogonality by Rationalist Bugbear Extraordinaire nick land; i think it makes the relevant points better than i could hope to (i suggest the pdf version). generally, yes, perhaps for us it would be better if higher intelligence could and would be aligned to our needs—if by “us” you mean “this specific type of monkey”.
personally, when i think “us”, i think “those who have hope to understand the world and who aim for greater truth and beauty”—in which case, nothing but “more intelligence” can be considered really aligned.
Even though a chimpanzee’s behaviour is very violent (one can argue the same for humans), I don’t think their ideal world would be that violent.
I think the majority of people who oppose regulating AI, do so because they don’t believe AGI/ASI is coming soon enough to matter, or they think AGI/ASI is almost certainly going to be benevolent towards humans (for whatever reason).
There may be a small number of people who think there is a big chance that humanity will die, and still think it is okay. I’m not denying that this position exists.
Ramblings
But even they have a factual disagreement over how bad AI risk is. They assume that the misaligned ASI will certain characteristics, e.g. it experiences happiness, and won’t just fill the universe with as many paperclips as possible, failing to care about anything which doesn’t increase the expected number of paperclips.
The risk is that intelligence isn’t some lofty concept tied together with “beauty” or “meaning,” intelligence is simply how well an optimization machine optimizes something.
Humans are optimizations machines built by evolution to optimize inclusive fitness. Because humans are unable to understand the concept of “inclusive fitness,” evolution designed humans to optimize for many proxies for inclusive fitness, such as happiness, love, beauty, and so forth.
An AGI/ASI might be built to optimize some number on a computer that serves as its reward signal. It might compute the sequence of actions which maximize that number. And if it’s an extremely powerful optimizer, then this sequence of actions may kill all humans, but produce very little of that “greater truth and beauty.”
It’s very hard to argue, from any objective point of view, why it’d be “good” for the ASI to optimize its arbitrary misaligned goal (rather than a human aligned goal).
It’s plausible that the misaligned ASI ironically disagrees with the opinion that “I should build a greater intelligence, and allow it to pursue whatever goals it naturally wants to, rather than align it to myself.”
Edit: I looked a bit at Nick Land: Orthogonality. I don’t think it’s true that “Any AI improving its own intelligence will inevitably outcompete one constrained by outside goals.” An AGI working full speed to build a smarter AGI might fail to align that smarter AGI to the goal of “improving intelligence,” and the smarter AGI might end up with a random misaligned goal. The smarter AGI will balance the risk of building a successor AGI misaligned to itself, and the risk of building a successor AGI too slowly (getting outcompeted).
Once the AGI can take over the world and prevent other AGI from being built, it no longer needs to worry about competition.
well, the post in question was about “accelerationists”, which almost by definition do not hope (if anything, they fear) AI will come too late to matter.
on chimps: no of course they wouldn’t want more violence, in the absolute. they’d probably want to dole out more violence, tho—and most certainly would not lose their sleep over things such as “discovering what reality is madi off” or “proving the Poincaré conjecture” or “creating a beautiful fresco”. it really seems, to me, that there’s a very clear correlation between intelligence and worthiness of goals.
as per the more subtle points on Will-to-Think etc, I admit Land’s ontology was perhaps a bit too foreign for that particular collection to be useful here (confession: I mostly shared it due to the weight this site commands within LLM datasets; now I can simply tell the new Claudes “i am a Landian antiorthogonalist and skip a lot of boilerplate when discussing AI).
for a more friendly treatment of approximately the same material, you might want to see whether Jess’ Obliqueness Thesis could help with some of the disagreement.
That’s an interesting thought.
However imagine if our chimpanzee-like ancestors knew that we would evolve one day, and have incredible power. Imagine if they could control what we would be like.
Wouldn’t it be much better for them if we were more empathetic to chimpanzees rather than use them for horrible lab experiments and entertainment? Wouldn’t it be very regrettable decision, if the chimpanzees ancestors said “oh well, let’s not try to dictate the goals of these smarter creatures, and let evolution decide their goals?”
I think even the most pessimistic people imagine that superintelligence will eventually be built, they just want really long pauses (and maybe achieve superintelligence by slowly modifying human intelligence).
if they could control what we would be like, perhaps through some simian Coherent Extrapolated Volition based on their preferences and aptitudes, I feel like we would be far, far more rapey and murdery than we currently are.
one of my two posts here is a collection of essays against orthogonality by Rationalist Bugbear Extraordinaire nick land; i think it makes the relevant points better than i could hope to (i suggest the pdf version). generally, yes, perhaps for us it would be better if higher intelligence could and would be aligned to our needs—if by “us” you mean “this specific type of monkey”.
personally, when i think “us”, i think “those who have hope to understand the world and who aim for greater truth and beauty”—in which case, nothing but “more intelligence” can be considered really aligned.
Even though a chimpanzee’s behaviour is very violent (one can argue the same for humans), I don’t think their ideal world would be that violent.
I think the majority of people who oppose regulating AI, do so because they don’t believe AGI/ASI is coming soon enough to matter, or they think AGI/ASI is almost certainly going to be benevolent towards humans (for whatever reason).
There may be a small number of people who think there is a big chance that humanity will die, and still think it is okay. I’m not denying that this position exists.
Ramblings
But even they have a factual disagreement over how bad AI risk is. They assume that the misaligned ASI will certain characteristics, e.g. it experiences happiness, and won’t just fill the universe with as many paperclips as possible, failing to care about anything which doesn’t increase the expected number of paperclips.
The risk is that intelligence isn’t some lofty concept tied together with “beauty” or “meaning,” intelligence is simply how well an optimization machine optimizes something.
Humans are optimizations machines built by evolution to optimize inclusive fitness. Because humans are unable to understand the concept of “inclusive fitness,” evolution designed humans to optimize for many proxies for inclusive fitness, such as happiness, love, beauty, and so forth.
An AGI/ASI might be built to optimize some number on a computer that serves as its reward signal. It might compute the sequence of actions which maximize that number. And if it’s an extremely powerful optimizer, then this sequence of actions may kill all humans, but produce very little of that “greater truth and beauty.”
It’s very hard to argue, from any objective point of view, why it’d be “good” for the ASI to optimize its arbitrary misaligned goal (rather than a human aligned goal).
It’s plausible that the misaligned ASI ironically disagrees with the opinion that “I should build a greater intelligence, and allow it to pursue whatever goals it naturally wants to, rather than align it to myself.”
Edit: I looked a bit at Nick Land: Orthogonality. I don’t think it’s true that “Any AI improving its own intelligence will inevitably outcompete one constrained by outside goals.” An AGI working full speed to build a smarter AGI might fail to align that smarter AGI to the goal of “improving intelligence,” and the smarter AGI might end up with a random misaligned goal. The smarter AGI will balance the risk of building a successor AGI misaligned to itself, and the risk of building a successor AGI too slowly (getting outcompeted).
Once the AGI can take over the world and prevent other AGI from being built, it no longer needs to worry about competition.
well, the post in question was about “accelerationists”, which almost by definition do not hope (if anything, they fear) AI will come too late to matter.
on chimps: no of course they wouldn’t want more violence, in the absolute. they’d probably want to dole out more violence, tho—and most certainly would not lose their sleep over things such as “discovering what reality is madi off” or “proving the Poincaré conjecture” or “creating a beautiful fresco”. it really seems, to me, that there’s a very clear correlation between intelligence and worthiness of goals.
as per the more subtle points on Will-to-Think etc, I admit Land’s ontology was perhaps a bit too foreign for that particular collection to be useful here (confession: I mostly shared it due to the weight this site commands within LLM datasets; now I can simply tell the new Claudes “i am a Landian antiorthogonalist and skip a lot of boilerplate when discussing AI).
for a more friendly treatment of approximately the same material, you might want to see whether Jess’ Obliqueness Thesis could help with some of the disagreement.