analogy doesn’t by itself seem compelling, given that humanity as a whole (rather than particular groups within it or individuals) is a sufficiently salient thing in the world
etc. because don’t you think humanity from the point of view of ASI at the ‘branch point’ of deciding its continued existence may well be on the order of importance of an individual to a billionaire?
Minimal alignment is a necessary premise, I’m not saying humanity’s salience as a philanthropic cause is universally compelling to AIs. There is a number of observations that make this case stronger: the language prior in LLMs, preference training for chatbots, first AGIs might need nothing fundamentally different from this, and AGI-driven Pause on superintelligence increases the chances that the eventual superintelligences in charge are strongly value aligned with these first AGIs. Then in addition to the premise of a minimally aligned superintelligence, there’s the essentially arbitrarily small cost of a permanently disempowered future of humanity.
So the overall argument indeed doesn’t work without humanity actually being sufficiently salient to the values of superintelligences that are likely to end up in charge, and the argument from low cost only helps up to a point.
I agree, I’m probably not as sure about sufficient alignment but yes.
I suppose this also assumes a kind of orderly world where it actually is within the means of humanity, AGIs (within their Molochian frames), and trivial means of later superintelligences to preserve humans. (US office construction spending and data center spending are about to cross https://x.com/LanceRoberts/status/1953042283709768078 .)
Thanks for the reply, I have gripes with
etc. because don’t you think humanity from the point of view of ASI at the ‘branch point’ of deciding its continued existence may well be on the order of importance of an individual to a billionaire?
Minimal alignment is a necessary premise, I’m not saying humanity’s salience as a philanthropic cause is universally compelling to AIs. There is a number of observations that make this case stronger: the language prior in LLMs, preference training for chatbots, first AGIs might need nothing fundamentally different from this, and AGI-driven Pause on superintelligence increases the chances that the eventual superintelligences in charge are strongly value aligned with these first AGIs. Then in addition to the premise of a minimally aligned superintelligence, there’s the essentially arbitrarily small cost of a permanently disempowered future of humanity.
So the overall argument indeed doesn’t work without humanity actually being sufficiently salient to the values of superintelligences that are likely to end up in charge, and the argument from low cost only helps up to a point.
I agree, I’m probably not as sure about sufficient alignment but yes.
I suppose this also assumes a kind of orderly world where it actually is within the means of humanity, AGIs (within their Molochian frames), and trivial means of later superintelligences to preserve humans. (US office construction spending and data center spending are about to cross https://x.com/LanceRoberts/status/1953042283709768078 .)