Keeping the humans alive at this point is extremely cheap in terms of fraction of long term resource consumption while avoiding killing humans might substantially reduce the AI’s chance of successful takeover.
I think my crux is I don’t expect keeping humans alive to be very cheap at all, such that we can ignore extremely small kindness/indifference for the purpose of AI alignment, for the reasons described here:
Keeping the humans alive at this point is extremely cheap in terms of fraction of long term resource consumption while avoiding killing humans might substantially reduce the AI’s chance of successful takeover.
I think my crux is I don’t expect keeping humans alive to be very cheap at all, such that we can ignore extremely small kindness/indifference for the purpose of AI alignment, for the reasons described here:
https://www.lesswrong.com/posts/xvBZPEccSfM8Fsobt/what-are-the-best-arguments-for-against-ais-being-slightly#wy9cSASwJCu7bjM6H