So far reception to this post seems fairly mixed, with some upvotes and slightly more downvotes. So apparently I haven’t made the case in a way most people find conclusive — though as yet none of them have bothered to leave a comment explaining their reasons for disagreement. I’m wondering if I should do another post working through the argument in exhaustive detail, showing each of the steps, what facts it relies upon, and where they come from.
I think there are big chunks of the argument missing, which is why I’m commenting. I think those chunks are found in the posts I mentioned. This post focuses on what we’d want an AGI to do and why, and its understanding of that. But the much more debated and questionable step is how to make sure that it wants to do what we want.
So far reception to this post seems fairly mixed, with some upvotes and slightly more downvotes. So apparently I haven’t made the case in a way most people find conclusive — though as yet none of them have bothered to leave a comment explaining their reasons for disagreement. I’m wondering if I should do another post working through the argument in exhaustive detail, showing each of the steps, what facts it relies upon, and where they come from.
I think there are big chunks of the argument missing, which is why I’m commenting. I think those chunks are found in the posts I mentioned. This post focuses on what we’d want an AGI to do and why, and its understanding of that. But the much more debated and questionable step is how to make sure that it wants to do what we want.