One experiment is worth more than all the opinions.
IMHO, no, there is not a coherent argument for the human worth hypothesis. My money is on it being disproven.
But, I assert the human worth hypothesis is the explicit belief of smart people like Scott Aaronson and the implicit belief of a lot of other people who think AI will be just fine. As Scott says Orthogonality is “a central linchpin” of the doom argument.
Can we be more clear about what people do believe at get at it with experiments?? That’s the question I’m asking.
It’s hard to construct experiments to prove all kinds of minds are possible, that is, to prove Orthogonality.
I think it may be less hard to quantify what an agent values. (Deception, yes. Still...)
Okay, a “hard zone” rather than a no-go zone. Which begs the question “How hard?” and consequently how much comfort should one take in the belief?
Thank you for reading and commenting.