It’s clearer now what you are saying, but I don’t see why you are attributing that point to me specifically (it’s mostly gesturing at value alignment as opposed to intent alignment).
it’s fine if the AI takes over and makes us pets
This sounds like permanent disempowerment. Intent alignment to bad decisions would certainly be a problem, but that doesn’t imply denying opportunity for unbounded growth, where in particular eventually decisions won’t have such issues.
it’s just that the goal is decided by the human
If goals are “decided”, then it’s not value alignment, and bad decisions lead to disasters.
(Overall, this framing seems unhelpful when given in response to someone arguing that values are poorly defined.)
It’s clearer now what you are saying, but I don’t see why you are attributing that point to me specifically (it’s mostly gesturing at value alignment as opposed to intent alignment).
This sounds like permanent disempowerment. Intent alignment to bad decisions would certainly be a problem, but that doesn’t imply denying opportunity for unbounded growth, where in particular eventually decisions won’t have such issues.
If goals are “decided”, then it’s not value alignment, and bad decisions lead to disasters.
(Overall, this framing seems unhelpful when given in response to someone arguing that values are poorly defined.)