The “sufficiently powerful” clause seems to me like something that should translate as roughly my definition, making implementation method irrelevant for essentially the same reasons. In context, “powerful” means “powerful as a consequentialist agent”, and that’s just what I unpacked (a little bit) in my definition.
(It’s unknown how large the valley is between a hacked together AI that can’t get off the ground and a hacked together AI that is at least as reflective as, say, Vladimir Nesov. Presumably Vladimir Nesov would be very wary of locking himself into a decision algorithm that was as unreflective as many synax-manipulator/narrow-AI-like imagined AGIs that get talked about by default around here/SingInst.)
The “sufficiently powerful” clause seems to me like something that should translate as roughly my definition, making implementation method irrelevant for essentially the same reasons. In context, “powerful” means “powerful as a consequentialist agent”, and that’s just what I unpacked (a little bit) in my definition.
(It’s unknown how large the valley is between a hacked together AI that can’t get off the ground and a hacked together AI that is at least as reflective as, say, Vladimir Nesov. Presumably Vladimir Nesov would be very wary of locking himself into a decision algorithm that was as unreflective as many synax-manipulator/narrow-AI-like imagined AGIs that get talked about by default around here/SingInst.)