Rational agents/things are not synonymous with good things. A paperclip maximizer is the canonical example of an agent acting rationally. As far as most people are concerned, including me, the paperclip maximizer is not acting in a good way.
Although these days Roko is probably uninterested in whether I agree with him, I agree with that passage.
According to my definition, “epistemically rational” means “effective at achieving one’s goals”. If the goals are incompatible with my goals, I’m going to hope that the agent remains epistemically irrational.
(Garcia used “intelligent” and “ethical” for my “epistemically rational” and “has goals compatible with my goals”.)
Since 1971, Garcia’s been stressing that increasing a person’s epistemic rationality increases that person’s capacity for good and capacity for evil, so you should try to determine whether the person will do good or do evil before you increase the epistemic rationality of the person. (Of course your definition of “good” might differ from mine.)
The smartest person (Ph. D. in math from a top program, successful entrepreneur) I ever met before I met Eliezer was probably unethical or evil. I say “probably” only to highlight that one cannot be highly confident of one’s judgement about someone’s ethics or evilness even if one has observed them closely. But most people here would probably agree with me that this person was unethical or evil.
Although these days Roko is probably uninterested in whether I agree with him, I agree with that passage.
According to my definition, “epistemically rational” means “effective at achieving one’s goals”. If the goals are incompatible with my goals, I’m going to hope that the agent remains epistemically irrational.
(Garcia used “intelligent” and “ethical” for my “epistemically rational” and “has goals compatible with my goals”.)
Since 1971, Garcia’s been stressing that increasing a person’s epistemic rationality increases that person’s capacity for good and capacity for evil, so you should try to determine whether the person will do good or do evil before you increase the epistemic rationality of the person. (Of course your definition of “good” might differ from mine.)
The smartest person (Ph. D. in math from a top program, successful entrepreneur) I ever met before I met Eliezer was probably unethical or evil. I say “probably” only to highlight that one cannot be highly confident of one’s judgement about someone’s ethics or evilness even if one has observed them closely. But most people here would probably agree with me that this person was unethical or evil.
no! not at all.