Thanks for this! This page will keep me busy for a while. Ethics is my favorite branch of philosophy, which is my favorite hobby (having abandoned the idea of philosophizing for money); and until this page, pondering the use of ethics in the development of Friendly AI was not on my mental radar.
What is meant by that question about “should”? If it’s a general inquiry, I have always considered it like so: If it is said you “should” do action Y, then action Y is thought to cause some outcome X, and this outcome X is thought to be desirable.
In common usage, there’s weak ‘should’ and strong ‘should’.
Weak should is simply a suggestion.
A: “What should I do?” (Please suggest a course of action.)
B: “You should do X.” (Have you considered X?)
Strong should argues for a new course of action instead of the one the listener originally proposed. It comes with the subtext that the listener may initially disagree with the suggested course of action, but suggests that the listener re-evaluate their disagreement in light of the speaker’s conviction.
A: “I want to steal the cookies”
B: “No, you really should not”
There are many possible reasons for why the listener is suggested to re-evaluate their disagreement, all of which are usually subtextual.
a) regret.
A: “I want to steal the cookies”
B: “No, you really should not” [If you were to follow the stated course of action instead of following my suggestion of an alternative course of action, you would regret taking your original course of action.]
b) that the listener would freely choose the alternative course of action if they were better advised
A: “I want to steal the cookies”
B: “No, you really should not, and here’s why:”
OR
A: “I want to steal the cookies”
B: “No, you really should not.” [And I could convince you that not stealing the cookies is the better course of action, if I had the time or inclination to do so, but instead I will ask you to trust me that this would be the case.]
c) benefits or consequences that the listener was previously unaware of
A: “I want to steal the cookies”
B: (threatening) “No, you really should not.” [You may want to reconsider your stated course of action given the new information that I have signalled that I will punish you for that course of action.]
This is clear and well-written, and makes sense to me. I don’t think any of it conflicts with my statement, though (if you mean to correct rather than expand upon). My original statement is just a more general version of your more detailed divisions: in each case, “should” argues for a course of action, given an objective. The objective is often implicit, and sometimes you must infer or guess it.
“You shouldn’t steal those cookies [...if you want to be moral].” More formally stated, perhaps something like: “not doing this will be morally correct; do not do it if you want to be a moral person.”
“You should do X [...if you want to have fun].” More formally: “Doing X will be fun; do it if fun is desired.”
Thanks for this! This page will keep me busy for a while. Ethics is my favorite branch of philosophy, which is my favorite hobby (having abandoned the idea of philosophizing for money); and until this page, pondering the use of ethics in the development of Friendly AI was not on my mental radar.
What is meant by that question about “should”? If it’s a general inquiry, I have always considered it like so: If it is said you “should” do action Y, then action Y is thought to cause some outcome X, and this outcome X is thought to be desirable.
In common usage, there’s weak ‘should’ and strong ‘should’.
Weak should is simply a suggestion.
A: “What should I do?” (Please suggest a course of action.)
B: “You should do X.” (Have you considered X?)
Strong should argues for a new course of action instead of the one the listener originally proposed. It comes with the subtext that the listener may initially disagree with the suggested course of action, but suggests that the listener re-evaluate their disagreement in light of the speaker’s conviction.
A: “I want to steal the cookies”
B: “No, you really should not”
There are many possible reasons for why the listener is suggested to re-evaluate their disagreement, all of which are usually subtextual.
a) regret.
A: “I want to steal the cookies”
B: “No, you really should not” [If you were to follow the stated course of action instead of following my suggestion of an alternative course of action, you would regret taking your original course of action.]
b) that the listener would freely choose the alternative course of action if they were better advised
A: “I want to steal the cookies”
B: “No, you really should not, and here’s why:”
OR
A: “I want to steal the cookies”
B: “No, you really should not.” [And I could convince you that not stealing the cookies is the better course of action, if I had the time or inclination to do so, but instead I will ask you to trust me that this would be the case.]
c) benefits or consequences that the listener was previously unaware of
A: “I want to steal the cookies”
B: (threatening) “No, you really should not.” [You may want to reconsider your stated course of action given the new information that I have signalled that I will punish you for that course of action.]
This is clear and well-written, and makes sense to me. I don’t think any of it conflicts with my statement, though (if you mean to correct rather than expand upon). My original statement is just a more general version of your more detailed divisions: in each case, “should” argues for a course of action, given an objective. The objective is often implicit, and sometimes you must infer or guess it.
“You shouldn’t steal those cookies [...if you want to be moral].” More formally stated, perhaps something like: “not doing this will be morally correct; do not do it if you want to be a moral person.”
“You should do X [...if you want to have fun].” More formally: “Doing X will be fun; do it if fun is desired.”
I misinterpreted your comment as a question, that’s all.