How worth doing something is depends on the product of its success chance and its payoff, but it’s not clear that anticipations of goodness scale as much as consequences of goodness do, which could lead to predictably unmotivating plans (which ‘should be’ motivating).
However, I have a question. How would you distinguish a case where anticipations of goodness are not matching expected consequences of goodness (aside: I think “goodness of consequences” is a less awkward / more accurate formulation here, actually), from a case where expected goodness of consequences differs from claimed expected goodness of consequences?
In other words:
Alice: You should work on Project X!
Bob: Why?
Alice: Project X is very important! If accomplished, the consequences will be [stuff]!
Bob: Really?
Alice: Yeah! Because of [reasons]!
Bob, thinking: That sounds dubious but I can’t really explain why…
Bob: I am convinced.
Bob, thinking: I am not convinced…
Alice: Great! Then you’ll work on Project X, right? Because it’s so important?
Bob, thinking: There’s no good reason for me to say no…
Bob: Of course I’ll work on Project X.
Bob, thinking: I won’t work on Project X.
Later:
Alice: Bob, why haven’t you been working on Project X?!
Bob, thinking: If I tell her that I was never convinced in the first place, that will look bad…
Bob: Uh, motivation…al… problems. My, uh, System 1. And stuff. You know how it is.
Alice: Confounded System 1! Don’t worry, Bob, I’ll figure out a way around this problem!
Bob: Great! I look forward to being able to work on Project X, which is important.
Example: you can think AGI alignment is worth working hard on even if (a) you only assign a 30% probability to success, and (b) you’re not incredibly excited and overjoyed to be working on it.
By assumption, this also isn’t a case where you find the work so inherently bleh that it’s actually not a good fit for you and you shouldn’t try. If you’d be sufficiently excited in the world where you thought the success odds were 70%, and your system 2 doesn’t think the difference between 70% and 30% odds is decision-relevant in this case, then it seems like something’s going wrong if you’re insufficiently motivated in the 30% case.
I appreciate you being the voice of reason, but I’m actually with Eliezer on this one. (a) isn’t an immutable fact about you, it’s a matter of your policy—what things you allow yourself to get excited about. And if you end up doing nothing exciting for years on end, like most people, then your policy might be suboptimal.
Er, but… (a) is a stipulation that Eliezer himself specified. In his post. I was quoting him!
Eliezer was asking the question: how do you get people to keep working on something that they’re not “personally madly driven to accomplish”, etc. If you can make yourself (or someone else) “madly driven to accomplish” the thing, well, then… that answers that? I… don’t see how your comment isn’t a non sequitur :(
And if you end up doing nothing exciting for years on end, like most people, then your policy might be suboptimal.
True, of course, I agree with this! But… who on earth said anything about doing nothing exciting?
Suppose you are doing something exciting, but then there’s some other thing that you’re not excited about, but that you (allegedly) think that it’s important to keep working on, but that you don’t keep working on, due to lack of motivation. Your criticism doesn’t apply, but Eliezer’s question does.
But… what’s wrong with this?
If:
(a) Someone is “not personally madly driven to accomplish the thing”, and
(b) They are not being paid money to do it, and
(c) They don’t think it’s pretty likely to succeed…
… then… why should they keep working on it?
How worth doing something is depends on the product of its success chance and its payoff, but it’s not clear that anticipations of goodness scale as much as consequences of goodness do, which could lead to predictably unmotivating plans (which ‘should be’ motivating).
This is a reasonable point.
However, I have a question. How would you distinguish a case where anticipations of goodness are not matching expected consequences of goodness (aside: I think “goodness of consequences” is a less awkward / more accurate formulation here, actually), from a case where expected goodness of consequences differs from claimed expected goodness of consequences?
In other words:
Alice: You should work on Project X!
Bob: Why?
Alice: Project X is very important! If accomplished, the consequences will be [stuff]!
Bob: Really?
Alice: Yeah! Because of [reasons]!
Bob, thinking: That sounds dubious but I can’t really explain why…
Bob: I am convinced.
Bob, thinking: I am not convinced…
Alice: Great! Then you’ll work on Project X, right? Because it’s so important?
Bob, thinking: There’s no good reason for me to say no…
Bob: Of course I’ll work on Project X.
Bob, thinking: I won’t work on Project X.
Later:
Alice: Bob, why haven’t you been working on Project X?!
Bob, thinking: If I tell her that I was never convinced in the first place, that will look bad…
Bob: Uh, motivation…al… problems. My, uh, System 1. And stuff. You know how it is.
Alice: Confounded System 1! Don’t worry, Bob, I’ll figure out a way around this problem!
Bob: Great! I look forward to being able to work on Project X, which is important.
Bob, thinking: Phew…
Edit: See also “epistemic learned helplessness” (which, as Scott points out, is exactly the correct response much of the time).
Example: you can think AGI alignment is worth working hard on even if (a) you only assign a 30% probability to success, and (b) you’re not incredibly excited and overjoyed to be working on it.
By assumption, this also isn’t a case where you find the work so inherently bleh that it’s actually not a good fit for you and you shouldn’t try. If you’d be sufficiently excited in the world where you thought the success odds were 70%, and your system 2 doesn’t think the difference between 70% and 30% odds is decision-relevant in this case, then it seems like something’s going wrong if you’re insufficiently motivated in the 30% case.
You seem to be conflating “madly driven” with “incredibly excited and overjoyed”, and also “expect to succeed” with “excited”?
I appreciate you being the voice of reason, but I’m actually with Eliezer on this one. (a) isn’t an immutable fact about you, it’s a matter of your policy—what things you allow yourself to get excited about. And if you end up doing nothing exciting for years on end, like most people, then your policy might be suboptimal.
Er, but… (a) is a stipulation that Eliezer himself specified. In his post. I was quoting him!
Eliezer was asking the question: how do you get people to keep working on something that they’re not “personally madly driven to accomplish”, etc. If you can make yourself (or someone else) “madly driven to accomplish” the thing, well, then… that answers that? I… don’t see how your comment isn’t a non sequitur :(
True, of course, I agree with this! But… who on earth said anything about doing nothing exciting?
Suppose you are doing something exciting, but then there’s some other thing that you’re not excited about, but that you (allegedly) think that it’s important to keep working on, but that you don’t keep working on, due to lack of motivation. Your criticism doesn’t apply, but Eliezer’s question does.