I’d brought up a version of the tool/agent distinction, and was told firmly that people aren’t smart or fast enough to direct an AI. (Sorry, this is from memory—I don’t have the foggiest how to do an efficient search to find that exchange.)
I’m not sure that’s a complete answer—how possible is it to augment a human towards being able to manage an AI? On the other hand, a human like that isn’t going to be much like humans 1.0, so problems of Friendliness are still in play.
Perhaps what’s needed is building akrasia into the world—a resistance to sudden change. This has its own risks, but sudden existential threats are rare. [1]
At this point, I think the work on teaching rationality is more reliably important than the work on FAI. FAI involves some long inferential chains. The idea that people could improve their lives a lot by thinking more carefully about what they’re doing and acting on those thoughts (with willingness to take feedback) is a much more plausible idea, even if you factor in the idea that rationality can be taught.
[1] Good enough for fiction—we’re already living in a world like that. We call the built-in akrasia Murphy.
You may be thinking of this exchange, which I found only because I remembered having been involved in it.
I continue to think that “tool” is a bad term to use here, because people’s understanding of what it refers to vary so relevantly.
As for what is valuable work… hm.
I think teaching people to reason in truth-preserving and value-preserving ways is worth doing. I think formalizing a decision theory that captures universal human intuitions about what the right thing to do in various situations is worth doing. I think formalizing a decision theory that captures non-universal but extant “right thing” intuitions is potentially worth doing, but requires a lot of auxiliary work to actually be worth doing. I think formalizing a decision theory that arrives at judgments about the right thing to do in various situations where those judgments are counterintuitive for most/all humans but reliably lead, if implemented, to results that those same humans reliably endorse more the results of their intuitive judgments is worth doing. I think building systems that can solve real-world problems efficiently is worth doing, all else being equal, though I agree that powerful tools frequently have unexpected consequences that create worse problems than they solve, in which case it’s not worth doing. I think designing frameworks within which problem-solving systems can be built, such that the chances of unexpected negative consequences are lower inside that framework than outside of it, is worth doing.
I don’t find it likely that SI is actually doing any of those things particularly more effectively than other organizations.
Thanks for the link—that was what I was thinking of.
Do you have other organizations which teach rationality in mind? Offhand, the only thing I can think of is cognitive behavioral therapy, and it’s not exactly an organization.
The article is interesting, but I’m not sure it is relevant as the humans involved weren’t directing or monitoring the overall process, just taking part in it. Analogously even if an AGI requires my assistance/authorization to do certain things, that doesn’t give me any control over it unless I understand the consequences.
I’d brought up a version of the tool/agent distinction, and was told firmly that people aren’t smart or fast enough to direct an AI. (Sorry, this is from memory—I don’t have the foggiest how to do an efficient search to find that exchange.)
I’m not sure that’s a complete answer—how possible is it to augment a human towards being able to manage an AI? On the other hand, a human like that isn’t going to be much like humans 1.0, so problems of Friendliness are still in play.
Perhaps what’s needed is building akrasia into the world—a resistance to sudden change. This has its own risks, but sudden existential threats are rare. [1]
At this point, I think the work on teaching rationality is more reliably important than the work on FAI. FAI involves some long inferential chains. The idea that people could improve their lives a lot by thinking more carefully about what they’re doing and acting on those thoughts (with willingness to take feedback) is a much more plausible idea, even if you factor in the idea that rationality can be taught.
[1] Good enough for fiction—we’re already living in a world like that. We call the built-in akrasia Murphy.
You may be thinking of this exchange, which I found only because I remembered having been involved in it.
I continue to think that “tool” is a bad term to use here, because people’s understanding of what it refers to vary so relevantly.
As for what is valuable work… hm.
I think teaching people to reason in truth-preserving and value-preserving ways is worth doing.
I think formalizing a decision theory that captures universal human intuitions about what the right thing to do in various situations is worth doing.
I think formalizing a decision theory that captures non-universal but extant “right thing” intuitions is potentially worth doing, but requires a lot of auxiliary work to actually be worth doing.
I think formalizing a decision theory that arrives at judgments about the right thing to do in various situations where those judgments are counterintuitive for most/all humans but reliably lead, if implemented, to results that those same humans reliably endorse more the results of their intuitive judgments is worth doing.
I think building systems that can solve real-world problems efficiently is worth doing, all else being equal, though I agree that powerful tools frequently have unexpected consequences that create worse problems than they solve, in which case it’s not worth doing.
I think designing frameworks within which problem-solving systems can be built, such that the chances of unexpected negative consequences are lower inside that framework than outside of it, is worth doing.
I don’t find it likely that SI is actually doing any of those things particularly more effectively than other organizations.
Thanks for the link—that was what I was thinking of.
Do you have other organizations which teach rationality in mind? Offhand, the only thing I can think of is cognitive behavioral therapy, and it’s not exactly an organization.
No, I don’t have anything specific in mind.
:-)
The article is interesting, but I’m not sure it is relevant as the humans involved weren’t directing or monitoring the overall process, just taking part in it. Analogously even if an AGI requires my assistance/authorization to do certain things, that doesn’t give me any control over it unless I understand the consequences.
Also general warning against ‘generalising from fictional evidence.’