So there’s probably some optimal point to stop encouraging open development, and given the current state of the theory (nil) I believe the time hasn’t come yet.
Your argument seems premised on the assumption that there will be an endgame. If we assume some large probability that we end up deciding not to have an endgame at all (i.e., not to try to actually build FAI with unenhanced humans), then it’s no longer clear “the time hasn’t come yet”.
Even if we assume that with probability ~1 there will be an effort to directly build FAI, given the slippery slope effects we have to stop encouraging open research well before the closed project starts. The main deciding factors for “when” must be how large the open research community has gotten, how strong the slippery slope effects are, and how much “pull” SingInst has against those effects. The “current state of the theory” seems to have little to do with it. (Edit: No that’s too strong. Let me amend it to “one consideration among many”.)
If we assume some large probability that we end up deciding not to have an endgame at all (i.e., not to try to actually build FAI with unenhanced humans), then it’s no longer clear “the time hasn’t come yet”.
This is something we’ll know better further down the road, so as long as it’s possible to defer this decision (i.e. while the downside is not too great, however that should be estimated), it’s the right thing to do. I still can’t rule out that there might be a preference definition procedure (that refers to humans) simple enough to be implemented pre-WBE, and decision theory seems to be an attack on this possibility (clarifying why this is naive, for example, in which case it’ll also serve as an argument to the powerful in the WBE race).
The “current state of the theory” seems to have little to do with it. (Edit: No that’s too strong. Let me amend it to “one consideration among many”.)
Well, maybe not specifically current, but what can be expected eventually, for the closed project to benefit from, which does seem to me like a major consideration in the possibility of its success.
Your argument seems premised on the assumption that there will be an endgame. If we assume some large probability that we end up deciding not to have an endgame at all (i.e., not to try to actually build FAI with unenhanced humans), then it’s no longer clear “the time hasn’t come yet”.
Even if we assume that with probability ~1 there will be an effort to directly build FAI, given the slippery slope effects we have to stop encouraging open research well before the closed project starts. The main deciding factors for “when” must be how large the open research community has gotten, how strong the slippery slope effects are, and how much “pull” SingInst has against those effects. The “current state of the theory” seems to have little to do with it. (Edit: No that’s too strong. Let me amend it to “one consideration among many”.)
This is something we’ll know better further down the road, so as long as it’s possible to defer this decision (i.e. while the downside is not too great, however that should be estimated), it’s the right thing to do. I still can’t rule out that there might be a preference definition procedure (that refers to humans) simple enough to be implemented pre-WBE, and decision theory seems to be an attack on this possibility (clarifying why this is naive, for example, in which case it’ll also serve as an argument to the powerful in the WBE race).
Well, maybe not specifically current, but what can be expected eventually, for the closed project to benefit from, which does seem to me like a major consideration in the possibility of its success.