but rather that it is extremely unlikely that we can prevent everyone else from building one.
Well, SI’s mission makes sense on the premise that the best way to prevent a badly built AGI from being developed or deployed is to build a friendly AGI which has that as one of its goals. ‘Best way’ here is a compromise between, on the one hand, the effectiveness of the FAI relative to other approaches, and on the other, the danger presented by the FAI itself as opposed to other approaches.
So I think Holden’s position is that the ratio of danger vs. effectiveness does not weigh favorably for FAI as opposed to tool AI. So to argue against Holden, we would have to argue either that FAI will be less dangerous than he thinks, or that tool AI will be less effective than he thinks.
Indeed, we would have to argue that to argue against Holden.
My initial reaction was to counter this with a claim that we should not be arguing against anyone in the first place, but rather looking for probable truth (concentrate anticipations). And then I realized how stupid that was: Arguments Are Soldiers. If SI (and by the Blue vs Green principle, any SI-supporter) can’t even defend a few claims and defeat its opponents, it is obviously stupid and not worth paying attention to.
SI needs some amount of support, yet support-maximization strategies carry a very high risk of introducing highly dangerous intellectual contamination through various forms (including self-reinforcing biases in the minds of researchers and future supporters) that could turn out to cause even more existential risk. Yet, at the same time, not gathering enough support quickly enough dramatically augments the risk that someone, somewhere, is going to trip on a power cable and poof, all humans are just gone.
I am definitely not masterful enough in mathematics and bayescraft to calculate the optimal route through this differential probabilistic maze, but I suspect others could provide a very good estimate.
Also, it’s very much worth noting that these very considerations, on a meta level, are an integral part of SI’s mission, so figuring out whether that premise you stated is true or not, and whether there are better solutions or not actually is SI’s objective. Basically, while I might understand some of the cognitive causes for it, I am still very much rationally confused when someone questions SI’s usefulness by questioning the efficiency of subgoal X, while SI’s original and (to my understanding) primary mission is precisely to calculate the efficiency of subgoal X.
Well, SI’s mission makes sense on the premise that the best way to prevent a badly built AGI from being developed or deployed is to build a friendly AGI which has that as one of its goals. ‘Best way’ here is a compromise between, on the one hand, the effectiveness of the FAI relative to other approaches, and on the other, the danger presented by the FAI itself as opposed to other approaches.
So I think Holden’s position is that the ratio of danger vs. effectiveness does not weigh favorably for FAI as opposed to tool AI. So to argue against Holden, we would have to argue either that FAI will be less dangerous than he thinks, or that tool AI will be less effective than he thinks.
I take it the latter is the more plausible.
Indeed, we would have to argue that to argue against Holden.
My initial reaction was to counter this with a claim that we should not be arguing against anyone in the first place, but rather looking for probable truth (concentrate anticipations). And then I realized how stupid that was: Arguments Are Soldiers. If SI (and by the Blue vs Green principle, any SI-supporter) can’t even defend a few claims and defeat its opponents, it is obviously stupid and not worth paying attention to.
SI needs some amount of support, yet support-maximization strategies carry a very high risk of introducing highly dangerous intellectual contamination through various forms (including self-reinforcing biases in the minds of researchers and future supporters) that could turn out to cause even more existential risk. Yet, at the same time, not gathering enough support quickly enough dramatically augments the risk that someone, somewhere, is going to trip on a power cable and poof, all humans are just gone.
I am definitely not masterful enough in mathematics and bayescraft to calculate the optimal route through this differential probabilistic maze, but I suspect others could provide a very good estimate.
Also, it’s very much worth noting that these very considerations, on a meta level, are an integral part of SI’s mission, so figuring out whether that premise you stated is true or not, and whether there are better solutions or not actually is SI’s objective. Basically, while I might understand some of the cognitive causes for it, I am still very much rationally confused when someone questions SI’s usefulness by questioning the efficiency of subgoal X, while SI’s original and (to my understanding) primary mission is precisely to calculate the efficiency of subgoal X.