I agree with timtyler’s comment that Objections 1 and 2 are bogus, especially 2. The tool-AGI discussion reveals significant misunderstanding, I feel. Despite this, I think it is still a great and useful post.
Another sort of tangential issue is that this post fails to consider whether or not lots of disparate labs are just going to undertake AGI research regardless of SIAI. If lots of labs are doing that, it could be dangerous (if SIAI arguments are sound). So one upside to funding an organization like SIAI is that it will kind of rake the attention to a central point. Remember that one of SIAI’s short term goals is to decelerate generic AGI research in favor of accelerating AGI safety research.
This post doesn’t seem to account for the fact that by not funding SIAI you simply face the same number of counterfactual disparate labs pursuing AGI with their own willy-nilly sources of funding, but no aggregator organization to serve as a kind of steering committee. Regardless of whether SIAI’s specific vision is the one that happens to come true, something should be said for the inherent danger of a bunch of labs trying to build their own stand-alone paperclip maxmizers, which they may very well believe are tool-AGIs, and then bam, game over.
I agree with timtyler’s comment that Objections 1 and 2 are bogus, especially 2. The tool-AGI discussion reveals significant misunderstanding, I feel. Despite this, I think it is still a great and useful post.
Another sort of tangential issue is that this post fails to consider whether or not lots of disparate labs are just going to undertake AGI research regardless of SIAI. If lots of labs are doing that, it could be dangerous (if SIAI arguments are sound). So one upside to funding an organization like SIAI is that it will kind of rake the attention to a central point. Remember that one of SIAI’s short term goals is to decelerate generic AGI research in favor of accelerating AGI safety research.
This post doesn’t seem to account for the fact that by not funding SIAI you simply face the same number of counterfactual disparate labs pursuing AGI with their own willy-nilly sources of funding, but no aggregator organization to serve as a kind of steering committee. Regardless of whether SIAI’s specific vision is the one that happens to come true, something should be said for the inherent danger of a bunch of labs trying to build their own stand-alone paperclip maxmizers, which they may very well believe are tool-AGIs, and then bam, game over.