Re paragraph 3: it seems like these are mostly considerations that might strengthen your conclusions if we granted that there was a big productivity difference between my design and a “a standard utility-maximizing AI based on a crude model of their current understanding of ethics.” But I would already be happy to classify a large productivity loss as a failure, so let’s just concentrate on the claimed productivity loss.
If there is competition, everyone has strong incentives to quickly build “full-fledged FAIs” which can solve these ethical problems and know exactly what they should and shouldn’t do
These incentives only operate if there is a big productivity difference.
Beyond that, if the kinds of issues peope run into are “the AI faces a lot of everyday ethical questions in the course of acquiring resources,” then it really seems like what you need is a not-catastrophically-wrong model of human morality, which would probably just be built in mundane ways. I don’t see a strong argument that this would require lots of impresive conceptual progress, rather than being simiar-in-kind to building a predictive model of anything else. But I suspect this is just a reflection of the disagreement about paragraph 2, which we should flesh out in the sibling.
Someone could make a copy of an existing AI based on your design, change the code or configuration files to make themselves the overseer and remove the mandatory oversights, and then ask the AI to make a “full-fledged FAI” for them, and if they happen to be of the incautious type, this will probably result in the kind of crude normative AI mentioned above (or worse, if they approve a bunch of “improvements” that end up subverting their intentions altogether).
This doesn’t seem like a very general argument against the possibility of mandatory oversight or technological handicapping, and the measures you describe seem like strawmen. I agree that whatever kind of oversight you employ, it will be possible to subvert it, whatever tax you charge it will be possible to evade it, and so on. But doing so will often come with a cost (as it does today), and it just doesn’t seem that hard to get it up to a 1% loss (say). We could talk more about the particular measures that could be taken for oversight; I’m sure we can both imagine many regulatory and technological approaches that would be more annoying to sidestep than an entry in a configuration file, but I suspect our disagreement comes form us imagining different productivity gaps.
The prospect of someone designing their own AI, which is very architecturally different from the rest of the world, just doesn’t seem especially troubling, unless you imagine that the rest of the world is using a significantly handicapped design. (See the first sentence of this reply.)
Re paragraph 3: it seems like these are mostly considerations that might strengthen your conclusions if we granted that there was a big productivity difference between my design and a “a standard utility-maximizing AI based on a crude model of their current understanding of ethics.” But I would already be happy to classify a large productivity loss as a failure, so let’s just concentrate on the claimed productivity loss.
These incentives only operate if there is a big productivity difference.
Beyond that, if the kinds of issues peope run into are “the AI faces a lot of everyday ethical questions in the course of acquiring resources,” then it really seems like what you need is a not-catastrophically-wrong model of human morality, which would probably just be built in mundane ways. I don’t see a strong argument that this would require lots of impresive conceptual progress, rather than being simiar-in-kind to building a predictive model of anything else. But I suspect this is just a reflection of the disagreement about paragraph 2, which we should flesh out in the sibling.
This doesn’t seem like a very general argument against the possibility of mandatory oversight or technological handicapping, and the measures you describe seem like strawmen. I agree that whatever kind of oversight you employ, it will be possible to subvert it, whatever tax you charge it will be possible to evade it, and so on. But doing so will often come with a cost (as it does today), and it just doesn’t seem that hard to get it up to a 1% loss (say). We could talk more about the particular measures that could be taken for oversight; I’m sure we can both imagine many regulatory and technological approaches that would be more annoying to sidestep than an entry in a configuration file, but I suspect our disagreement comes form us imagining different productivity gaps.
The prospect of someone designing their own AI, which is very architecturally different from the rest of the world, just doesn’t seem especially troubling, unless you imagine that the rest of the world is using a significantly handicapped design. (See the first sentence of this reply.)