Historical simulations: assuming it preserves identity etc, why not…
Utility function: I know that my chances of maximizing my utiliy function are quite… slim, to say the least.
Path to best future(humanity): proofs do not lag so far behind right now. Modern type systems are now pretty good, and we have proof assistants that makes the “prove your whole program” quite feasible –though not cheap yet. Plus, the leading edge is generally the easiest to prove, because it tends to lie on solid mathematical ground. We don’t do proofs because they’re generally expensive, and we use ancient technologies that leak lots of low-level details, and make proofs much harder. (I program for a living.)
But I see at least the possibility of a slightly different path: still take precautions, just don’t prove the thing.
Oh, and I forgot: if we solve safety before capability, incrementally designing the AI by trial-and-error would be quite reasonable. The definite milestone will be harder to define in this case. I guess I’ll have to update a bit.
Historical simulations: assuming it preserves identity etc, why not…
Utility function: I know that my chances of maximizing my utiliy function are quite… slim, to say the least.
Path to best future(humanity): proofs do not lag so far behind right now. Modern type systems are now pretty good, and we have proof assistants that makes the “prove your whole program” quite feasible –though not cheap yet. Plus, the leading edge is generally the easiest to prove, because it tends to lie on solid mathematical ground. We don’t do proofs because they’re generally expensive, and we use ancient technologies that leak lots of low-level details, and make proofs much harder. (I program for a living.)
But I see at least the possibility of a slightly different path: still take precautions, just don’t prove the thing.
Oh, and I forgot: if we solve safety before capability, incrementally designing the AI by trial-and-error would be quite reasonable. The definite milestone will be harder to define in this case. I guess I’ll have to update a bit.