Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable.
Remove the word “AI” and I think this claim is not really changed in any way. AI systems are the most general systems.
The statement is isomorphic to the more general claim that “In general, it is possible for a sequence of actions to result in bad outcomes even if those actions had been determined to result in good outcomes before taking them.”
I don’t think that is true in general. Furthermore, when you get close to AI, your theoretical steps and your physical steps begin to more closely coincide. If my theoretical steps say that the next theoretical step will be fine, and so was this one, then I am also more confident that the physical steps were mostly similar to this. As we develop AI, the more developed AI’s should be making us more confident that things will go well than the previous AI’s, and so on.
Therefore, if there is ever a point at which we feel confident that AI could be developed safely, it should be at the same time that we develop a near-AGI that makes us feel near-confident that it could be developed safely.
If not, then not six months, nor any conceivable time-frame would be expected to result in something that made everyone feel confident enough. If you believe the statement above (my restatement of it), that isn’t going to even be possible, in principle.
Remove the word “AI” and I think this claim is not really changed in any way. AI systems are the most general systems.
The statement is isomorphic to the more general claim that “In general, it is possible for a sequence of actions to result in bad outcomes even if those actions had been determined to result in good outcomes before taking them.”
I don’t think that is true in general. Furthermore, when you get close to AI, your theoretical steps and your physical steps begin to more closely coincide. If my theoretical steps say that the next theoretical step will be fine, and so was this one, then I am also more confident that the physical steps were mostly similar to this. As we develop AI, the more developed AI’s should be making us more confident that things will go well than the previous AI’s, and so on.
Therefore, if there is ever a point at which we feel confident that AI could be developed safely, it should be at the same time that we develop a near-AGI that makes us feel near-confident that it could be developed safely.
If not, then not six months, nor any conceivable time-frame would be expected to result in something that made everyone feel confident enough. If you believe the statement above (my restatement of it), that isn’t going to even be possible, in principle.