Upvoted for the assumptions and constraints. In fact, it would have been better if you had just left it at that. Maybe consider splitting that into a separate post?
Too much reasoning about “FOOM” AI here on LW is just plain fantastical. You just can’t win if you assume the opponent is God. If you think that an evil genius AGI could live on the z80 processor in your calculator while plotting world domination, then we’re all screwed and only the hail-Mary of a provably safe from first prinicples FAI design can save us, something which we don’t even know is possible or not.
If on the other hand we live in the physical world and the UFAI must operate according to the fundamental principles of physics and computation as we understand them, that’s a very different story. Suddenly much more mundane but practical solutions have a fighting chance, such as auditable oracle-AI. And as you demonstrate, the failure/breakout modes become situations which are predictable and with proper planning, preventable to a high degree of assurance.
I applaud you for breaking the mold. I’ll let others critique the specifics.
Upvoted for the assumptions and constraints. In fact, it would have been better if you had just left it at that. Maybe consider splitting that into a separate post?
Too much reasoning about “FOOM” AI here on LW is just plain fantastical. You just can’t win if you assume the opponent is God. If you think that an evil genius AGI could live on the z80 processor in your calculator while plotting world domination, then we’re all screwed and only the hail-Mary of a provably safe from first prinicples FAI design can save us, something which we don’t even know is possible or not.
If on the other hand we live in the physical world and the UFAI must operate according to the fundamental principles of physics and computation as we understand them, that’s a very different story. Suddenly much more mundane but practical solutions have a fighting chance, such as auditable oracle-AI. And as you demonstrate, the failure/breakout modes become situations which are predictable and with proper planning, preventable to a high degree of assurance.
I applaud you for breaking the mold. I’ll let others critique the specifics.