Things decided by our moral system are not relative, arbitrary or meaningless, any more than it’s relative, arbitrary or meaningless to say “X is a prime number”
Which moral system the human race uses is relative, arbitrary, and meaningless, just as there’s no reason for the pebble sorters to like prime numbers instead of composite numbers, perfect numbers, or even numbers.
A smart AI could follow our moral system as well or better than we ourselves can, just as the Pebble-Sorters’ AI can hopefully discover that they’re using prime numbers and thus settle the 1957 question once and for all.
But it would have to “want” to first. If the Pebble-Sorters just build an AI and say “Do whatever seems right to you”, it won’t start making prime-numbered heaps, unless an AI made by us humans and set to “Do whatever seems right to you” would also start making prime-numbered pebble-heaps. More likely, a Pebble-Sorter AI set do “Do whatever seems right to you” would sit there inertly, or fail spectacularly.
So the Pebble-Sorters would be best off using something like CEV.
Things I get from this:
Things decided by our moral system are not relative, arbitrary or meaningless, any more than it’s relative, arbitrary or meaningless to say “X is a prime number”
Which moral system the human race uses is relative, arbitrary, and meaningless, just as there’s no reason for the pebble sorters to like prime numbers instead of composite numbers, perfect numbers, or even numbers.
A smart AI could follow our moral system as well or better than we ourselves can, just as the Pebble-Sorters’ AI can hopefully discover that they’re using prime numbers and thus settle the 1957 question once and for all.
But it would have to “want” to first. If the Pebble-Sorters just build an AI and say “Do whatever seems right to you”, it won’t start making prime-numbered heaps, unless an AI made by us humans and set to “Do whatever seems right to you” would also start making prime-numbered pebble-heaps. More likely, a Pebble-Sorter AI set do “Do whatever seems right to you” would sit there inertly, or fail spectacularly.
So the Pebble-Sorters would be best off using something like CEV.