Things I get from this:
Things decided by our moral system are not relative, arbitrary or meaningless, any more than it’s relative, arbitrary or meaningless to say “X is a prime number”
Which moral system the human race uses is relative, arbitrary, and meaningless, just as there’s no reason for the pebble sorters to like prime numbers instead of composite numbers, perfect numbers, or even numbers.
A smart AI could follow our moral system as well or better than we ourselves can, just as the Pebble-Sorters’ AI can hopefully discover that they’re using prime numbers and thus settle the 1957 question once and for all.
But it would have to “want” to first. If the Pebble-Sorters just build an AI and say “Do whatever seems right to you”, it won’t start making prime-numbered heaps, unless an AI made by us humans and set to “Do whatever seems right to you” would also start making prime-numbered pebble-heaps. More likely, a Pebble-Sorter AI set do “Do whatever seems right to you” would sit there inertly, or fail spectacularly.
So the Pebble-Sorters would be best off using something like CEV.
Political Weirdtopia: Citizens decide it is unfair for a democracy to count only the raw number of people who support a position without considering the intensity with which they believe it. Of course, one can’t simply ask people to self-report the intensity with which they believe a position on their ballot, so stronger measures are required. Voting machines are redesigned to force voters to pull down a lever for each issue/candidate. The lever delivers a small electric shock, increasing in intensity each second the voter holds it down. The number of votes a person gets for a particular issue or candidate is a function of how long they keep holding down the lever.
In (choose one: more/less) enlightened sects of this society, the electric shock is capped at a certain level to avoid potential fatalities among overzealous voters. But in the (choose one: more/less) enlightened sects, voters can keep pulling down on the lever as long as they can stand the pain and their heart keeps working. Citizens consider this a convenient and entirely voluntary way to purge fanaticism from the gene pool.
The society lasts for several centuries before being taken over by a tiny cabal of people with Congenital Insensitivity to Pain Disorder.