The pebble sorters must have some wiring to compare values. So they can decide if a 5 pile is better than two piles of two and three, so they can decide if buying a new shovel or more pebbles will advance their goal the most, and to strategize for the upcoming war between the pebblesorters and the humans.
If they pebblesorters build an AI to tell them how to win the war, that AI will quickly see that any single peblesorter has more strategic value than any pile, even if the piles are the only things it is told to defend.
The origin of my style of absolute morality in the peblesorter solar system is that if they don’t make good choices about what is important, then what is important will not be served.
An aside to Eliezer:
Up until this post, I felt like you were arguing that FAI was impossible or too hard for us, even though I knew you were promoting FAI. Hearing all the ways FAI won’t work left me strongly considering the notion that there is no reliable way we can get FAI to work. You might want to reorganize things for the book so that readers don’t get the wrong idea.
The pebble sorters must have some wiring to compare values. So they can decide if a 5 pile is better than two piles of two and three, so they can decide if buying a new shovel or more pebbles will advance their goal the most, and to strategize for the upcoming war between the pebblesorters and the humans.
If they pebblesorters build an AI to tell them how to win the war, that AI will quickly see that any single peblesorter has more strategic value than any pile, even if the piles are the only things it is told to defend.
The origin of my style of absolute morality in the peblesorter solar system is that if they don’t make good choices about what is important, then what is important will not be served.
An aside to Eliezer: Up until this post, I felt like you were arguing that FAI was impossible or too hard for us, even though I knew you were promoting FAI. Hearing all the ways FAI won’t work left me strongly considering the notion that there is no reliable way we can get FAI to work. You might want to reorganize things for the book so that readers don’t get the wrong idea.