ShardPhoenix: “the basis of human morality is (presumably) so much more complicated than the “prime numbers = good” presented here that the analogy is a bit strained”
I actually didn’t notice that the Pebblesorters like primes until I read this comment. Somehow I feel as if this supports Eliezer’s point in some way which I can’t notice on my own either.
There is a pattern to what kinds of heaps the Pebblesorters find “right” and “wrong”, but they haven’t figured it out yet. They have always just used their intuition to decide if a heap was right or wrong, but their intuition got less precise in extreme cases like very large heaps. The Pebblesorters would have been better off if only they could have figured out the pattern and applied it to extreme heaps, rather than fighting over differences of intuition.
Also if they had just figured out the pattern, they could have programmed it into the AI rather than hoping that the AI’s intuition would be exactly the same as their own, or manually programming the AI with every special case.
I think this was the main point of the essay but it went right over my head at first.
But it seems weird to me that they have computers and algorithms if they can’t figure out this pattern. That messed with my suspension of disbelief for a bit.
Whether or not you were able to suspend disbelief seems irrelevant, as the purpose of the post is not to tell a plausible story. It’s to illustrate certain concepts. In fact, if you had been able to suspend your disbelief entirely then the post would have failed, as your attention would have been on the story, rather than the underlying points being made.
Criticising a parable such as this for its implausibility is rather like doing the same for the trolley problem, or the utility monster. I think it misses the point.
ShardPhoenix: “the basis of human morality is (presumably) so much more complicated than the “prime numbers = good” presented here that the analogy is a bit strained”
I actually didn’t notice that the Pebblesorters like primes until I read this comment. Somehow I feel as if this supports Eliezer’s point in some way which I can’t notice on my own either.
There is a pattern to what kinds of heaps the Pebblesorters find “right” and “wrong”, but they haven’t figured it out yet. They have always just used their intuition to decide if a heap was right or wrong, but their intuition got less precise in extreme cases like very large heaps. The Pebblesorters would have been better off if only they could have figured out the pattern and applied it to extreme heaps, rather than fighting over differences of intuition.
Also if they had just figured out the pattern, they could have programmed it into the AI rather than hoping that the AI’s intuition would be exactly the same as their own, or manually programming the AI with every special case.
I think this was the main point of the essay but it went right over my head at first.
But it seems weird to me that they have computers and algorithms if they can’t figure out this pattern. That messed with my suspension of disbelief for a bit.
Well if the pattern was too complicated, then a reader of the blog post wouldn’t be able to notice it.
Sure, that explains why the story was written with this flaw, but it doesn’t remove the flaw. But I don’t have a better suggestion.
It does remove the flaw, because it’s a thought experiment. It doesn’t have to be plausible. It merely tests our evaluative judgements and intuitions.
It’s a fact that it messed with my suspension of disbelief for a bit. It would be better if it hadn’t. I still like the story; it’s just a minor flaw.
Whether or not you were able to suspend disbelief seems irrelevant, as the purpose of the post is not to tell a plausible story. It’s to illustrate certain concepts. In fact, if you had been able to suspend your disbelief entirely then the post would have failed, as your attention would have been on the story, rather than the underlying points being made.
Criticising a parable such as this for its implausibility is rather like doing the same for the trolley problem, or the utility monster. I think it misses the point.