This is an old post and probably very out of date, but: I think if you try to define an impartial observer’s preferences as whatever selects (C,C) in two other agents’ PD, you get inconsistencies very rapidly once you have one of those agents stuck in two Prisoner’s Dilemmas at once.
I also don’t think we should use euphemisms like ‘impartial’ for an incredibly partial Cooperation Fetishist that’s willing to give up everything else of value (e.g., billions of human lives) to go through the motions of satisfying non-sentient processes like sea slugs or paperclip maximizers.
you get inconsistencies very rapidly once you have one of those agents stuck in two Prisoner’s Dilemmas at once.
Multi-player interactions are tricky and we don’t have a good solution for them yet.
that’s willing to give up everything else of value (e.g., billions of human lives)
It’s not that its willing to give up everything of value—it’s that it doesn’t have our values. Without sharing our values, there’s no reason for it to prefer our opinions over sea slugs.
This is an old post and probably very out of date, but: I think if you try to define an impartial observer’s preferences as whatever selects (C,C) in two other agents’ PD, you get inconsistencies very rapidly once you have one of those agents stuck in two Prisoner’s Dilemmas at once.
I also don’t think we should use euphemisms like ‘impartial’ for an incredibly partial Cooperation Fetishist that’s willing to give up everything else of value (e.g., billions of human lives) to go through the motions of satisfying non-sentient processes like sea slugs or paperclip maximizers.
Multi-player interactions are tricky and we don’t have a good solution for them yet.
It’s not that its willing to give up everything of value—it’s that it doesn’t have our values. Without sharing our values, there’s no reason for it to prefer our opinions over sea slugs.