I’m looking at the question of whether it’s certainly the case that getting an FAI is a matter of zeroing in directly on a tiny percentage of AI-space.
It seems to me that an underlying premise is that there’s no reason for a GAI to be Friendly, so Friendliness has to be carefully built into its goals. This isn’t unreasonable, but there might be non-obvious pulls towards or away from Friendliness, and if they exist, they need to be considered. At the very least, there may be general moral considerations which incline towards Friendliness, and which would be more stable than starting from a definition of humanity and then trying to protect that.
Here’s an example of a seemingly open choice where there are non-obvious biases towards particular outcomes—D&D alignments. You look at the tidy little two-dimensional chart, and you might think you can equally play any alignment which appeals to you.
The truth is that Chaotic and/or Evil and/or Neutral alignments tend to make coordination inside parties more difficult. It’s possible to play them successfully, but it takes more skill than making Lawful and/or Good work,. Some players find out that playing from the first batch with too much gusto makes gaming less fun. Some GMs put restrictions on the first batch of alignments or how they can be played.
I’m looking at the question of whether it’s certainly the case that getting an FAI is a matter of zeroing in directly on a tiny percentage of AI-space.
It seems to me that an underlying premise is that there’s no reason for a GAI to be Friendly, so Friendliness has to be carefully built into its goals. This isn’t unreasonable, but there might be non-obvious pulls towards or away from Friendliness, and if they exist, they need to be considered. At the very least, there may be general moral considerations which incline towards Friendliness, and which would be more stable than starting from a definition of humanity and then trying to protect that.
Here’s an example of a seemingly open choice where there are non-obvious biases towards particular outcomes—D&D alignments. You look at the tidy little two-dimensional chart, and you might think you can equally play any alignment which appeals to you.
The truth is that Chaotic and/or Evil and/or Neutral alignments tend to make coordination inside parties more difficult. It’s possible to play them successfully, but it takes more skill than making Lawful and/or Good work,. Some players find out that playing from the first batch with too much gusto makes gaming less fun. Some GMs put restrictions on the first batch of alignments or how they can be played.