Suppose you find “57+72=” already written. Do you write down “130″ because you are slightly off-target relative to the goal of computing 57+72? Do you write down “129” because it’s what follows by the nature of what’s already written? Do you notice it’s made out of digits you can use for something else, such as “25+77=”?
I think the case for writing down “130” because of misalignment-in-detail is the weakest. Also, even a superintelligence can’t help 57+72 decide it would be heading for 129, or make 57+72′s efforts irrelevant, as 129 still follows entirely from 57+72′s own nature, it’s 57+72 and nothing else that would need to strive for it. A superintelligence writing down “129″ doesn’t change that, and neither does writing down “130”, or rewriting the digits into 25+77.
“Slightly off-target for X” is a more complex goal than just “X”. But if X is in the training data, then just repeating X isn’t contributing any originality — but playing a variation on the theme of X might be. Sometimes the right answer is to follow X broadly, while also keeping a specified distance; see boids.
Suppose you find “57+72=” already written. Do you write down “130″ because you are slightly off-target relative to the goal of computing 57+72? Do you write down “129” because it’s what follows by the nature of what’s already written? Do you notice it’s made out of digits you can use for something else, such as “25+77=”?
I think the case for writing down “130” because of misalignment-in-detail is the weakest. Also, even a superintelligence can’t help 57+72 decide it would be heading for 129, or make 57+72′s efforts irrelevant, as 129 still follows entirely from 57+72′s own nature, it’s 57+72 and nothing else that would need to strive for it. A superintelligence writing down “129″ doesn’t change that, and neither does writing down “130”, or rewriting the digits into 25+77.
“Slightly off-target for X” is a more complex goal than just “X”. But if X is in the training data, then just repeating X isn’t contributing any originality — but playing a variation on the theme of X might be. Sometimes the right answer is to follow X broadly, while also keeping a specified distance; see boids.