The thing about the Pareto frontier of Kolmogorov complexity vs prediction score is that most programs aren’t on it. In particular, it seems unlikely that p_1, the seed AI written by humans, is going to be on the frontier. Even p_2, the successor AI, might not be on it either. We can’t equovicate between all programs that get the same prediction score, differences between them will be observable in the way they make predictions.
The thing about the Pareto frontier of Kolmogorov complexity vs prediction score is that most programs aren’t on it. In particular, it seems unlikely that p_1, the seed AI written by humans, is going to be on the frontier. Even p_2, the successor AI, might not be on it either. We can’t equovicate between all programs that get the same prediction score, differences between them will be observable in the way they make predictions.
I don’t disagree with any of what you say here—I just read Anton as assuming we have a program on that frontier