I agree that power and capability concerns seem important here. Even if we accept A[n+1] is not going to fail on C due to being more powerful than A[n], it seems likely to me that A[n] will not be capable enough to sufficiently assess A[n+1] such that we can get good enough guarantees that C holds for A[n+1].
If we look at the probability of C holding over the whole chain of induction I think things look even worse. Let’s say there is a 99.9% likelihood that C will hold on any step of induction. then over many iterations as we multiply the probability that C holds we find probability that C holds over the whole process falls with a lower probability of C holding overall the more iterations needed.
Taken together this suggests a serious challenge, because to minimize risk of power differentials between successors to increase the chance of C holding we would want many small iterations, but this risks increasing the risk that small probabilities of failure in each iteration will compound such that it eventually becomes likely that C does not hold.
To meet this harkens back to a lesson we long ago learned in engineering: the more moving parts in your system the more likely it will fail.
I agree that power and capability concerns seem important here. Even if we accept A[n+1] is not going to fail on C due to being more powerful than A[n], it seems likely to me that A[n] will not be capable enough to sufficiently assess A[n+1] such that we can get good enough guarantees that C holds for A[n+1].
If we look at the probability of C holding over the whole chain of induction I think things look even worse. Let’s say there is a 99.9% likelihood that C will hold on any step of induction. then over many iterations as we multiply the probability that C holds we find probability that C holds over the whole process falls with a lower probability of C holding overall the more iterations needed.
Taken together this suggests a serious challenge, because to minimize risk of power differentials between successors to increase the chance of C holding we would want many small iterations, but this risks increasing the risk that small probabilities of failure in each iteration will compound such that it eventually becomes likely that C does not hold.
To meet this harkens back to a lesson we long ago learned in engineering: the more moving parts in your system the more likely it will fail.