At some points in your comment you use the criterion “likely to be valid”, at other points you use the criterion “guaranteed to be valid”. These are very different! I think almost everyone agrees that we’re unlikely to get predictions which are guaranteed to be valid out-of-distribution. But that’s true of every science apart from fundamental physics: they all apply coarse-grained models, whose predictive power out-of-distribution varies very widely. There are indeed some domains in which it’s very weak (like ecology), but also some domains in which it’s pretty strong (like chemistry). There are some reasons to think interpretability will be more like the former (networks are very complicated!) and some reasons to think it’ll be more like the latter (experiments with networks are very reproducible). I don’t think this is the type of thing which can be predicted very well in advance, because it’s very hard to know what types of fundamental breakthroughs may arise.
More generally, the notion of “computational irreducibility” doesn’t seem very useful to me, because it takes a continuous property (some systems are easier or harder to make predictions about) and turns it into a binary property (is it computationally reducible or not), which I think obscures more than it clarifies.
At some points in your comment you use the criterion “likely to be valid”, at other points you use the criterion “guaranteed to be valid”. These are very different! I think almost everyone agrees that we’re unlikely to get predictions which are guaranteed to be valid out-of-distribution. But that’s true of every science apart from fundamental physics: they all apply coarse-grained models, whose predictive power out-of-distribution varies very widely. There are indeed some domains in which it’s very weak (like ecology), but also some domains in which it’s pretty strong (like chemistry). There are some reasons to think interpretability will be more like the former (networks are very complicated!) and some reasons to think it’ll be more like the latter (experiments with networks are very reproducible). I don’t think this is the type of thing which can be predicted very well in advance, because it’s very hard to know what types of fundamental breakthroughs may arise.
More generally, the notion of “computational irreducibility” doesn’t seem very useful to me, because it takes a continuous property (some systems are easier or harder to make predictions about) and turns it into a binary property (is it computationally reducible or not), which I think obscures more than it clarifies.