It sounds like there’s some close ties to logical inductors here, both in terms of the flavor of the problem, and some difficulties I expect in translating theory into practice.
A logical inductor is kinda like an approximation. But it’s more accurate to call it lots and lots of approximations—it tries to keep track of every single approximation within some large class, which is essential to the proof that it only does finitely worse than any approximation
within that class.
A hierarchical model doesn’t naturally fall out of such a mixture, it seems. If you pose a general problem, you might just get a general solution. You could try to encourage specialized solutions by somehow ensuring that the problem has several different scales of interest, and sharply limit storage space so that the approximation can’t afford special cases that are too similar. But even then I think there’s a high probability that the best solution (according to something that is as theoretically convenient as logical inductors) would be alien—something humans wouldn’t pick out as the laws of physics in a million tries.
A tangent:
It sounds like there’s some close ties to logical inductors here, both in terms of the flavor of the problem, and some difficulties I expect in translating theory into practice.
A logical inductor is kinda like an approximation. But it’s more accurate to call it lots and lots of approximations—it tries to keep track of every single approximation within some large class, which is essential to the proof that it only does finitely worse than any approximation within that class.
A hierarchical model doesn’t naturally fall out of such a mixture, it seems. If you pose a general problem, you might just get a general solution. You could try to encourage specialized solutions by somehow ensuring that the problem has several different scales of interest, and sharply limit storage space so that the approximation can’t afford special cases that are too similar. But even then I think there’s a high probability that the best solution (according to something that is as theoretically convenient as logical inductors) would be alien—something humans wouldn’t pick out as the laws of physics in a million tries.