[Question] Instrumental Occam?

In The Real Rules have No Ex­cep­tions, Said says (ap­par­ently speak­ing of in­stru­men­tal rules, not epistemic ones):

Pre­fer sim­plic­ity in your rules. Be vigilant that your rules do not grow too com­plex; make sure you are not re­lax­ing the le­gi­t­i­macy crite­ria of your ex­cep­tions. Pe­ri­od­i­cally au­dit your rules, in­spect­ing them for com­plex­ity; try to for­mu­late sim­pler ver­sions of com­plex rules.

This is a very plau­si­ble prin­ci­ple. I have had similar thoughts my­self. The idea seems to have merit in prac­tice. But is there any the­o­ret­i­cal analogue?

Min­i­mum de­scrip­tion length (Oc­cam’s Ra­zor) is an epistemic prin­ci­ple. In Bayesian terms, prob­a­bil­ities must sum to one, which trans­lates in­for­ma­tion-the­o­ret­i­cally (via op­ti­mal en­cod­ings) to the idea that there are only so many short codes, so we have to as­sign them care­fully to the most prob­a­ble things.

How­ever, ex­pected util­ity is not a limited re­source in this way. Up­dat­ing to think an op­tion is bet­ter than pre­vi­ously thought doesn’t nec­es­sar­ily make other op­tions worse. Select­ing poli­cies is not so differ­ent from se­lect­ing raw ac­tions. And for Bayesian de­ci­sion the­ory, it doesn’t re­ally mat­ter whether a policy is speci­fied as a big table of ob­ser­va­tion/​ac­tion pairs, vs an el­e­gantly speci­fied rule.

Op­ti­mal­ity cares not for el­e­gance!

Yet, even in the rel­a­tively for­mal world of ma­chine learn­ing, the prac­tice seems con­trary to this. When you are op­ti­miz­ing a neu­ral net­work, you don’t ac­tu­ally care that much whether it’s some­thing like a hy­poth­e­sis (mak­ing pre­dic­tions) or some­thing like a policy (car­ry­ing out ac­tions). You ap­ply the same kind of reg­u­lariza­tion ei­ther way, as far as I un­der­stand (reg­u­lariza­tion be­ing the ma­chine-learner’s gen­er­al­iza­tion of Oc­cam). (Cor­rec­tion: this seems not to be the case.)

We might say that this is be­cause (in some sense) the in­stru­men­tal un­cer­tainty and the epistemic un­cer­tainty are ac­tu­ally be­ing wrapped up to­gether. But (1) this re­ply seems overly quick to me at pre­sent, and I’d want to un­der­stand in de­tail whether this can be jus­tified; (2) I’m cu­ri­ous if there is a purely in­stru­men­tal ver­sion of Oc­cam to be ar­tic­u­lated; it seems in­tu­itively re­ally plau­si­ble to me, though tech­ni­cally quite mys­te­ri­ous.

So: is it pos­si­ble to for­mu­late an in­stru­men­tal ver­sion of Oc­cam? Can we jus­tify a sim­plic­ity bias in our poli­cies?