Perhaps, though, we are invoking some as-yet-undefined notion of “simplicity”
I’d guess we’re invoking a sense of what works and what doesn’t. If I consider the possibility of henceforth acting on the beliefs given by SI with a UTM that’s normal except it starts with very nearly all its probability mass on “if I smear myself in pumpkin pie filling and climb the Eiffel tower on a full moon I will become wealthy”, I’m like, yeah, that’s not… a well-designed machine for anything. It’s not a minimal design, it’s got a bunch of extra stuff put in there for no apparent reason, and it could have worked at least as well without the extra stuff. If I can take away parts without changing the function, that’s a better design.
This is appealing to my notion of “minimal design”, which is appealing to the concepts I use to think about possible designs; it’s minimal if you couldn’t remove “parts” or “ideas” or “information” without making it not work. Having unnecessary parts is at least wasteful. It’s also sort of evidence that something went wrong in the design process, like to explain the presence of the unnecessary stuff, I might infer that there was a designer working at cross-purposes (e.g. my past more confused self, or even an adversary trying to hack my design).
This is obviously not a worked out answer and begs the question, but it feels like a main way to look for an answer.
I’d guess we’re invoking a sense of what works and what doesn’t. If I consider the possibility of henceforth acting on the beliefs given by SI with a UTM that’s normal except it starts with very nearly all its probability mass on “if I smear myself in pumpkin pie filling and climb the Eiffel tower on a full moon I will become wealthy”, I’m like, yeah, that’s not… a well-designed machine for anything. It’s not a minimal design, it’s got a bunch of extra stuff put in there for no apparent reason, and it could have worked at least as well without the extra stuff. If I can take away parts without changing the function, that’s a better design.
This is appealing to my notion of “minimal design”, which is appealing to the concepts I use to think about possible designs; it’s minimal if you couldn’t remove “parts” or “ideas” or “information” without making it not work. Having unnecessary parts is at least wasteful. It’s also sort of evidence that something went wrong in the design process, like to explain the presence of the unnecessary stuff, I might infer that there was a designer working at cross-purposes (e.g. my past more confused self, or even an adversary trying to hack my design).
This is obviously not a worked out answer and begs the question, but it feels like a main way to look for an answer.