I feel like I haven’t quite clarified for myself what the cybernetic agent is doing wrong (or if you don’t want to assume physicalism, just put scare quotes around “doing wrong”) when it sees the all-black input. There might be simple differences that have many implications.
Suppose that all hypotheses are in terms of physical universe + bridging law. We might accuse the cybernetic agent of violating a dogma of information flow by making its physical-universe-hypothesis depend on the output of the bridging law. But it doesn’t necessarily have to! Consider the hypotheses that has a universe just complicated enough to explain my past observations up until the all-black input, and then a bridging law that is normal (locating myself within the modeled universe etc.) until I see the all-black input, upon which it becomes something very simple (like all black from now on). This seems simple to the cybernetic agent but is nonsensical from a physicalist perspective.
So we might say that all information backflow from the output of the bridging law back into any part of the hypothesis is a mistake. But although this might work for idealized microphysical hypotheses, in practice I think I actually violate both levels of information-flow dogma when I use my expected thoughts and feelings to predict what I’ll do and perceive in the physical world. Although you could still deny that such practical self-modeling “counts,” and make my hypotheses something more Platonic, that are imputed to me but that I don’t carry around in my physical brain, or we could imagine some “inner RL agent” steering my brain that gets to obey the strict rules of physicalism even if my brain as a whole doesn’t. I find this latter unsatisfactory—I want to explore my own whole-brain views on physicalism, not the views of a tiny fraction of me, and flipping this around, if we build a Turing RL agent I want the thing as a whole to be physicalist, not just a small part of it.
In short, the information flow dogma seems promising for toy models of reasoning, but it feels to me like it must only be an expression of some deeper principle that even us bounded humans can obey.
One important fact I didn’t yet mention is that we don’t have a final hypothesis for the world—we’re constantly getting new bits that have to be incorporated into our physical hypothesis and bridging law. Without this, I think the perverse “all black from now on” hypothesis isn’t very appealing (though still more appealing to the Cartesian agent than to the physicalist), because it’s more complex than the true hypothesis. But with new bits flowing in all the time, the simplest predictions are always going to predict that this information flow suddenly stops.
Is there some kind of Copernican principle here that makes sense to a physicalist but sounds like nonsense to a Cartesian? Like “No, this is not the instant when you’re going to stop learning new things forever.” Is physicalism related to looking for patterns on the meta-levels?
Warning: rambling.
I feel like I haven’t quite clarified for myself what the cybernetic agent is doing wrong (or if you don’t want to assume physicalism, just put scare quotes around “doing wrong”) when it sees the all-black input. There might be simple differences that have many implications.
Suppose that all hypotheses are in terms of physical universe + bridging law. We might accuse the cybernetic agent of violating a dogma of information flow by making its physical-universe-hypothesis depend on the output of the bridging law. But it doesn’t necessarily have to! Consider the hypotheses that has a universe just complicated enough to explain my past observations up until the all-black input, and then a bridging law that is normal (locating myself within the modeled universe etc.) until I see the all-black input, upon which it becomes something very simple (like all black from now on). This seems simple to the cybernetic agent but is nonsensical from a physicalist perspective.
So we might say that all information backflow from the output of the bridging law back into any part of the hypothesis is a mistake. But although this might work for idealized microphysical hypotheses, in practice I think I actually violate both levels of information-flow dogma when I use my expected thoughts and feelings to predict what I’ll do and perceive in the physical world. Although you could still deny that such practical self-modeling “counts,” and make my hypotheses something more Platonic, that are imputed to me but that I don’t carry around in my physical brain, or we could imagine some “inner RL agent” steering my brain that gets to obey the strict rules of physicalism even if my brain as a whole doesn’t. I find this latter unsatisfactory—I want to explore my own whole-brain views on physicalism, not the views of a tiny fraction of me, and flipping this around, if we build a Turing RL agent I want the thing as a whole to be physicalist, not just a small part of it.
In short, the information flow dogma seems promising for toy models of reasoning, but it feels to me like it must only be an expression of some deeper principle that even us bounded humans can obey.
One important fact I didn’t yet mention is that we don’t have a final hypothesis for the world—we’re constantly getting new bits that have to be incorporated into our physical hypothesis and bridging law. Without this, I think the perverse “all black from now on” hypothesis isn’t very appealing (though still more appealing to the Cartesian agent than to the physicalist), because it’s more complex than the true hypothesis. But with new bits flowing in all the time, the simplest predictions are always going to predict that this information flow suddenly stops.
Is there some kind of Copernican principle here that makes sense to a physicalist but sounds like nonsense to a Cartesian? Like “No, this is not the instant when you’re going to stop learning new things forever.” Is physicalism related to looking for patterns on the meta-levels?