causation might be in the map rather than the territory
Of course it is. There is no atom of causation anywhere. It’s a tool for embedded agents to construct useful models in an internally partially predictable universe.
“Backward causation” may or may not be a useful model at times, but it is certainly nothing but a model.
As a trained (though not practicing) physicist, I can see that you are making a large category error here. Relativity neither adds to not subtracts from the causation models. In a deterministic Newtonian universe you can imagine backward causation as a useful tool. Sadly, its usefulness it rather limited. For example, the diffusion/heat equation is not well posed when run backwards, it blows up after a finite integration time. An intuitive way to see that is that you cannot reconstruct the shape of a glass of water from the puddle you see on the ground some time after it was spilled. But in cases where the relevant PDEs are well posed in both time directions, backward causality is equivalent to forward causality, if not computationally, then at least in principle.
All that special relativity gives you is that the absolute temporal order of events is only defined when they are within a lightcone, not outside of it. General relativity gives you both less and more. On the one hand, the Hilbert action is formulated without referring to time evolution at all and poses no restriction on the type of matter sources, be they positive or negative density, subluminal or superluminal, finite or singlular. On the other hand, to calculate most interesting things, one needs to solve the initial value problem, and that one poses various restrictions on what topologies and matter sources one can start with. On the third hand, there is a lot of freedom to define what constitutes “now”, as many different spacetime foliations are on equal footing.
If you add quantum mechanics to the mix, the Born rule, needed to calculate anything useful regardless of one’s favorite interpretation, breaks linearity and unitarity at the moment of interaction (loosely speaking) and is not time-reversal invariant.
The entropic argument is also without merit: there is no reason to believe that entropy would decrease in a “high-entropy world”, whatever that might mean. We do not even know how observer-independent entropy is (Jaynes argued that apparent entropy depends on the observer’s knowledge of the world).
Basically, you are confusing map and territory. If backward causality helps you make more accurate maps, go wild, just don’t claim that you are doing anything other than constructing models.
I think we can go a bit farther in predicting that backwards causation will be a useful concept in some very specific cases, which will break down far above the scale of the normal second law.
We “see” backwards causation when we know the outcome but not how the system will get there. What does this behavior sound like a hallmark of? Optimization processes! We can predict in advance that backwards causation will be a useful idea to talk about the behavior of some optimization processes, but that it will stop contributing useful information when we want to zoom in past the “intentional stance” level of description.