Is backwards causation necessarily absurd?

In New­comb’s prob­lem an agent picks ei­ther one-box or two-box and finds that no mat­ter which op­tion they picked, a pre­dic­tor pre­dicted them in ad­vance. I’ve gone to a lot of effort to ex­plain how this can be with­out re­quiring back­wards cau­sa­tion (The Pre­dic­tion Prob­lem, De­con­fus­ing Log­i­cal Coun­ter­fac­tu­als), yet now I find my­self won­der­ing if back­wards cau­sa­tion is such a bad ex­pla­na­tion af­ter all.

Un­for­tu­nately I’m not a physi­cist, so take what I say with a grain of salt, but there seems to be a rea­son­able ar­gu­ment that ei­ther time or its di­rec­tion is an illu­sion. One promi­nent the­ory of time is Eter­nal­ism in which there is no ob­jec­tive flow of time and terms like “past”, “pre­sent” and “fu­ture” can only be used in a rel­a­tive sense. An ar­gu­ment in favour of this is that it is of­ten very con­ve­nient in physics to model space-time as a 4-di­men­sional space. If time is just an­other di­men­sion, why should the fu­ture be treated differ­ently than the past? Noth­ing in this model differ­en­ti­ates the two. If we have two blocks X and Y next to each other, we can view ei­ther X as the left one or Y as the left one de­pend­ing on the di­rec­tion we look at it from. Similarly, if A causes B in the tra­di­tional for­wards sense, why can’t we sym­met­ri­cally view B as back­wards caus­ing A, where again if we viewed it an­other way A to B would be back­wards cau­sa­tion and B to A would be for­wards cau­sa­tion.

Another rel­a­tivis­tic ar­gu­ment against time flow­ing is that si­mul­tane­ity is only defined rel­a­tive to a refer­ence frame. There­fore, there is no unified pre­sent which is sup­posed to be what is flow­ing.

Thirdly, en­tropy has of­ten been the ar­row of time with other phys­i­cal laws claimed to be re­versible. We are in a low-en­tropy world so en­tropy in­creases. How­ever, if we were in a high-en­tropy world, it would de­crease, so time and cau­sa­tion would seem to be go­ing back­wards (from our per­spec­tive). This would seem to sug­gest that back­wards cau­sa­tion is just as valid a phe­nomenon as back­ward cau­sa­tion.

I want to re­mind read­ers again that I am not a physi­cist. This post is more in­tended to spark dis­cus­sion that any­thing else.

(Another pos­si­bil­ity I haven’t dis­cussed is that cau­sa­tion might be in the map rather than the ter­ri­tory)