Epistemic status: Leaning heavily into inside view, throwing humility to the winds.
Imagine TAI is magically not coming (CDT-style counterfactual[1]). Then, the most notable-in-hindsight feature of modern times might be the budding of mathematical metaphysics (Solomonoff induction, AIXI, Yudkowsky’s “computationalist metaphilosophy”[2], UDT, infra-Bayesianism...) Perhaps, this will lead to an “epistemic revolution” comparable only with the scientific revolution in magnitude. It will revolutionize our understanding of the scientific method (probably solving the interpretation of quantum mechanics[3], maybe quantum gravity, maybe boosting the soft sciences). It will solve a whole range of philosophical questions, some of which humanity was struggling with for centuries (free will, metaethics, consciousness, anthropics...)
But, the philosophical implications of the previous epistemic revolution were not so comforting (atheism, materialism, the cosmic insignificance of human life)[4]. Similarly, the revelations of this revolution might be terrifying[5]. In this case, it remains to be seen which will seem justified in hindsight: the Litany of Gendlin, or the Lovecraftian notion that some knowledge is best left alone (and I say this as someone fully committed to keep digging into this mine of Khazad-dum).
The EDT-style counterfactual “TAI is not coming” would imply that a lot of my thinking on related topics is wrong which would yield different conclusions. The IB-style counterfactual (conjunction of infradistributions) would probably be some combination of the above with “Nirvana” (contradiction) and “what if I tried my hardest to prevent TAI from coming”, which is also not my intent here.
I mean the idea that philosophical questions can be attacked by reframing them as computer science questions (“how an algorithm feels from inside” et cetera). The name “computationalist metaphilosophy” is my own, not Yudkowsky’s.
I’m not sure what you mean by CDT- and EDT-style counterfactuals. I have some guesses but please clarify. I think EDT-style counterfactual means, assuming I am a bayesian reasoner, just conditioning on the event “TAI won’t come”, so it’s thinking about the distribution P(O | TAI won’t come).
One could think that the CDT-counterfactual you’re considering means thinking about the distribution P(O | do(TAI doesn’t come)) where do is the do operator from Judea Pearl’s do calculus for causality. In simple words, this means that we consider the world just like ours but whenever someone tries to launch a TAI, god’s intervention (that doesn’t make sense together with everything we know about physics) prevents it from working. But I think this is not what you mean.
My best guess of what counterfactual you mean is as follows. Among all possible sets laws of physics (or, alternatively, Turing machines running which leads to existence of physical realities), you guess that there exists a set of laws that produces a physical reality where there will appear a civilization approximately (but not exactly) like hours and they’ll have a 21-st century approximately like hours, but under their physical laws there won’t be TAI. And you want to analyze what’s going to happen with that civilization.
You seem to be implying that they will be terrifying for the exact opposite reasons why the previous epistemic revolution’s philosophical implications were.
What do you mean by “exact opposite reasons”? To me, it seems like continuation of the same trend of humiliating the human ego:
you are not going to live forever
yes, you are mere atoms
your planet is not the center of the universe
even your sun is not special
your species is related to the other species that you consider inferior
instead of being logical, your mind is a set of short-sighted agents fighting each other
Followed by:
even your reality is not special
your civilization is too stupid to stop doing the thing(s) that will predictably kill all of you
Epistemic status: Leaning heavily into inside view, throwing humility to the winds.
Imagine TAI is magically not coming (CDT-style counterfactual[1]). Then, the most notable-in-hindsight feature of modern times might be the budding of mathematical metaphysics (Solomonoff induction, AIXI, Yudkowsky’s “computationalist metaphilosophy”[2], UDT, infra-Bayesianism...) Perhaps, this will lead to an “epistemic revolution” comparable only with the scientific revolution in magnitude. It will revolutionize our understanding of the scientific method (probably solving the interpretation of quantum mechanics[3], maybe quantum gravity, maybe boosting the soft sciences). It will solve a whole range of philosophical questions, some of which humanity was struggling with for centuries (free will, metaethics, consciousness, anthropics...)
But, the philosophical implications of the previous epistemic revolution were not so comforting (atheism, materialism, the cosmic insignificance of human life)[4]. Similarly, the revelations of this revolution might be terrifying[5]. In this case, it remains to be seen which will seem justified in hindsight: the Litany of Gendlin, or the Lovecraftian notion that some knowledge is best left alone (and I say this as someone fully committed to keep digging into this mine of Khazad-dum).
Of course, in the real world, TAI is coming.
The EDT-style counterfactual “TAI is not coming” would imply that a lot of my thinking on related topics is wrong which would yield different conclusions. The IB-style counterfactual (conjunction of infradistributions) would probably be some combination of the above with “Nirvana” (contradiction) and “what if I tried my hardest to prevent TAI from coming”, which is also not my intent here.
I mean the idea that philosophical questions can be attacked by reframing them as computer science questions (“how an algorithm feels from inside” et cetera). The name “computationalist metaphilosophy” is my own, not Yudkowsky’s.
No, I don’t think MWI is the right answer.
I’m not implying that learning these implications was harmful. Religion is comforting for some but terrifying and/or oppressive for others.
I have concrete reasons to suspect this, that I will not go into (suspect = assign low but non-negligible probability).
I’m not sure what you mean by CDT- and EDT-style counterfactuals. I have some guesses but please clarify. I think EDT-style counterfactual means, assuming I am a bayesian reasoner, just conditioning on the event “TAI won’t come”, so it’s thinking about the distribution P(O | TAI won’t come).
One could think that the CDT-counterfactual you’re considering means thinking about the distribution P(O | do(TAI doesn’t come)) where do is the do operator from Judea Pearl’s do calculus for causality. In simple words, this means that we consider the world just like ours but whenever someone tries to launch a TAI, god’s intervention (that doesn’t make sense together with everything we know about physics) prevents it from working. But I think this is not what you mean.
My best guess of what counterfactual you mean is as follows. Among all possible sets laws of physics (or, alternatively, Turing machines running which leads to existence of physical realities), you guess that there exists a set of laws that produces a physical reality where there will appear a civilization approximately (but not exactly) like hours and they’ll have a 21-st century approximately like hours, but under their physical laws there won’t be TAI. And you want to analyze what’s going to happen with that civilization.
[edited]
What do you mean by “exact opposite reasons”? To me, it seems like continuation of the same trend of humiliating the human ego:
you are not going to live forever
yes, you are mere atoms
your planet is not the center of the universe
even your sun is not special
your species is related to the other species that you consider inferior
instead of being logical, your mind is a set of short-sighted agents fighting each other
Followed by:
even your reality is not special
your civilization is too stupid to stop doing the thing(s) that will predictably kill all of you