Critch thinks something like “how you and your species conduct yourselves during the acute risk period is a stronger consideration”?
I wouldn’t argue hard for it being “a stronger consideration” because I think that’s a harder question.
But I would argue hard for it being “a consideration”, especially if someone was like “boo this is worthless and should round to zero”. And, if someone finds the “cosmic” framing distracting, I would also argue the following, in terms of real-world relevance this century:
humanity and AI are both less likely to wind up in mutually destructive conflict if we both pay non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
individual groups of humans are less likely to get into needlessly destructive AI-powered wars with each other if we pay some non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
humanity and AI are both less likely to face simulation shutdown if we pay nonzero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
These would primarily not be arguments to override all considerations, but to be considerations at all.
Nod. (fwiw I meant “stronger consideration” to be “stronger than I think habryka thinks it is”in relative terms)
humanity and AI are both less likely to face simulation shutdown if we pay nonzero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
I was thinking about this last night, and then remembered in Acausal normalcy you had also argued that that simulation is pretty expensive and probably not how most acausal interaction probably works, and then was a bit confused about what you were arguing here and/or what was likely to be true, a la
Writing out arguments or formal proofs about each other is much more computationally efficient, because nested arguments naturally avoid stack overflows in a way that nested simulations do not.
Personally I think the world as we know it is more likely to be in a vivarium than a simulation, though many of the same principles apply terms of there being a powerful outside force that can shut down Earth-originating civilization if it appears to have bad morals. And yes, people writing proofs about us without simulating us is another pathway for our actions to matter separately from their naive causal consequences. And, although fewer people understand the proof-theoretic angle, do I think the effect is even stronger than the simulation effect because simulations are just a special case of proofs.
I wouldn’t argue hard for it being “a stronger consideration” because I think that’s a harder question.
But I would argue hard for it being “a consideration”, especially if someone was like “boo this is worthless and should round to zero”. And, if someone finds the “cosmic” framing distracting, I would also argue the following, in terms of real-world relevance this century:
humanity and AI are both less likely to wind up in mutually destructive conflict if we both pay non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
individual groups of humans are less likely to get into needlessly destructive AI-powered wars with each other if we pay some non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
humanity and AI are both less likely to face simulation shutdown if we pay nonzero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
These would primarily not be arguments to override all considerations, but to be considerations at all.
Nod. (fwiw I meant “stronger consideration” to be “stronger than I think habryka thinks it is”in relative terms)
I was thinking about this last night, and then remembered in Acausal normalcy you had also argued that that simulation is pretty expensive and probably not how most acausal interaction probably works, and then was a bit confused about what you were arguing here and/or what was likely to be true, a la
Personally I think the world as we know it is more likely to be in a vivarium than a simulation, though many of the same principles apply terms of there being a powerful outside force that can shut down Earth-originating civilization if it appears to have bad morals. And yes, people writing proofs about us without simulating us is another pathway for our actions to matter separately from their naive causal consequences. And, although fewer people understand the proof-theoretic angle, do I think the effect is even stronger than the simulation effect because simulations are just a special case of proofs.