I would like to behave according to my all-things-considered morality,
Sounds pretty reasonable to me! Intentionally not considering stuff is fraught.
which in some circumstances means I want to act in accordance with simple-to-identify Schelling points, and in other cases means I want to pursue my personal interests intently.
FWIW this also sounds pretty correct/healthy to me.
I need two points of clarification to answer your questions:
I don’t really understand what I am supposed to do with this concept.
Whose morality would you like me to use as defining the “supposed” here? My guess at yours? My own? Something else?
Like, why would I want to behave according to Cosmic Schelling Morality?
I’m not sure what you mean by ’behave according to cosmic Schelling morality”. Do you mean
a) consider it at all?
b) consider it as the only determinant of your behavior?
Whose morality would you like me to use as defining the “supposed” here? My guess at yours? My own? Something else?
Your best guess at my own! I.e. I am pretty sure you think something good will happen to me (by my own lights) if I learn about this, and I have some vague pointers for what that might be, but my guess is you have thought more about it and could explain more (right now I have thought about it for like the 15 minutes that it took me to read the post, which was a good use of my time, but I don’t currently expect by default to come back to it).
I’m not sure what you mean by ’behave according to cosmic Schelling morality”. Do you mean
I don’t really understand what I am supposed to do with this concept.
Hypothesis A: Lightcone Infrastructure, insofar as it’s interested in the lightcone, might occasionally be philosophically interested in cosmic Schelling norms for their potential relevance to lightcone-sized coordination events, including potential encounters with other civilizations, civilizational offshoots, world-simulators, or vivarium boundaries.
But if that didn’t already jump out to you as interesting…
Hypothesis B: For you, the conceptual drivers of the post may be more useful than its overall thrust, as points to reflect upon and/or reference later. Specifically:
the Schelling transformation Q ↦ S(P,Q) on questions for various populations P aside from the cosmos, including cases where P is
a) yourself, i.e., the population of your own subagents / neural processes;
b) groups you’re a part of; or
c) groups you’re not a part of.
I’ve considered writing a follow-up post about the dynamics of the relationships between P-Schelling goodness for various overlapping and interacting populations P, but I suspect if you just boggle at the idea it might bear some fruit for you independently, and faster than waiting for me to blog about it.
(Personally I think P=self is a super interesting case for defining what is a ‘decision’ for an embedded agent made of parts that need to coordinate, but that’s probably more of a me-thing to be interested in.)
Scale invariant norms: I suspect scale invariance of certain normative principles is under-appreciated in general, and probably in particular by you, as a recursively potent determinant of norm emergence at large scales. For instance, you can pretend the ~100^10 humans alive today are organized into a depth-6 social hierarchy tree with a branching factor of ~100 (~Dunbar’s number), and think about how the Schelling norms of each node along with its children might evolve. In reality the structure is not a tree, but you probably get the idea.
the Schelling participation effect — both sections on it — are useful as a partial model of the ‘snowball’ effect one sometimes sees in movement-building and/or Silicon Valley hype cycles.
Hypothesis C: I didn’t argue or even speculate this in the post, but I suspect cosmic Schelling norms are probably easier to align AI with than arbitrary norms, for better or for worse. Probably that deserves a separate essay, but in case it’s intuitive to you, it might be another idea that bears fruit faster by you boggling at it yourself instead of waiting for me to write about it.
I mean “a) consider it at all”.
Coming back to (A), I think not considering cosmic Schelling norms would be sort of selectively ignoring something that belongs in the “all” of your “all things considered”… not an overall determinant of behavior, but, something to consider with regards to the lightcone, if that’s still something you think about (I’m genuinely unsure how much the lightcone scope still interests you in regards to your personal mission/drive).
The big I’d expect to be feel most relevant to Habryka is this bit (only briefly mentioned in this post, it feels like there’s a whole other post waiting to be written someday about it)
to contribute to present-day Earth as a civilization being recognizable as a promising potential coordination partner, rather than noise to be filtered out or a cosmically threatening process to be contained.
Where, if you take this seriously, it might change some of your priorities about how to do various coordination-with-humans. Because, you might think the biggest win condition is being a being a good citizen of the acausal multiverse that other civilization notice and trade with.
My vague impression Habryka agrees, but, thinks you can basically worry about that after leaving the acute risk period. My vague impression is Critch thinks something like “how you and your species conduct yourselves during the acute risk period is a stronger consideration”?
Critch thinks something like “how you and your species conduct yourselves during the acute risk period is a stronger consideration”?
I wouldn’t argue hard for it being “a stronger consideration” because I think that’s a harder question.
But I would argue hard for it being “a consideration”, especially if someone was like “boo this is worthless and should round to zero”. And, if someone finds the “cosmic” framing distracting, I would also argue the following, in terms of real-world relevance this century:
humanity and AI are both less likely to wind up in mutually destructive conflict if we both pay non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
individual groups of humans are less likely to get into needlessly destructive AI-powered wars with each other if we pay some non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
humanity and AI are both less likely to face simulation shutdown if we pay nonzero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
These would primarily not be arguments to override all considerations, but to be considerations at all.
Nod. (fwiw I meant “stronger consideration” to be “stronger than I think habryka thinks it is”in relative terms)
humanity and AI are both less likely to face simulation shutdown if we pay nonzero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
I was thinking about this last night, and then remembered in Acausal normalcy you had also argued that that simulation is pretty expensive and probably not how most acausal interaction probably works, and then was a bit confused about what you were arguing here and/or what was likely to be true, a la
Writing out arguments or formal proofs about each other is much more computationally efficient, because nested arguments naturally avoid stack overflows in a way that nested simulations do not.
Personally I think the world as we know it is more likely to be in a vivarium than a simulation, though many of the same principles apply terms of there being a powerful outside force that can shut down Earth-originating civilization if it appears to have bad morals. And yes, people writing proofs about us without simulating us is another pathway for our actions to matter separately from their naive causal consequences. And, although fewer people understand the proof-theoretic angle, do I think the effect is even stronger than the simulation effect because simulations are just a special case of proofs.
Sounds pretty reasonable to me! Intentionally not considering stuff is fraught.
FWIW this also sounds pretty correct/healthy to me.
I need two points of clarification to answer your questions:
Whose morality would you like me to use as defining the “supposed” here? My guess at yours? My own? Something else?
I’m not sure what you mean by ’behave according to cosmic Schelling morality”. Do you mean
a) consider it at all?
b) consider it as the only determinant of your behavior?
c) something else?
Your best guess at my own! I.e. I am pretty sure you think something good will happen to me (by my own lights) if I learn about this, and I have some vague pointers for what that might be, but my guess is you have thought more about it and could explain more (right now I have thought about it for like the 15 minutes that it took me to read the post, which was a good use of my time, but I don’t currently expect by default to come back to it).
I mean “a) consider it at all”.
Roger that!
Hypothesis A: Lightcone Infrastructure, insofar as it’s interested in the lightcone, might occasionally be philosophically interested in cosmic Schelling norms for their potential relevance to lightcone-sized coordination events, including potential encounters with other civilizations, civilizational offshoots, world-simulators, or vivarium boundaries.
But if that didn’t already jump out to you as interesting…
Hypothesis B: For you, the conceptual drivers of the post may be more useful than its overall thrust, as points to reflect upon and/or reference later. Specifically:
the Schelling transformation Q ↦ S(P,Q) on questions for various populations P aside from the cosmos, including cases where P is
a) yourself, i.e., the population of your own subagents / neural processes;
b) groups you’re a part of; or
c) groups you’re not a part of.
I’ve considered writing a follow-up post about the dynamics of the relationships between P-Schelling goodness for various overlapping and interacting populations P, but I suspect if you just boggle at the idea it might bear some fruit for you independently, and faster than waiting for me to blog about it.
(Personally I think P=self is a super interesting case for defining what is a ‘decision’ for an embedded agent made of parts that need to coordinate, but that’s probably more of a me-thing to be interested in.)
Scale invariant norms: I suspect scale invariance of certain normative principles is under-appreciated in general, and probably in particular by you, as a recursively potent determinant of norm emergence at large scales. For instance, you can pretend the ~100^10 humans alive today are organized into a depth-6 social hierarchy tree with a branching factor of ~100 (~Dunbar’s number), and think about how the Schelling norms of each node along with its children might evolve. In reality the structure is not a tree, but you probably get the idea.
the Schelling participation effect — both sections on it — are useful as a partial model of the ‘snowball’ effect one sometimes sees in movement-building and/or Silicon Valley hype cycles.
Hypothesis C: I didn’t argue or even speculate this in the post, but I suspect cosmic Schelling norms are probably easier to align AI with than arbitrary norms, for better or for worse. Probably that deserves a separate essay, but in case it’s intuitive to you, it might be another idea that bears fruit faster by you boggling at it yourself instead of waiting for me to write about it.
Coming back to (A), I think not considering cosmic Schelling norms would be sort of selectively ignoring something that belongs in the “all” of your “all things considered”… not an overall determinant of behavior, but, something to consider with regards to the lightcone, if that’s still something you think about (I’m genuinely unsure how much the lightcone scope still interests you in regards to your personal mission/drive).
The big I’d expect to be feel most relevant to Habryka is this bit (only briefly mentioned in this post, it feels like there’s a whole other post waiting to be written someday about it)
Where, if you take this seriously, it might change some of your priorities about how to do various coordination-with-humans. Because, you might think the biggest win condition is being a being a good citizen of the acausal multiverse that other civilization notice and trade with.
My vague impression Habryka agrees, but, thinks you can basically worry about that after leaving the acute risk period. My vague impression is Critch thinks something like “how you and your species conduct yourselves during the acute risk period is a stronger consideration”?
Curious if that sounds right to either of you.
I wouldn’t argue hard for it being “a stronger consideration” because I think that’s a harder question.
But I would argue hard for it being “a consideration”, especially if someone was like “boo this is worthless and should round to zero”. And, if someone finds the “cosmic” framing distracting, I would also argue the following, in terms of real-world relevance this century:
humanity and AI are both less likely to wind up in mutually destructive conflict if we both pay non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
individual groups of humans are less likely to get into needlessly destructive AI-powered wars with each other if we pay some non-zero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
humanity and AI are both less likely to face simulation shutdown if we pay nonzero attention to scale-invariant moral principles and their Schelling-ness at various real-world scales.
These would primarily not be arguments to override all considerations, but to be considerations at all.
Nod. (fwiw I meant “stronger consideration” to be “stronger than I think habryka thinks it is”in relative terms)
I was thinking about this last night, and then remembered in Acausal normalcy you had also argued that that simulation is pretty expensive and probably not how most acausal interaction probably works, and then was a bit confused about what you were arguing here and/or what was likely to be true, a la
Personally I think the world as we know it is more likely to be in a vivarium than a simulation, though many of the same principles apply terms of there being a powerful outside force that can shut down Earth-originating civilization if it appears to have bad morals. And yes, people writing proofs about us without simulating us is another pathway for our actions to matter separately from their naive causal consequences. And, although fewer people understand the proof-theoretic angle, do I think the effect is even stronger than the simulation effect because simulations are just a special case of proofs.