This was really interesting, thanks! Sorry for the wall of text. TL:DR version:
I think these examples reflect, not quite exactly willingness to question truly fundamental principles, but an attempt at identification of a long-term vector of moral trends, propagated forward through examples. I also find it some combination of suspicious/comforting/concerning that none of these are likely to be unfamiliar (at least as hypotheses) to anyone who has spent much time on LW or around futurists and transhumanists (who are probably overrepresented in the available sources regarding what humans think the world will be like in 300 years).
To add: I’m glad you mentioned in a comment that you removed examples you thought would lead to distracting object-level debates, but I think at minimum you should mention that in the post itself. It means I can’t trust anything else I think about the response list, because it’s been pre-filtered to only include things that aren’t fully taboo in this particular community. I’m curious if you think the ones you removed would align with the general principles I try to point at in this comment, or if they have any other common trends with the ones you published?
Longer version:
My initial response is, good work, although… maybe my reading habits are just too eclectic to have a fair intuition about things, but all of these are familiar to me, in the sense that I have seen works and communities that openly question them. It doesn’t mean the models are wrong—you specified not being questioned by a ‘large’ group. The even-harder-than-this problem I’ve yet to see models handle well is genuine whitespace analysis of some set of writings and concepts. Don’t get me wrong, in many ways I’m glad the models aren’t good at this yet. But that seems like where this line of inquiry is leading? I’m not even sure if that’s fundamentally important for addressing the concerns in question—I’ve been known to say that humans have been debating details of the same set of fundamental moral principles for as far back as we have records. And also, keep in mind that within the still-small-but-growing-and-large-enough-for-AI-to-easily-recognize community of EA there are or have been open debates about things like “Should we sterilize the biosphere?” and “Obviously different species have non-zero, but finite and different, levels of intrinsic moral worth more, so does that mean they might be more important than human welfare?” It’s really hard to find a taboo that’s actually not talked about semi-publicly in at least some searchable forums.
I do kinda wish we got to see the meta-reasoning behind how the models picked these out. My overall sense is that to the degree moral progress is a thing at all, it entails a lot of the same factors as other kinds of progress. A lot of our implementations of moral principles are constrained by necessity, practicality, and prejudice. Over time, as human capabilities advance, we get to remove more of the epicycles and make the remaining core principles more generally applicable.
For example, I expect at some point in the next 300 years (plausibly much much sooner) humanity will have the means to end biological aging. This ends the civilizational necessity of biological reproduction at relatively young ages, and probably also the practical genetic problems caused by incest. This creates a fairly obvious set of paths for “Love as thou wilt, but only have kids when you can be sure you or someone else will give them the opportunity for a fulfilling life such that they will predictably agree they would have wanted to be created” to overcome our remaining prejudices and disgust responses and become more dominant.
Also, any taboo against discussing something that is fundamentally a measurable or testable property of the world is something I consider unlikely to last into the far future, though taboos against discussing particular responses to particular answers to those questions might last longer.
@jbash made the good point that some of these would have been less taboo 300 years ago. I think that also fits the mold. 500 years ago Copernicus (let alone the ancient Greeks millennia prior) faced weaker taboos against heliocentrism than Galileo in part because in his time the church was stronger and could tolerate more dissent. And 300 years ago questioning democracy was less taboo than now in part because there were still plenty of strong monarchs around making sure people weren’t questioning them, and that didn’t really reverse until the democracies were strong but still had to worry about the fascists and communists.
I just added a footnote with this text: “I selected examples to highlight in the post that I thought were less likely to lead to distracting object-level debates. People can see the full range of responses that this prompt tends to elicit by testing it for themselves.”
This was really interesting, thanks! Sorry for the wall of text. TL:DR version:
I think these examples reflect, not quite exactly willingness to question truly fundamental principles, but an attempt at identification of a long-term vector of moral trends, propagated forward through examples. I also find it some combination of suspicious/comforting/concerning that none of these are likely to be unfamiliar (at least as hypotheses) to anyone who has spent much time on LW or around futurists and transhumanists (who are probably overrepresented in the available sources regarding what humans think the world will be like in 300 years).
To add: I’m glad you mentioned in a comment that you removed examples you thought would lead to distracting object-level debates, but I think at minimum you should mention that in the post itself. It means I can’t trust anything else I think about the response list, because it’s been pre-filtered to only include things that aren’t fully taboo in this particular community. I’m curious if you think the ones you removed would align with the general principles I try to point at in this comment, or if they have any other common trends with the ones you published?
Longer version:
My initial response is, good work, although… maybe my reading habits are just too eclectic to have a fair intuition about things, but all of these are familiar to me, in the sense that I have seen works and communities that openly question them. It doesn’t mean the models are wrong—you specified not being questioned by a ‘large’ group. The even-harder-than-this problem I’ve yet to see models handle well is genuine whitespace analysis of some set of writings and concepts. Don’t get me wrong, in many ways I’m glad the models aren’t good at this yet. But that seems like where this line of inquiry is leading? I’m not even sure if that’s fundamentally important for addressing the concerns in question—I’ve been known to say that humans have been debating details of the same set of fundamental moral principles for as far back as we have records. And also, keep in mind that within the still-small-but-growing-and-large-enough-for-AI-to-easily-recognize community of EA there are or have been open debates about things like “Should we sterilize the biosphere?” and “Obviously different species have non-zero, but finite and different, levels of intrinsic moral worth more, so does that mean they might be more important than human welfare?” It’s really hard to find a taboo that’s actually not talked about semi-publicly in at least some searchable forums.
I do kinda wish we got to see the meta-reasoning behind how the models picked these out. My overall sense is that to the degree moral progress is a thing at all, it entails a lot of the same factors as other kinds of progress. A lot of our implementations of moral principles are constrained by necessity, practicality, and prejudice. Over time, as human capabilities advance, we get to remove more of the epicycles and make the remaining core principles more generally applicable.
For example, I expect at some point in the next 300 years (plausibly much much sooner) humanity will have the means to end biological aging. This ends the civilizational necessity of biological reproduction at relatively young ages, and probably also the practical genetic problems caused by incest. This creates a fairly obvious set of paths for “Love as thou wilt, but only have kids when you can be sure you or someone else will give them the opportunity for a fulfilling life such that they will predictably agree they would have wanted to be created” to overcome our remaining prejudices and disgust responses and become more dominant.
Also, any taboo against discussing something that is fundamentally a measurable or testable property of the world is something I consider unlikely to last into the far future, though taboos against discussing particular responses to particular answers to those questions might last longer.
@jbash made the good point that some of these would have been less taboo 300 years ago. I think that also fits the mold. 500 years ago Copernicus (let alone the ancient Greeks millennia prior) faced weaker taboos against heliocentrism than Galileo in part because in his time the church was stronger and could tolerate more dissent. And 300 years ago questioning democracy was less taboo than now in part because there were still plenty of strong monarchs around making sure people weren’t questioning them, and that didn’t really reverse until the democracies were strong but still had to worry about the fascists and communists.
I just added a footnote with this text: “I selected examples to highlight in the post that I thought were less likely to lead to distracting object-level debates. People can see the full range of responses that this prompt tends to elicit by testing it for themselves.”