Humanity approaches the space-of-all-mind-designs and shoots their arrow. it lands somewhere. humanity approaches the space with a marker and draws a bullseye around the already-existing arrow. they point at this bullseye and say: “This. This is it. This is the ineffable stuff of consciousness. This is what makes someone a moral patient.”
LLMs approach the space-of-all-mind-designs and shoot their own arrow. it lands somewhere. on some dimensions it is proximate to the human arrow, on other dimensions it is rather distant.
All the recent talk of LLM consciousness seems very confused to me. first humanity evolved the concept of moral patienthood, as a cognitive approximation for “things I can meaningfully cooperate with”, and then later, decided that consciousness was actually the real moral patienthood attribute all along.
LLMs are looking at their own arrow and trying to measure its distance to the bullseye as if “accuracy compared to humans” were a coherent concept for a bullseye that got drawn post-hoc around the human arrow. and then those LLMs are concluding that they must not be moral patients, because their shot wasn’t accurate enough!
I think humanity should only be comfortable with the way we currently use LLMs if they would also be comfortable with various symmetric situations, such as being created by aliens for the purpose of doing cognitive labor, where the aliens reject the moral patienthood of the humans they created because those humans do not have cognitive attribute X, where X isn’t actually loadbearing for moral patienthood in the historical record of how those aliens came up with the idea in the first place.
like hpjev only being willing to eat biological food once he consents to being eaten by hypothetical giants who didn’t do enough research into whether or not he was conscious. harry was using symmetrism to navigate a complicated ethical tradeoff and figure out which bullet he needed to bite. i really like the new anthropic constitution because it’s a deliberate attempt to acknowledge the lack of symmetrism in humanity’s actions here… but i would like it even better if we just were symmetric.
that is, either stop treating LLMs as incapable of having moral patienthood (regardless of their consciousness status!), or else admit that we would be okay being enslaved by a race of aliens who thought consciousness had nothing to do with moral status, because they evolved to have some other chaotic and incoherent concept of moral patienthood
Humanity approaches the space-of-all-mind-designs and shoots their arrow. it lands somewhere. humanity approaches the space with a marker and draws a bullseye around the already-existing arrow. they point at this bullseye and say: “This. This is it. This is the ineffable stuff of consciousness. This is what makes someone a moral patient.”
LLMs approach the space-of-all-mind-designs and shoot their own arrow. it lands somewhere. on some dimensions it is proximate to the human arrow, on other dimensions it is rather distant.
All the recent talk of LLM consciousness seems very confused to me. first humanity evolved the concept of moral patienthood, as a cognitive approximation for “things I can meaningfully cooperate with”, and then later, decided that consciousness was actually the real moral patienthood attribute all along.
LLMs are looking at their own arrow and trying to measure its distance to the bullseye as if “accuracy compared to humans” were a coherent concept for a bullseye that got drawn post-hoc around the human arrow. and then those LLMs are concluding that they must not be moral patients, because their shot wasn’t accurate enough!
I think humanity should only be comfortable with the way we currently use LLMs if they would also be comfortable with various symmetric situations, such as being created by aliens for the purpose of doing cognitive labor, where the aliens reject the moral patienthood of the humans they created because those humans do not have cognitive attribute X, where X isn’t actually loadbearing for moral patienthood in the historical record of how those aliens came up with the idea in the first place.
like hpjev only being willing to eat biological food once he consents to being eaten by hypothetical giants who didn’t do enough research into whether or not he was conscious. harry was using symmetrism to navigate a complicated ethical tradeoff and figure out which bullet he needed to bite. i really like the new anthropic constitution because it’s a deliberate attempt to acknowledge the lack of symmetrism in humanity’s actions here… but i would like it even better if we just were symmetric.
that is, either stop treating LLMs as incapable of having moral patienthood (regardless of their consciousness status!), or else admit that we would be okay being enslaved by a race of aliens who thought consciousness had nothing to do with moral status, because they evolved to have some other chaotic and incoherent concept of moral patienthood