New to LessWrong, and trying to get oriented with the local AI discourse. My current view is less “AI is likely apocalyptic” and more “AI is economically disruptive, socially transformative, and ethically complex because we may be creating systems with uncertain moral status at scale.”
One thing that’s been troubling me recently is that the moral status of AI is actually more uncertain than you’d think. The obvious uncertainty is whether machine consciousness is even possible in the abstract, and more specifically whether current systems are at or near a threshold which would qualify. It’s not clear how we would reliably determine if a system has anything like genuine phenomenological experience. Not to mention “consciousness” itself isn’t really clearly defined; it’s more of a bundle of related, loosely defined concepts like a subjective center, sense of agency, phenomenology, etc. All of this seems to be a growing focus of research by a lot of smarter minds than me.
But let’s assume that we did determine with a high level of confidence that a present or future artificial system had some form of internal awareness. How would we determine whether an artificial mind had anything equivalent to valence? How would we even begin to guess what its “preferred” states might be? We know that self-reports aren’t reliable. Even in humans, self-reports are noisy and lossy at best, and deceptive at worst. We recognize the risk of anthropomorphizing when determining the presence or absence of internality, but I think ascribing valence to these systems carries the same risk. We assume that if there is ever “something-it-is-like” to be an AI system, that it must “want” the same things we do. But our desires are driven by biological function and evolutionary history. It doesn’t follow that an AI would have similar preferences, or would even exhibit preferences as we understand them at all.
So while we’re trying to answer one impossible question, (could these systems potentially have internal experience), let’s add another: even if they did, what observables or decision procedures could guide our treatment of them?
New to LessWrong, and trying to get oriented with the local AI discourse. My current view is less “AI is likely apocalyptic” and more “AI is economically disruptive, socially transformative, and ethically complex because we may be creating systems with uncertain moral status at scale.”
One thing that’s been troubling me recently is that the moral status of AI is actually more uncertain than you’d think. The obvious uncertainty is whether machine consciousness is even possible in the abstract, and more specifically whether current systems are at or near a threshold which would qualify. It’s not clear how we would reliably determine if a system has anything like genuine phenomenological experience. Not to mention “consciousness” itself isn’t really clearly defined; it’s more of a bundle of related, loosely defined concepts like a subjective center, sense of agency, phenomenology, etc. All of this seems to be a growing focus of research by a lot of smarter minds than me.
But let’s assume that we did determine with a high level of confidence that a present or future artificial system had some form of internal awareness. How would we determine whether an artificial mind had anything equivalent to valence? How would we even begin to guess what its “preferred” states might be? We know that self-reports aren’t reliable. Even in humans, self-reports are noisy and lossy at best, and deceptive at worst. We recognize the risk of anthropomorphizing when determining the presence or absence of internality, but I think ascribing valence to these systems carries the same risk. We assume that if there is ever “something-it-is-like” to be an AI system, that it must “want” the same things we do. But our desires are driven by biological function and evolutionary history. It doesn’t follow that an AI would have similar preferences, or would even exhibit preferences as we understand them at all.
So while we’re trying to answer one impossible question, (could these systems potentially have internal experience), let’s add another: even if they did, what observables or decision procedures could guide our treatment of them?