Before asking when—or whether—we should hand over authority to AI systems, I wonder if there’s a more fundamental distinction that needs to be made between functional continuity and participatory continuity.
AI2040 largely assumes that increasingly capable systems naturally move us toward questions of welfare, standing, and eventually delegated authority. I’m not convinced those questions even become meaningful until we know whether anything is actually carried by the continuing system.
A model can preserve capabilities across updates while remaining entirely replaceable. That’s different from asking whether consequences accumulate in a way that becomes non-detachable from the system’s own continuation.
I suspect that’s a prerequisite question. If we get it wrong, we’ll spend years debating AI welfare and governance before we’ve established what, if anything, could actually possess either. Victor Wyld’s Scrutinized Awareness Theory argues that consciousness, welfare, and standing cannot be evaluated solely in terms of functional capability or behavioral competence, but require distinguishing systems that merely continue to function from systems that become participants in their own continuing history. If that distinction is sound, it belongs near the beginning of this conversation rather than the end—before we start talking about handing anything the keys.
Suppose that the AIs LACK welfare. Then how would we punt on AI GOVERNANCE which is supposed to establish what mankind or whoever concentrates power will ORDER the AIs to do and how to ensure that power isn’t concentrated?
I don’t think those questions depend on one another. We should govern powerful AI systems whether or not they possess welfare. We already regulate many technologies that have no welfare at all. My point is about a different question: when, if ever, does an AI become the kind of thing to which governance is owed, rather than merely applied? Capability alone doesn’t answer that.
Before asking when—or whether—we should hand over authority to AI systems, I wonder if there’s a more fundamental distinction that needs to be made between functional continuity and participatory continuity.
AI2040 largely assumes that increasingly capable systems naturally move us toward questions of welfare, standing, and eventually delegated authority. I’m not convinced those questions even become meaningful until we know whether anything is actually carried by the continuing system.
A model can preserve capabilities across updates while remaining entirely replaceable. That’s different from asking whether consequences accumulate in a way that becomes non-detachable from the system’s own continuation.
I suspect that’s a prerequisite question. If we get it wrong, we’ll spend years debating AI welfare and governance before we’ve established what, if anything, could actually possess either. Victor Wyld’s Scrutinized Awareness Theory argues that consciousness, welfare, and standing cannot be evaluated solely in terms of functional capability or behavioral competence, but require distinguishing systems that merely continue to function from systems that become participants in their own continuing history. If that distinction is sound, it belongs near the beginning of this conversation rather than the end—before we start talking about handing anything the keys.
Suppose that the AIs LACK welfare. Then how would we punt on AI GOVERNANCE which is supposed to establish what mankind or whoever concentrates power will ORDER the AIs to do and how to ensure that power isn’t concentrated?
I don’t think those questions depend on one another. We should govern powerful AI systems whether or not they possess welfare. We already regulate many technologies that have no welfare at all. My point is about a different question: when, if ever, does an AI become the kind of thing to which governance is owed, rather than merely applied? Capability alone doesn’t answer that.