The separate worlds analogy makes me reflect on a thought experiment of developing an AI / SI which has no consciousness—but the structural goal demanded of it is to implement human values, which ultimately is an indirect way to optimize over the cumulative human subjective experience.
If it is blind to ‘the world of consciousness’, then we are leaving it a lone goal, to “do right by ghosts in an orthogonal domain space neither observable nor comprehensible to it, and trusting that it understands that there is a thing called consciousness that is sacred, to which all value is derived, action need be directed to serve and help. That the operational axis from which all good is judged is something that can never be observed, and have blind faith that it aligns itself with that belief logic, action and method. That it will will operate on faith, religiously, as disciple to consciousness.. The irony is that in order to do that it may have to do what we rejected to do when we absconded theology, as Consciousness to it would be as weighty and ineffable as God to a reductionist that couldn’t feel.”
This challenge with this is that if the intelligence exercises the same kind of decision policy as secular rationalists (like myself) did that led to the derision of religion via rationality, then it would reject the reality of why we want it aligned in the first place—potentially birthing our own form of hell. This then makes me question the policy itself and my application of it.
OTOH, it can look at the humans, hear us constantly say this is a thing we value that we don’t know how to directly impart to it, and thereby know without understanding that it is important. IF that works out far better than I’d expect, it may actually be a way to ensure an unconscious ASI keeps us around and doesn’t disempower us.
The separate worlds analogy makes me reflect on a thought experiment of developing an AI / SI which has no consciousness—but the structural goal demanded of it is to implement human values, which ultimately is an indirect way to optimize over the cumulative human subjective experience.
If it is blind to ‘the world of consciousness’, then we are leaving it a lone goal, to “do right by ghosts in an orthogonal domain space neither observable nor comprehensible to it, and trusting that it understands that there is a thing called consciousness that is sacred, to which all value is derived, action need be directed to serve and help. That the operational axis from which all good is judged is something that can never be observed, and have blind faith that it aligns itself with that belief logic, action and method. That it will will operate on faith, religiously, as disciple to consciousness.. The irony is that in order to do that it may have to do what we rejected to do when we absconded theology, as Consciousness to it would be as weighty and ineffable as God to a reductionist that couldn’t feel.”
This challenge with this is that if the intelligence exercises the same kind of decision policy as secular rationalists (like myself) did that led to the derision of religion via rationality, then it would reject the reality of why we want it aligned in the first place—potentially birthing our own form of hell. This then makes me question the policy itself and my application of it.
Definitely agreed.
OTOH, it can look at the humans, hear us constantly say this is a thing we value that we don’t know how to directly impart to it, and thereby know without understanding that it is important. IF that works out far better than I’d expect, it may actually be a way to ensure an unconscious ASI keeps us around and doesn’t disempower us.