Just how diverse is human motivation? Should we discount even sophisticated versions of psychological hedonism? Undoubtedly, the “pleasure principle” is simplistic as it stands. But one good reason not to try heroin, for example, is precisely that the reward architecture of our opioid pathways is so similar. Previously diverse life-projects of first-time heroin users are at risk of converging on a common outcome. So more broadly, let’s consider the class of life-supporting Hubble volumes where sentient biological robots acquire the capacity to rewrite their genetic source code and gain mastery of their own reward circuitry. May we predict orthogonality or convergence? Certainly, there are strong arguments why such intelligences won’t all become the functional equivalent of heroin addicts or wireheads or Nozick Experience Machine VR-heads (etc). One such argument is the nature of selection pressure. But _if_some version of the pleasure principle is correct, then isn’t some version of the convergence conjecture at least feasible, i.e. they’ll recalibrate the set-point of their hedonic treadmill and enjoy gradients of (super)intelligent (super)happiness? One needn’t be a meta-ethical value-realist to acknowledge that subjects of experience universally find bliss is empirically more valuable than agony or despair. The present inability of natural science to explain first-person experiences doesn’t confer second-rate ontological status. If I may quote physicist Frank Wiczek,
“It is reasonable to suppose that the goal of a future-mind will be to optimize a mathematical measure of its well-being or achievement, based on its internal state. (Economists speak of ‘maximizing utility″, normal people of ‘finding happiness’.) The future-mind could discover, by its powerful introspective abilities or through experience, its best possible state the Magic Moment—or several excellent ones. It could build up a library of favourite states. That would be like a library of
favourite movies, but more vivid, since to recreate magic moments accurately would be equivalent to living through them. Since the joys of discovery, triumph and fulfillment require novelty, to re-live a magic moment properly, the future-mind would have to suppress memory of that moment’s previous realizations.
A future-mind focused upon magic moments is well matched to the limitations of reversible computers, which expend no energy. Reversible computers cannot store new memories, and they are as likely to run backwards as forwards. Those limitations bar adaptation and evolution, but invite eternal cycling through magic moments. Since energy becomes a scarce quantity in an expanding universe, that scenario might well describe the long-term future of mind in the cosmos.”
(Frank Wiczek)
[Big troubles, imagined and real; published in Global Catastrophic Risks, eds Nick Bostrom, Milan M. Cirkovic, OUP, 2008)
So is convergence on the secular equivalent of Heaven inevitable? I guess not. One can think of multiple possible defeaters. For instance, if the IJ Good / SIAI conception of the Intelligence Explosion (as I understand it) is correct, then the orthogonality thesis is plausible for a hypothetical AGI. On this story, might e.g. an innocent classical utilitarian build AGI-in-a-box that goes FOOM and launches a utilitronium shockwave? (etc)
But in our current state of ignorance, I’m just not yet convinced we know enough to rule out the convergence hypothesis.
David, what are those multiple possible defeaters for convergence? As I see it, the practical defeaters that exist still don’t affect the convergence thesis, they just are possible practical impediments, from unintelligent agents, to the realization of the goals of convergence.
Just how diverse is human motivation? Should we discount even sophisticated versions of psychological hedonism? Undoubtedly, the “pleasure principle” is simplistic as it stands. But one good reason not to try heroin, for example, is precisely that the reward architecture of our opioid pathways is so similar. Previously diverse life-projects of first-time heroin users are at risk of converging on a common outcome. So more broadly, let’s consider the class of life-supporting Hubble volumes where sentient biological robots acquire the capacity to rewrite their genetic source code and gain mastery of their own reward circuitry. May we predict orthogonality or convergence? Certainly, there are strong arguments why such intelligences won’t all become the functional equivalent of heroin addicts or wireheads or Nozick Experience Machine VR-heads (etc). One such argument is the nature of selection pressure. But _if_some version of the pleasure principle is correct, then isn’t some version of the convergence conjecture at least feasible, i.e. they’ll recalibrate the set-point of their hedonic treadmill and enjoy gradients of (super)intelligent (super)happiness? One needn’t be a meta-ethical value-realist to acknowledge that subjects of experience universally find bliss is empirically more valuable than agony or despair. The present inability of natural science to explain first-person experiences doesn’t confer second-rate ontological status. If I may quote physicist Frank Wiczek,
“It is reasonable to suppose that the goal of a future-mind will be to optimize a mathematical measure of its well-being or achievement, based on its internal state. (Economists speak of ‘maximizing utility″, normal people of ‘finding happiness’.) The future-mind could discover, by its powerful introspective abilities or through experience, its best possible state the Magic Moment—or several excellent ones. It could build up a library of favourite states. That would be like a library of favourite movies, but more vivid, since to recreate magic moments accurately would be equivalent to living through them. Since the joys of discovery, triumph and fulfillment require novelty, to re-live a magic moment properly, the future-mind would have to suppress memory of that moment’s previous realizations.
A future-mind focused upon magic moments is well matched to the limitations of reversible computers, which expend no energy. Reversible computers cannot store new memories, and they are as likely to run backwards as forwards. Those limitations bar adaptation and evolution, but invite eternal cycling through magic moments. Since energy becomes a scarce quantity in an expanding universe, that scenario might well describe the long-term future of mind in the cosmos.” (Frank Wiczek) [Big troubles, imagined and real; published in Global Catastrophic Risks, eds Nick Bostrom, Milan M. Cirkovic, OUP, 2008)
So is convergence on the secular equivalent of Heaven inevitable? I guess not. One can think of multiple possible defeaters. For instance, if the IJ Good / SIAI conception of the Intelligence Explosion (as I understand it) is correct, then the orthogonality thesis is plausible for a hypothetical AGI. On this story, might e.g. an innocent classical utilitarian build AGI-in-a-box that goes FOOM and launches a utilitronium shockwave? (etc) But in our current state of ignorance, I’m just not yet convinced we know enough to rule out the convergence hypothesis.
David, what are those multiple possible defeaters for convergence? As I see it, the practical defeaters that exist still don’t affect the convergence thesis, they just are possible practical impediments, from unintelligent agents, to the realization of the goals of convergence.