My argument: The reason the orthogonal hypothesis is false is that 100% (full) general intelligence is not possible without 100% (perfect) moral motivations. That is to say, any AI that is not completely moral is limited to some degree.
Expert Counter-argument: But the Marcus Hutter AIXI model shows that it’s possible to have optimal (fully general intelligence) decision makers with any arbitrary goals.
My reply: The AIXI model of Hutter is not a true general intelligence, because the initial goals are fixed (i.e it cannot reflect upon and modify its own goals). Further, the AIXI model is not pratical in real-time due to requirements for infinite (or impractically high) computational resources. You need correct priors for accurate real-time reasoning.
Expert Counter-argument: What have priors got to do with morality? Further, the problem of priors has been solved. The Kolmogorov complexity prior is a precise formalization of Occam’s razor.
My reply: The Kolmogorov complexity prior is uncomputable
Expert Counter-argument: There are perfectly workable approximations! For instance, the Schmidhuber speed prior and methods such as Monte Carlo methods can be deployed.
My reply: There is no proof that such approximations can be scaled up from their success in limited domains to full general intelligence.
Expert Counter-argument: The problem of priors isn’t relevant. The initial priors ‘wash out’ with enough evidence. The convergence theorems show this.
My reply: In fact, there is no convergence for different initial Bayesian models. Further, the computational explosion with the increase in problem complexity results in failure of Occam approximations to scale.
Expert Counter-argument: Even if this is true (which is highly debatable), what have priors got to do with morality any way?
My reply: Occam’s razor is the smoking gun which establishes a link between priors and value judgements. Fully general intelligence requires a formal notion of Occam razor. But Occam priors are equivalent to an aesthetic notion of what constitutes ‘simplicity’. Schmidhuber argued that the notion of ‘beauty’ is tied into the ability to compress information. But if this is true, then intelligence cannot be independent of aesthetic/value judgements.
Expert Counter-argument: This sounds like a weak idea to me. I don’t see the link between aesthetic judgements and Occam priors. As to Schmidhuber’s attempt to tie a notion of ‘beauty’ to data compression, didn’t Rayhawk critique the idea on the grounds that Schmidhuber’s notion of beauty isn’t actually required for an optimal decision making agent? And even if the general idea was correct, why should this notion of ‘beauty’ correspond to what humans value any way?
My reply: Occam priors could be approximated by deploying categorization and analogical inference...grouping things into categories to compress information is equivalent to the introduction of inductive biases, which already looks human-like. Kant argued that intelligent minds require a set of universal a-priori categories and ontological primitives look like the modern version of Kant’s categories.
Expert Counter-argument: This is just vague hand-waving! Exactly how would your analogical inference/categorization scheme work?
My reply: A small set of initial ontological primitives (defined in terms of prototypes rather than sufficient conditions) would be built-into the seed-AI. These would serve as reference classes enabling calculation of the semantic distances between features of concepts in feature-space. This would define the basic notion of ‘similarity’ needed for analogical inference, which would involve cross-domain mappings between ontologies and allow fully general (cross-domain) reasoning.
Expert Counter-argument: OK, although I’m not sure you know what you’re talking about, and I still don’t see the connection to morality
My reply: Here’s the connection to morality: the ontology would be used to form representations of our values in the form of decision-making narratives. To compress the representations of values into these coherent narratives, aesthetic and ethical value judgments would be needed. This whole system would be defined in terms of information theory.
Expert Counter-argument: Even if there was a ‘natural’ notion of morality that emerges from all this, which sounds just like wild speculation to me, why should I (or an AI) want to follow it anyway?
My reply: Deviations from correct compression techniques degrade representations of decision-making narratives, thus degrading general reasoning abilities. Since correct compression techniques are tied to aesthetic/value judgements, any deviation from perfect morality would reduce the effective intelligence of the AI. You cannot reason that you should deviate from perfect morality without contradicting yourself and degrading your own cognitive processes! So knowledge of universal morality would automatically be self-motivating.
Expert Counter-argument: Sounds unlikely to me, besides, as Bostrom pointed out, even if there was a universal morality, the knowledge of which is self-motivating, the AI need not be built in the way you described, in which case it wouldn’t care about correct reasoning as you have defined it. Pick a random mind from mind-space, it doesn’t care about humans.
My reply: Most programs picked randomly from mind-space don’t work. UAIs would not work properly. Most of the AIs picked randomly from mind-space just won’t work. I’m not claiming intelligence implies morality, I’m claiming the converse of this (i.e ‘morality implies intelligence’). I’m not saying UAIs can’t exist, I’m saying they won’t be true general intelligences. I’m saying they won’t be able to self-improve to the super-intelligent level.
My argument: The reason the orthogonal hypothesis is false is that 100% (full) general intelligence is not possible without 100% (perfect) moral motivations. That is to say, any AI that is not completely moral is limited to some degree.
Expert: define your terms, justifying why you apply the label ‘moral’ to the particular motivations you have in mind, then present evidence for your thesis.
I think this argument-complex is stronger than the AI risk folk admit (and I know how to strengthen your argument at various points). A plausible off-beat counter is that humans have been getting less moral (and their aesthetic tastes have gotten worse) over time despite their getting closer to building AI—what you see in history is might consistently rewriting the rules of morality to make right, and predicting this continued trend, while accurate in some descriptive sense (e.g. if you take people’s values at face value as defining morality), it doesn’t seem like sound moral philosophy. In this sense a singularity would be like a glorious communist revolution—seemingly inevitable, seemingly the logical endpoint of morality, yet in fact incredibly destructive both physically and culturally. The problem with AI is that, even if in the limit intelligence and morality (might and right) are the same thing, it seems like an AI would be able to set up the equivalent of a communist dictatorship and hold on to it for as long as it takes a black hole to evaporate. And even if the new communist dictatorship were better than what came before it, it still seems like we have a shot to ensure that AI will jump straight to 100% intelligence and 100% morality without getting caught up somewhere along the way. But of course, even the communist dictatorship scenario isn’t really compatible with the orthogonality thesis...
My argument: The reason the orthogonal hypothesis is false is that 100% (full) general intelligence is not possible without 100% (perfect) moral motivations. That is to say, any AI that is not completely moral is limited to some degree.
Expert Counter-argument: But the Marcus Hutter AIXI model shows that it’s possible to have optimal (fully general intelligence) decision makers with any arbitrary goals.
My reply: The AIXI model of Hutter is not a true general intelligence, because the initial goals are fixed (i.e it cannot reflect upon and modify its own goals). Further, the AIXI model is not pratical in real-time due to requirements for infinite (or impractically high) computational resources. You need correct priors for accurate real-time reasoning.
Expert Counter-argument: What have priors got to do with morality? Further, the problem of priors has been solved. The Kolmogorov complexity prior is a precise formalization of Occam’s razor.
My reply: The Kolmogorov complexity prior is uncomputable
Expert Counter-argument: There are perfectly workable approximations! For instance, the Schmidhuber speed prior and methods such as Monte Carlo methods can be deployed.
My reply: There is no proof that such approximations can be scaled up from their success in limited domains to full general intelligence.
Expert Counter-argument: The problem of priors isn’t relevant. The initial priors ‘wash out’ with enough evidence. The convergence theorems show this.
My reply: In fact, there is no convergence for different initial Bayesian models. Further, the computational explosion with the increase in problem complexity results in failure of Occam approximations to scale.
Expert Counter-argument: Even if this is true (which is highly debatable), what have priors got to do with morality any way?
My reply: Occam’s razor is the smoking gun which establishes a link between priors and value judgements. Fully general intelligence requires a formal notion of Occam razor. But Occam priors are equivalent to an aesthetic notion of what constitutes ‘simplicity’. Schmidhuber argued that the notion of ‘beauty’ is tied into the ability to compress information. But if this is true, then intelligence cannot be independent of aesthetic/value judgements.
Expert Counter-argument: This sounds like a weak idea to me. I don’t see the link between aesthetic judgements and Occam priors. As to Schmidhuber’s attempt to tie a notion of ‘beauty’ to data compression, didn’t Rayhawk critique the idea on the grounds that Schmidhuber’s notion of beauty isn’t actually required for an optimal decision making agent? And even if the general idea was correct, why should this notion of ‘beauty’ correspond to what humans value any way?
My reply: Occam priors could be approximated by deploying categorization and analogical inference...grouping things into categories to compress information is equivalent to the introduction of inductive biases, which already looks human-like. Kant argued that intelligent minds require a set of universal a-priori categories and ontological primitives look like the modern version of Kant’s categories.
Expert Counter-argument: This is just vague hand-waving! Exactly how would your analogical inference/categorization scheme work?
My reply: A small set of initial ontological primitives (defined in terms of prototypes rather than sufficient conditions) would be built-into the seed-AI. These would serve as reference classes enabling calculation of the semantic distances between features of concepts in feature-space. This would define the basic notion of ‘similarity’ needed for analogical inference, which would involve cross-domain mappings between ontologies and allow fully general (cross-domain) reasoning.
Expert Counter-argument: OK, although I’m not sure you know what you’re talking about, and I still don’t see the connection to morality
My reply: Here’s the connection to morality: the ontology would be used to form representations of our values in the form of decision-making narratives. To compress the representations of values into these coherent narratives, aesthetic and ethical value judgments would be needed. This whole system would be defined in terms of information theory.
Expert Counter-argument: Even if there was a ‘natural’ notion of morality that emerges from all this, which sounds just like wild speculation to me, why should I (or an AI) want to follow it anyway?
My reply: Deviations from correct compression techniques degrade representations of decision-making narratives, thus degrading general reasoning abilities. Since correct compression techniques are tied to aesthetic/value judgements, any deviation from perfect morality would reduce the effective intelligence of the AI. You cannot reason that you should deviate from perfect morality without contradicting yourself and degrading your own cognitive processes! So knowledge of universal morality would automatically be self-motivating.
Expert Counter-argument: Sounds unlikely to me, besides, as Bostrom pointed out, even if there was a universal morality, the knowledge of which is self-motivating, the AI need not be built in the way you described, in which case it wouldn’t care about correct reasoning as you have defined it. Pick a random mind from mind-space, it doesn’t care about humans.
My reply: Most programs picked randomly from mind-space don’t work. UAIs would not work properly. Most of the AIs picked randomly from mind-space just won’t work. I’m not claiming intelligence implies morality, I’m claiming the converse of this (i.e ‘morality implies intelligence’). I’m not saying UAIs can’t exist, I’m saying they won’t be true general intelligences. I’m saying they won’t be able to self-improve to the super-intelligent level.
Expert: Let me think this further for while.....
Expert: define your terms, justifying why you apply the label ‘moral’ to the particular motivations you have in mind, then present evidence for your thesis.
I think this argument-complex is stronger than the AI risk folk admit (and I know how to strengthen your argument at various points). A plausible off-beat counter is that humans have been getting less moral (and their aesthetic tastes have gotten worse) over time despite their getting closer to building AI—what you see in history is might consistently rewriting the rules of morality to make right, and predicting this continued trend, while accurate in some descriptive sense (e.g. if you take people’s values at face value as defining morality), it doesn’t seem like sound moral philosophy. In this sense a singularity would be like a glorious communist revolution—seemingly inevitable, seemingly the logical endpoint of morality, yet in fact incredibly destructive both physically and culturally. The problem with AI is that, even if in the limit intelligence and morality (might and right) are the same thing, it seems like an AI would be able to set up the equivalent of a communist dictatorship and hold on to it for as long as it takes a black hole to evaporate. And even if the new communist dictatorship were better than what came before it, it still seems like we have a shot to ensure that AI will jump straight to 100% intelligence and 100% morality without getting caught up somewhere along the way. But of course, even the communist dictatorship scenario isn’t really compatible with the orthogonality thesis...