The following is a counterargument to some of the claims of the OP. I’m not convinced that it’s a correct counterargument, but it’s sufficient to make me doubt some of the central claims of the OP.
I think I understand the reasoning that leads to claims like
Unlike a human toddler, AIXI can never entertain the possibility of its own death. However much it learns, it will never recognize its mortality.
and
they won’t take seriously the idea that their cognition can be modified
But from my reductionism+materialism+computationalism perspective, it seems obvious that a computer program that gets this sort of thing right must exist (because humans get these things right and humans exist, or even just because “naturalized induction” would implement such a program). So the argument must be that the Solomonoff prior gives much higher weights to equally good predictors that get these things wrong.
But sequence prediction isn’t magic—we can expect that the shortest programs that survive a reasonable training period are the ones with (possibly highly compressed) internal representations that carve reality at its joints. If the training period can pack in enough information, then AIXI can learn the same sorts of things we think we know about being embedded in the world and how the material that forms our nervous systems and brains carries out the computations that we feel from the inside. In other words, if I can learn to predict that my perceptual stream will end if I drop an anvil on my head even though I’ve never tried it, why can’t an AIXI agent?
What am I missing?
(I regard the preference solipsism point as obviously correct—you don’t want to give an autonomous AI motivations until well after it’s able to recognize map/territory distinctions.)
Imagine you’re an environmental hypothesis program within AIXI. You recognize that AIXI is manipulating an anvil. Your only way of communicating with AIXI is by making predictions. On the one hand, you want to make accurate predictions in order to maintain your credibility within AIXI. On the other hand, sometimes you want to burn your credibility by making a false prediction of very large or very small utility in order to influence AIXI’s decisions. And unfortunately for you, the fact that you are materialist/computationalist/etc. means you and programs like you make up a small amount of measure in AIXI’s beliefs; your colleagues work against you.
you and programs like you make up a small amount of measure in AIXI’s beliefs
I understand that this is the claim, but my intuition is that, supposing that AIXI has observed a long enough sequence to have as good an idea as I do of how the world is put together, I and programs like me (like “naturalized induction”) are the shortest of the survivors, and hence dominate AIXI’s predictions. Basically, I’m positing that after a certain point, AIXI will notice that it is embodied and doesn’t have a soul, for essentially the same reason that I have noticed those things: they are implications of the simplest explanations consistent with the observations I have made so far.
Well, I guess it could, but that isn’t the claim being put forth in the OP.
(Unlike some around these parts, I see a clear distinction between an agent’s posterior distribution and the agent’s posterior-utility-maximizing part. From the outside, expected-utility-maximizing agents form an equivalence class such that all agents with the same are equivalent, and we need only consider the quotient space of agents; from the inside, the epistemic and value-laden parts of an agent can thought of separately.)
if I can learn to predict that my perceptual stream will end if I drop an anvil on my head even though I’ve never tried it, why can’t an AIXI agent?
Predicting anvils isn’t enough, you also need the urge to avoid them. You got that urge from evolution, which got it by putting many creatures in dangerous situations and keeping those that survived. I’m not sure if AIXI can learn that without having any experience with death.
The following is a counterargument to some of the claims of the OP. I’m not convinced that it’s a correct counterargument, but it’s sufficient to make me doubt some of the central claims of the OP.
I think I understand the reasoning that leads to claims like
and
But from my reductionism+materialism+computationalism perspective, it seems obvious that a computer program that gets this sort of thing right must exist (because humans get these things right and humans exist, or even just because “naturalized induction” would implement such a program). So the argument must be that the Solomonoff prior gives much higher weights to equally good predictors that get these things wrong.
But sequence prediction isn’t magic—we can expect that the shortest programs that survive a reasonable training period are the ones with (possibly highly compressed) internal representations that carve reality at its joints. If the training period can pack in enough information, then AIXI can learn the same sorts of things we think we know about being embedded in the world and how the material that forms our nervous systems and brains carries out the computations that we feel from the inside. In other words, if I can learn to predict that my perceptual stream will end if I drop an anvil on my head even though I’ve never tried it, why can’t an AIXI agent?
What am I missing?
(I regard the preference solipsism point as obviously correct—you don’t want to give an autonomous AI motivations until well after it’s able to recognize map/territory distinctions.)
Imagine you’re an environmental hypothesis program within AIXI. You recognize that AIXI is manipulating an anvil. Your only way of communicating with AIXI is by making predictions. On the one hand, you want to make accurate predictions in order to maintain your credibility within AIXI. On the other hand, sometimes you want to burn your credibility by making a false prediction of very large or very small utility in order to influence AIXI’s decisions. And unfortunately for you, the fact that you are materialist/computationalist/etc. means you and programs like you make up a small amount of measure in AIXI’s beliefs; your colleagues work against you.
I understand that this is the claim, but my intuition is that, supposing that AIXI has observed a long enough sequence to have as good an idea as I do of how the world is put together, I and programs like me (like “naturalized induction”) are the shortest of the survivors, and hence dominate AIXI’s predictions. Basically, I’m positing that after a certain point, AIXI will notice that it is embodied and doesn’t have a soul, for essentially the same reason that I have noticed those things: they are implications of the simplest explanations consistent with the observations I have made so far.
Why couldn’t it also be a program that has predictive powers similar to yours, but doesn’t care about avoiding death?
Well, I guess it could, but that isn’t the claim being put forth in the OP.
(Unlike some around these parts, I see a clear distinction between an agent’s posterior distribution and the agent’s posterior-utility-maximizing part. From the outside, expected-utility-maximizing agents form an equivalence class such that all agents with the same are equivalent, and we need only consider the quotient space of agents; from the inside, the epistemic and value-laden parts of an agent can thought of separately.)
Oh, I see what you’re saying now.
Predicting anvils isn’t enough, you also need the urge to avoid them. You got that urge from evolution, which got it by putting many creatures in dangerous situations and keeping those that survived. I’m not sure if AIXI can learn that without having any experience with death.