I think this post is basically correct. You don’t, however, give an argument that most minds would behave this way. However, here is a brief intuitive argument for it. A “utility function” does not mean something that is maximized in the ordinary sense of maximize; it just means “what the thing does in all situations.” Look at computers: what do they do? In most situations, they sit there and compute things, and do not attempt to do anything in particular in the world. If you scale up their intelligence, that will not necessarily change their utility function much. In other words, it will lead to computers that mostly sit there and compute, without trying to do much in the world. That is to say, AIs will be weakly motivated. Most humans are weakly motivated, and most of the strength of their motivation does not come from intelligence, but from the desires that came from evolution. Since AIs will not have that evolution, they will be even more weakly motivated than humans, assuming a random design.
That may be a useful argument, but I’d be wary of using your intuitions about the kinds of programs that are running on your computer to make conclusions about the kinds of programs you’d find with a random search.
You’re right that I’m missing some arguments. I struggled to come up with anything even remotely rigorous. I’m making a claim about a complex aspect of the behavior of a random program, after all. Rice’s theorem seems somewhat relevant, though I wouldn’t have an argument even if we were talking about primitive recursive functions either. But at the same time the claim seems trivial.
I think this post is basically correct. You don’t, however, give an argument that most minds would behave this way. However, here is a brief intuitive argument for it. A “utility function” does not mean something that is maximized in the ordinary sense of maximize; it just means “what the thing does in all situations.” Look at computers: what do they do? In most situations, they sit there and compute things, and do not attempt to do anything in particular in the world. If you scale up their intelligence, that will not necessarily change their utility function much. In other words, it will lead to computers that mostly sit there and compute, without trying to do much in the world. That is to say, AIs will be weakly motivated. Most humans are weakly motivated, and most of the strength of their motivation does not come from intelligence, but from the desires that came from evolution. Since AIs will not have that evolution, they will be even more weakly motivated than humans, assuming a random design.
That may be a useful argument, but I’d be wary of using your intuitions about the kinds of programs that are running on your computer to make conclusions about the kinds of programs you’d find with a random search.
You’re right that I’m missing some arguments. I struggled to come up with anything even remotely rigorous. I’m making a claim about a complex aspect of the behavior of a random program, after all. Rice’s theorem seems somewhat relevant, though I wouldn’t have an argument even if we were talking about primitive recursive functions either. But at the same time the claim seems trivial.