One of the unfortunate limitations of modern complexity theory is that a set of problems that look isomorphic sometimes have very different complexity properties. Another awkwardness is that worst-case complexity isn’t a reliable guide to practical difficulty. “This sorta feels like a coloring problem” isn’t enough to show it’s intractable on the sort of instances we care about.
Separately, it’s not actually clear to me whether complexity is good or bad news. If you think that predicting human desires and motivations is infeasible computationally, you should probably worry less about super intelligent AI, since that complexity barrier will prevent the AI from being radically effective at manipulating us.
It would seem to require an unusually malicious universe for a superhuman AI to be feasible, for that AI to be able to manipulate us efficiently, but for it to be infeasible for us to write a program to specify constraints that we would be happy with in retrospect.
, but for it to be infeasible for us to write a program to specify constraints that we would be happy with in retrospect.
The point I believe that 27chaos is trying to argue isn’t that writing down the constraints would necessarily be hard (although it very likely is) but that trying to satisfy them may be tough.
One of the unfortunate limitations of modern complexity theory is that a set of problems that look isomorphic sometimes have very different complexity properties. Another awkwardness is that worst-case complexity isn’t a reliable guide to practical difficulty. “This sorta feels like a coloring problem” isn’t enough to show it’s intractable on the sort of instances we care about.
Separately, it’s not actually clear to me whether complexity is good or bad news. If you think that predicting human desires and motivations is infeasible computationally, you should probably worry less about super intelligent AI, since that complexity barrier will prevent the AI from being radically effective at manipulating us.
It would seem to require an unusually malicious universe for a superhuman AI to be feasible, for that AI to be able to manipulate us efficiently, but for it to be infeasible for us to write a program to specify constraints that we would be happy with in retrospect.
The point I believe that 27chaos is trying to argue isn’t that writing down the constraints would necessarily be hard (although it very likely is) but that trying to satisfy them may be tough.