I expect we’ll handle this (“expect” as in probability >50%, not probability 90%) primarily because we all want the same outcome, and we don’t yet see any obstacles clearly enough to project confidently that the obstacles are too hard to overcome.
When you say “we all want the same outcome”, do you mean we all want consequentialist systems, with our values and not subject to value drift, to be built before too much evolution has taken place? But many AGI researchers seem to prefer working on “heuristic soup” type designs (which makes sense if those AGI researchers are not themselves “properly consequentialist” and don’t care strongly about long range outcomes).
I don’t quite understand your point (3), which seems like it was addressed.
What I mean is that the kind of value-stable consequentialist that humans can build in the relevant time frame may be too inefficient to survive under competitive pressure from other cognitive/organizational architectures that will exist (even if it can survive as a singleton).
When you say “we all want the same outcome”, do you mean we all want consequentialist systems, with our values and not subject to value drift, to be built before too much evolution has taken place? But many AGI researchers seem to prefer working on “heuristic soup” type designs (which makes sense if those AGI researchers are not themselves “properly consequentialist” and don’t care strongly about long range outcomes).
What I mean is that the kind of value-stable consequentialist that humans can build in the relevant time frame may be too inefficient to survive under competitive pressure from other cognitive/organizational architectures that will exist (even if it can survive as a singleton).