My main issue is that some of the flaws are far from being “hard but soluble by a Novel Insight And Long Self-Correction”: insoluble as stated, too easy or outright erroneous.
Mankind being good at long-horizon strategy would require mankind to have a far greater skill at noticing the emergence of x-risk from novel places, like AI or the LHC. The problem is that x-risks are so rare that a very detailed description of all real risks with which IIRC 100 billion humans who ever existed have come up is well fittable in ~300 pages in Russian.
The case against early MIRI’s attempt to race towards Friendly AI would have to dismiss The Counterfactual Quiet AGI Timeline[1] and real-world worse actors like xAI.
I find it unlikely that careful strategising is disvalued in making high-level decisions where it actually matters. What could be erroneously valued is strategising to keep one’s position, like failing to inform a higher-level official about a problem.
I wonder what is even doable about zero-sum values and how they would affect a world where a distribution[2] of power is locked in, as is likely to happen[3] in an ASI-ruled world.
Being easy to manipulate could be similar to lacking the ability or the data necessary to come up with a better theory. For example, Newton’s mechanics was an excellent approximation of reality and was replaced with the paradigm involving spacetime after all variants of the Michelson–Morley experiment failed to notice Earth’s motion related to the aether.
My closest candidate solution to these problems is not some Philosophical Insight From The Future, but broadly educating people.
My main issue is that some of the flaws are far from being “hard but soluble by a Novel Insight And Long Self-Correction”: insoluble as stated, too easy or outright erroneous.
Mankind being good at long-horizon strategy would require mankind to have a far greater skill at noticing the emergence of x-risk from novel places, like AI or the LHC. The problem is that x-risks are so rare that a very detailed description of all real risks with which IIRC 100 billion humans who ever existed have come up is well fittable in ~300 pages in Russian.
The case against early MIRI’s attempt to race towards Friendly AI would have to dismiss The Counterfactual Quiet AGI Timeline[1] and real-world worse actors like xAI.
I find it unlikely that careful strategising is disvalued in making high-level decisions where it actually matters. What could be erroneously valued is strategising to keep one’s position, like failing to inform a higher-level official about a problem.
I wonder what is even doable about zero-sum values and how they would affect a world where a distribution[2] of power is locked in, as is likely to happen[3] in an ASI-ruled world.
Being easy to manipulate could be similar to lacking the ability or the data necessary to come up with a better theory. For example, Newton’s mechanics was an excellent approximation of reality and was replaced with the paradigm involving spacetime after all variants of the Michelson–Morley experiment failed to notice Earth’s motion related to the aether.
My closest candidate solution to these problems is not some Philosophical Insight From The Future, but broadly educating people.
My main objection is that the existence of safety-pilled AI labs might have had a higher bus factor than ablating Yudkowsky.
Including a uniform-like one, as happens in the Epilogue of AI 2040.
However, resources of Earth or the Solar System can also be quickly reallocated between humans.