It feels like there is a principle of mathematics which rules out algorithms that are “too good to be true”: a single “magic wand” to solve all problems. In a similar way, AGI is a “magic wand”: it solves “all” problems because you can simply delegate them to the AGI.
It seems to me that either human-level GI is impossible or it is not. But, we humans constitute an existence proof that it is not impossible. Therefore, Gödel does not apply; clearly human-level GI is not “too good to be true”. And, Gödel’s incompleteness theorems really have nothing to do with how hard it might be to come up with a solution to a computable problem.
I also have doubts regarding this:
This way the AGI design problem becomes an optimization problem: find a program with an intelligence metric as high as possible. The NP-connection now suggests the following conjecture: the AGI optimization program is of exponential complexity in program length. Of course we don’t necessarily need the best program of a given length...
The bold portion of the above section is key—there are various NP-hard optimization problems for which good approximate solutions can be found in polynomial time. Since human or super-human AGI does not require a maximally intelligent program, even if you could show that the AGI optimization problem is NP-hard that would say nothing about the difficulty of finding a human or super-human level AGI.
It seems to me that either human-level GI is impossible or it is not. But, we humans constitute an existence proof that it is not impossible.
Of course. This is why I’m not saying human-level GI is impossible. I’m saying that designing it from scratch is impossible.
And, Gödel’s incompleteness theorems really have nothing to do with how hard it might be to come up with a solution to a computable problem.
My argument is not in any sense a proof from Goedel incompleteness. Goedel incompleteness is just a suggestive analogy.
...there are various NP-hard optimization problems for which good approximate solutions can be found in polynomial time. Since human or super-human AGI does not require a maximally intelligent program, even if you could show that the AGI optimization problem is NP-hard that would say nothing about the difficulty of finding a human or super-human level AGI.
I mostly agree: except for the “nothing” part. I think it would definitely cause me to update towards “designing AGI is infeasible”. I hinted at a possible relation between intelligence and LK-complexity. If such a relation can be proven and human intelligence can be estimated we would have a more definite answer.
I have doubts regarding this:
It seems to me that either human-level GI is impossible or it is not. But, we humans constitute an existence proof that it is not impossible. Therefore, Gödel does not apply; clearly human-level GI is not “too good to be true”. And, Gödel’s incompleteness theorems really have nothing to do with how hard it might be to come up with a solution to a computable problem.
I also have doubts regarding this:
The bold portion of the above section is key—there are various NP-hard optimization problems for which good approximate solutions can be found in polynomial time. Since human or super-human AGI does not require a maximally intelligent program, even if you could show that the AGI optimization problem is NP-hard that would say nothing about the difficulty of finding a human or super-human level AGI.
Of course. This is why I’m not saying human-level GI is impossible. I’m saying that designing it from scratch is impossible.
My argument is not in any sense a proof from Goedel incompleteness. Goedel incompleteness is just a suggestive analogy.
I mostly agree: except for the “nothing” part. I think it would definitely cause me to update towards “designing AGI is infeasible”. I hinted at a possible relation between intelligence and LK-complexity. If such a relation can be proven and human intelligence can be estimated we would have a more definite answer.