I think AI-completeness is a quite seductive notion. Borrowing the concept of reduction from complexity/computability theory makes it sound technical, but unlike those fields I haven’t seen anyone actually describing eg how to use an AI with perfect language understanding to produce another one that proved theorems or philosophized.
Spontaneously it feels like everyone here should in principle be able to sketch the outlines of such a program (at least in the case of a base-AI that has perfect language comprehension that we want to reduce to), probably by some version of trying to teach the AI as we teach a child in natural language. I suspect that the details of some of these reductions might still be useful, especially the parts that don’t quite seem to work. For while I don’t think that we’ll see perfect machine translation before AGI, I’m much less convinced that there is a reduction from AGI to perfect translation AI. This illustrates what I suspect might be an interesting differences between two problem classes that we might both want to call AI-complete: the problems human programmers will likely not be able to solve before we create superintelligence, and the problems whose solutions we could (somewhat) easily re-purpose to solve the general problem of human-level AI. These classes look the same as in we shouldn’t expect to see problems from any of them solved without an imminent singularity, but differ in that the problems in the latter class could prove to be motivating examples and test-cases for AI work aimed at producing superintelligence.
I guess the core of what I’m trying to say is that arguments about AI-completeness has so far sounded like: “This problem is very very hard, we don’t really know how to solve it. AI in general is also very very hard, and we don’t know how to solve it. So they should be the same.” Heuristically there’s nothing wrong with this, except we should keep in mind that we could be very mistaken about what is actually hard. I’m just missing the part that goes: “This is very very hard. But if we knew it this other thing would be really easy.”
This sounds like an interesting project. I’ve studied quite some category theory myself, though mostly from the “oh pretty!” point of view, and dipped my feet into algebraic geometry because it sounded cool. I think that reading algebraic geometry with the sight set on cryptography would be more giving than the general swimming around in its sea that I’ve done before. So if you want a reading buddy, do tell. A fair warning though: I’m quite time limited these coming months, so will not be able to keep a particularly rapid pace.