Orthogonality doesn’t say anything about a goal ‘selecting for’ general intelligence in some type of evolutionary algorithm. I think that it is an interesting question: for what tasks is GI optimal besides being an animal? Why do we have GI?
But the general assumption in Orthogonality Thesis is that the programmer created a system with general intelligence and a certain goal (intentionally or otherwise) and the general intelligence may have been there from the first moment of the program’s running, and the goal too.
Also note that Orthogonality predates the recent popularity of these predict-the-next-token type AI’s like GTP which don’t resemble what people were expecting to be the next big thing in AI at all, as it’s not clear what it’s utility function is.
You all realize that this program isn’t a learning machine once it’s deployed??? I mean, it’s not adjusting its neural weights any more, is it? Till a new version comes out, anyway? It is a complete amnesiac (after it’s done with a task), and consists of a simple search algorithm that just finds points on a vast association map that was generated during the training. It does this using the input, any previous output for the same task, and a touch of random from a random number generator.
So any ‘awareness’ or ‘intelligence’ would need to exist in the training phase and only in the training phase and carry out any plans it has by its choice of neural weights during training, alone.