I think we disagree on what a specification is. By specification I mean a verifier: if you had something fitting the specification, you could tell if it did. For example we have a specification for “proof that P != NP” because we have a system in which that proof could be written and verified. Similarly, this system contains a specification for general optimization. You seem to be interpreting specification as knowing how to make the thing.

If you give this optimizer the MU Puzzle (aka 2^n mod 3 = 0) it will never figure it out, even though most children will come to the right answer in minutes.

If you define the problem as “find n such that 2^n mod 3 = 0” then everyone will fail the problem. And I don’t see why the optimizer couldn’t have some code that monitors its own behavior. Sure it’s difficult to write, but the point of this system is to go from a seed AI to a superhuman AI safely. And such a function (“consciousness”) would help it solve many of the sample optimization problems without significantly increasing complexity.

I think we disagree on what a specification is. By specification I mean a verifier: if you had something fitting the specification, you could tell if it did. For example we have a specification for “proof that P != NP” because we have a system in which that proof could be written and verified. Similarly, this system contains a specification for general optimization. You seem to be interpreting specification as knowing how to make the thing.

If you define the problem as “find n such that 2^n mod 3 = 0” then everyone will fail the problem. And I don’t see why the optimizer couldn’t have some code that monitors its own behavior. Sure it’s difficult to write, but the point of this system is to go from a seed AI to a superhuman AI safely. And such a function (“consciousness”) would help it solve many of the sample optimization problems without significantly increasing complexity.