Your definition of general intelligence would include SGD on large neural networks
I don’t count it in, actually. In my view, the boundaries of the algorithm here aren’t “SGD + NN”, but “the training loop” as a whole, which includes the dataset and the loss/reward function. A general intelligence implemented via SGD, then, would correspond to an on-line training loop that can autonomously (without assistance from another generally-intelligent entity, like a human overseer) learn to navigate any environment.
I don’t think any extant training-loop setup fits this definition. They all need externally-defined policy gradients. If the distribution on which they’re trained changes significantly, the policy gradient (loss/reward function) would need to be changed to suit — and that’d need to be done by something external to the training loop, which already understands the new environment (e. g., the human overseer) and knows how the policy gradient needs to be adapted to keep the system on-target.
(LLMs trained via SSL are a degenerate case: in their case the prediction gradient = the policy gradient. They also can’t autonomously generalize to generating new classes of text without first being shown a carefully curated dataset of such texts. They’re not an exception.)
I don’t count it in, actually. In my view, the boundaries of the algorithm here aren’t “SGD + NN”, but “the training loop” as a whole, which includes the dataset and the loss/reward function. A general intelligence implemented via SGD, then, would correspond to an on-line training loop that can autonomously (without assistance from another generally-intelligent entity, like a human overseer) learn to navigate any environment.
I don’t think any extant training-loop setup fits this definition. They all need externally-defined policy gradients. If the distribution on which they’re trained changes significantly, the policy gradient (loss/reward function) would need to be changed to suit — and that’d need to be done by something external to the training loop, which already understands the new environment (e. g., the human overseer) and knows how the policy gradient needs to be adapted to keep the system on-target.
(LLMs trained via SSL are a degenerate case: in their case the prediction gradient = the policy gradient. They also can’t autonomously generalize to generating new classes of text without first being shown a carefully curated dataset of such texts. They’re not an exception.)
I’m skeptical that locating the hyperparameters you mention is an AGI-complete task.