Imagine you had to solve a deep problem, but you were forced to pause your thinking every ten seconds. After every ten seconds of thinking, you had to write down one word and then have your memory of the problem wiped, conscious and unconscious. At the beginning of the next ten seconds, all you’d have access to is the words you’d written so far. Every deep exploration into combinatorial space would be cut short and have to begin anew. All your implicit memory about which paths seemed promising and which went nowhere would be wiped, as would any mental abstractions you’d formed along the way. If your mind operated like this, solving deep problems would be a nightmare. Encoding all your unconscious, parallel thinking into a single word is a hopeless endeavor. But this is how LLMs work!
I basically agree that lack of neuralese/recurrence/etc. is probably significantly hampering AI capabilities (though ofc, it clearly has benefits for training efficiency otherwise the companies wouldn’t be doing it) however I’m not so convinced that CoT alone can’t get us to the SC milestone. A direct reply to your argument: Yes, it would suck if you had to have your memory wiped and write stuff down every ten seconds. But that’s partly because you haven’t been trained to live that way! Imagine instead the following hypothetical: --Your natural brain short-term memory (but not long-term memory) is wiped every ten seconds. But you have a Neuralink device and retinal implant, and the neuralink device lets you ‘think’ arbitrary text into a text file, and the text is displayed on your retinal implant. At first, this would suck. But after years and years of training, you’d learn to use the text to think, much like ‘reasoning models’ do. Your thinking would presumably be less efficient, you’d still be worse off compared to ordinary humans, but it would be way way better than it was at the beginning before you had practice. After practice, you’d basically just be verbalizing your inner monologue constantly, putting it into the text stream, and then simultaneously reading it and also paying attention to what’s happening in the world around you.
I think that in theory there is nothing wrong with having your memory wiped every iteration, and that such an architecture could in theory get us to SC. I just think it’s not very efficient and there would be a lot of repeated computation happening between predicting each word.
I mean yeah, totally agree re: repeated computation and inefficiency. But there’s no rule that says the first SC has to be close to the limits of efficiency. On the contrary, just as how the first viable airplanes were extremely shitty compared to the airplanes of today, the first viable SC will probably be shitty in various ways (e.g. data-efficiency) and perhaps this’ll be one of those ways.
I basically agree that lack of neuralese/recurrence/etc. is probably significantly hampering AI capabilities (though ofc, it clearly has benefits for training efficiency otherwise the companies wouldn’t be doing it) however I’m not so convinced that CoT alone can’t get us to the SC milestone. A direct reply to your argument: Yes, it would suck if you had to have your memory wiped and write stuff down every ten seconds. But that’s partly because you haven’t been trained to live that way! Imagine instead the following hypothetical:
--Your natural brain short-term memory (but not long-term memory) is wiped every ten seconds. But you have a Neuralink device and retinal implant, and the neuralink device lets you ‘think’ arbitrary text into a text file, and the text is displayed on your retinal implant. At first, this would suck. But after years and years of training, you’d learn to use the text to think, much like ‘reasoning models’ do. Your thinking would presumably be less efficient, you’d still be worse off compared to ordinary humans, but it would be way way better than it was at the beginning before you had practice. After practice, you’d basically just be verbalizing your inner monologue constantly, putting it into the text stream, and then simultaneously reading it and also paying attention to what’s happening in the world around you.
I think that in theory there is nothing wrong with having your memory wiped every iteration, and that such an architecture could in theory get us to SC. I just think it’s not very efficient and there would be a lot of repeated computation happening between predicting each word.
I mean yeah, totally agree re: repeated computation and inefficiency. But there’s no rule that says the first SC has to be close to the limits of efficiency. On the contrary, just as how the first viable airplanes were extremely shitty compared to the airplanes of today, the first viable SC will probably be shitty in various ways (e.g. data-efficiency) and perhaps this’ll be one of those ways.