This could be obvious to most people here but can you briefly explain how Neuralese is a new thing and not just “how LLMs worked before COT was invented”?
Even without understanding that though, I found this post excellent at catching me up on the topic!
Not an expert, but I think the difference is this. Current and older LLMs produce each token via a “forward pass”: information only flows forwards in the model, what happens at a later layer can’t influence what happens at an earlier layer. What people call “neuralese” is to build neural nets where information can also flow backwards in the model, so that it can pass information back to itself “in its head” rather than only being able to pass information back to itself by outputting tokens and then reading them back in. This is a known technique and has been done before, but it’s hard to train large models with that architecture. I was going to try to explain why but realized I don’t understand well enough myself to explain it, so I’ll leave it there.
This could be obvious to most people here but can you briefly explain how Neuralese is a new thing and not just “how LLMs worked before COT was invented”?
Even without understanding that though, I found this post excellent at catching me up on the topic!
Not an expert, but I think the difference is this. Current and older LLMs produce each token via a “forward pass”: information only flows forwards in the model, what happens at a later layer can’t influence what happens at an earlier layer. What people call “neuralese” is to build neural nets where information can also flow backwards in the model, so that it can pass information back to itself “in its head” rather than only being able to pass information back to itself by outputting tokens and then reading them back in. This is a known technique and has been done before, but it’s hard to train large models with that architecture. I was going to try to explain why but realized I don’t understand well enough myself to explain it, so I’ll leave it there.