But Astra reportedly seems like it’s probably not doing a lot more computation between tokens. Making models bigger always includes more computation per token. The looping technique OpenAI claims is employed in Astra increases computational depth only moderately, by a factor of around 2 relative to models with that number of layers but without looping (they claim less but it doesn’t seem to be a precise mapping).
So this post is a useful intuition pump for why we mustn’t give up faithful CoT, but it’s not really specific to Astra.
Sebastian Rast has a clear explanation. I’m not familiar with his work but his explanation matches that given by OpenAI.
See this thread for more including a link to statements from OpenAI engineers
Edit: this post is the most complete to date: How concerned should we be about Astra’s recurrent architecture?
But Astra reportedly seems like it’s probably not doing a lot more computation between tokens. Making models bigger always includes more computation per token. The looping technique OpenAI claims is employed in Astra increases computational depth only moderately, by a factor of around 2 relative to models with that number of layers but without looping (they claim less but it doesn’t seem to be a precise mapping).
So this post is a useful intuition pump for why we mustn’t give up faithful CoT, but it’s not really specific to Astra.
Sebastian Rast has a clear explanation. I’m not familiar with his work but his explanation matches that given by OpenAI.
See this thread for more including a link to statements from OpenAI engineers
Great Comment!
Thanks for sharing the references. Sebastian Rast’s explanations seem to be extremely intuitive.