The bat thing might have just been Thomas Nagel, I can’t find the source I thought I remembered.
At one point I said LLMs forget everything they thought previously between predicting (say) token six and seven and half to work from scratch. Because of the way the attention mechanism works it is actually a little more complicated (see the top comment from hmys). What I said is (I believe) still overall right but I would put that detail less strongly.
Hofstadter apparently was the one who said a human-level chess AI would rather talk about poetry.
Did a podcast interview with Ayush Prakash on the AIXI model (and modern AI), very introductory/non-technical:
Some errata:
The bat thing might have just been Thomas Nagel, I can’t find the source I thought I remembered.
At one point I said LLMs forget everything they thought previously between predicting (say) token six and seven and half to work from scratch. Because of the way the attention mechanism works it is actually a little more complicated (see the top comment from hmys). What I said is (I believe) still overall right but I would put that detail less strongly.
Hofstadter apparently was the one who said a human-level chess AI would rather talk about poetry.