there’s a mental move of going up and down the ladder of abstraction, where you zoom in on some particularly difficult and/or confusing part of the problem, solve it, and then use what you learned from that to zoom back out and fill in a gap in the larger problem you were trying to solve. For an LLM, that seems like it’s harder, and indeed it’s one of the reasons I inside-view suspect LLMs as-currently-trained might not actually scale to AGI. [bold by me]
But that might already no longer be true with model that have short term memory and may might make moves like you. See my Leave No Context Behind—A Comment.
But that might already no longer be true with model that have short term memory and may might make moves like you. See my Leave No Context Behind—A Comment.