Some thoughts on the image at the end of this post:
The title of this M.C. Escher piece is Three Worlds. The image basically has four elements: the still water, the tree reflections, the floating leaves, the fish underneath. The image only makes sense if you realize that each element belongs to its own “world”/dimension, i.e. the trees you see are not actual trees in the water or floating on the water but reflections of trees elsewhere. Another idea I get from the piece is that a single surface contains information from other parts of the world if you look at it correctly.
I’ll draw out the analogy with LLMs. Think about the worlds/dimensions as being different lenses or levels of abstraction. The LLM is the still water, the thing we’re studying and trying to understand. It has an obvious and close connection to the text it outputs, the leaves. (Or maybe the leaves are the logits for the token being predicted?) You might look at this image and say “this is an image of leaves floating on water”; this is like looking at an LLM and saying it’s a next token predictor. It’s true, but it’s only a partial way of understanding what you’re looking at, it’s not the whole picture.
You might also look at the trees and say “ah, the 2d surface of the water is much of it just a reflection of the three-dimensional trees.” You might note how physically embodied and non-ephemeral the trees are compared to the water surface (and also to the leaves, which will only last a season at most, and to the fish, who is always moving and will live only a few years.) The analogy here of the trees is humans. You can say “ah, the ephemeral, non-embodied LLM is much of it just a reflection of humans.” This perspective answers questions of the form “why does the LLM say/do/believe X” with “because that’s what humans say.” (Or perhaps the tree is the user, and this perspective explains LLM behavior like sycophancy and speech register as being a reflection of its user.)
I’m lazy I’ve spent long enough on this text so I’ll leave the fish as an exercise for you, dear reader.
So. This is one way of looking at the Escher piece in the context of AI. I don’t claim it’s the best way—in fact, I think there are probably other ways of rendering an analogy or connection that are more insightful than mine. It’s a very cognitively generative piece of art, which is one reason I like it so much.
Some thoughts on the image at the end of this post:
The title of this M.C. Escher piece is Three Worlds. The image basically has four elements: the still water, the tree reflections, the floating leaves, the fish underneath. The image only makes sense if you realize that each element belongs to its own “world”/dimension, i.e. the trees you see are not actual trees in the water or floating on the water but reflections of trees elsewhere. Another idea I get from the piece is that a single surface contains information from other parts of the world if you look at it correctly.
I’ll draw out the analogy with LLMs. Think about the worlds/dimensions as being different lenses or levels of abstraction. The LLM is the still water, the thing we’re studying and trying to understand. It has an obvious and close connection to the text it outputs, the leaves. (Or maybe the leaves are the logits for the token being predicted?) You might look at this image and say “this is an image of leaves floating on water”; this is like looking at an LLM and saying it’s a next token predictor. It’s true, but it’s only a partial way of understanding what you’re looking at, it’s not the whole picture.
You might also look at the trees and say “ah, the 2d surface of the water is much of it just a reflection of the three-dimensional trees.” You might note how physically embodied and non-ephemeral the trees are compared to the water surface (and also to the leaves, which will only last a season at most, and to the fish, who is always moving and will live only a few years.) The analogy here of the trees is humans. You can say “ah, the ephemeral, non-embodied LLM is much of it just a reflection of humans.” This perspective answers questions of the form “why does the LLM say/do/believe X” with “because that’s what humans say.” (Or perhaps the tree is the user, and this perspective explains LLM behavior like sycophancy and speech register as being a reflection of its user.)
I’m lazyI’ve spent long enough on this text so I’ll leave the fish as an exercise for you, dear reader.So. This is one way of looking at the Escher piece in the context of AI. I don’t claim it’s the best way—in fact, I think there are probably other ways of rendering an analogy or connection that are more insightful than mine. It’s a very cognitively generative piece of art, which is one reason I like it so much.