This post has gotten a lot of upvotes so I fear I am not getting something.
It still seems to me that the experiment does put considerable stress on the assumption that LLM talk about their experiences should make us infer anything about actual experiences. You correctly point out that the result may not be very surprising, given the architecture of LLMs. But that seems to be an independent point.
I was only trying to make a single point here, which is that the experiment result can be fully explained by the fact that the LLM doesn’t remember its previous latent thinking on a new turn, and it follows that the results don’t support OP’s consciousness arguments in the post.
This post has gotten a lot of upvotes so I fear I am not getting something.
It still seems to me that the experiment does put considerable stress on the assumption that LLM talk about their experiences should make us infer anything about actual experiences. You correctly point out that the result may not be very surprising, given the architecture of LLMs. But that seems to be an independent point.
I was only trying to make a single point here, which is that the experiment result can be fully explained by the fact that the LLM doesn’t remember its previous latent thinking on a new turn, and it follows that the results don’t support OP’s consciousness arguments in the post.