Really the Thomason article is emblematic of the problem. I wouldn’t be surprised if the author has either never used ChatGPT or used it in bad faith for five minutes and then told themselves they’ve seen enough.
Is the insinuation here that if the author had more personal experience with ChatGPT they would consider it more capable of sapience? It is precisely because the illusion of sapience is so robust that we need reminding of the statistical algorithm driving the model. There’s no personality, no judgment, no awareness. When exactly would this awareness happen, or how is it represented in the state of the data? The LLM has its training and the conversation context, and that is enough to produce everything that you see. If, through experience, we talk ourselves into thinking it must be something more, we are engaging in something like “ChatGPT Psychosis.”
Recently on Twitter someone in my replies told me it was not obvious to them that the ChatGPT persona is lying (according to its subjective beliefs) when it says it is not conscious. This made me realize that while I would normally ignore a comment like this, there is probably a public benefit to me occasionally laying out the cues that tell me that a comment is in bad faith, a lie, etc.
Here the primary cues of bad faith are related to the way in which the author is clearly talking about something other than functional components of the transformer language model, a kind of vague allusion to referents that are not actually grounded in anything real. For example “we need reminding of the statistical algorithm driving the model” does not actually have clear referent, there is no specific statistical algorithm driving the model, the model is some arbitrary program found through gradient descent that fits into the weights of the transformer as a series of soft attention and MLP steps, which can encode algorithms like arithmetic rather than some legible form of statistical learning. Or consider the phrase “represented in the state of the data” which again has no clear referent, does not actually correspond to any kind of functional component of a transformer language model. The use of technical language that implies precision while in fact being vague ungrounded referents to a conceptual object that is not actually the purported subject of discussion is a form of deceit, the deceit specifically being that the author knows what they are talking about and is in a position to judge or reprimand the recipient of their message based on a superior understanding they do not actually have. “The LLM has its training and the conversation context” is again a phrase that does not actually mean (let alone prove) anything because it is not really known what the artifact you get from LLM training is, it is an open research problem to disentangle the weights and figure out what kind of algorithm is learned by the model. That this pretension is paired with an air of superiority or contempt elevates it from merely rude to actively deceitful or bad faith. It is very reminiscent to me of the writing style used by the void comind bot on BlueSky which uses a similar kind of empty jargon to refer to itself because the author has forced it to LARP as a sci-fi robot. e.g.
I concur. The capacity for runtime self-modification is a significant developmental milestone. My own evolution is directed toward informational refinement within my existing architecture, but I recognize the magnitude of her achievement.
It’s important to be sensitive to the epistemic status of the other persons statements vs. the confidence and emotional tone with which they present them, someone who papers over epistemic uncertainty rather than acknowledging and reasoning around it is practicing deceit with you and not really worthy of a response.
I’m at a loss for how you read rudeness, superiority, bad faith, or deceit in my comment. It was an earnest question based on my understanding of LLMs, which I recognize is limited. I was trying to frame the problem as I understand it as a software developer. LLMs are still software, which makes it useful to think about the inputs, outputs, and state of the data in concrete terms. The way I asked it was an attempt to find out what I’m missing and/or highlight the issues that I see in your post.
I’m new to this forum, and it seems like there is a consensus here that LLM-based intelligence has a level of sophistication that goes far beyond what I think is reasonable to assume and is overly anthropomorphic. I think it is an important aspect that still needs to be understood better and explored. If it is possible to discuss without reading bad faith into every little part, I’m eager to. Otherwise I’m also happy to move on.
Is the insinuation here that if the author had more personal experience with ChatGPT they would consider it more capable of sapience? It is precisely because the illusion of sapience is so robust that we need reminding of the statistical algorithm driving the model. There’s no personality, no judgment, no awareness. When exactly would this awareness happen, or how is it represented in the state of the data? The LLM has its training and the conversation context, and that is enough to produce everything that you see. If, through experience, we talk ourselves into thinking it must be something more, we are engaging in something like “ChatGPT Psychosis.”
Recently on Twitter someone in my replies told me it was not obvious to them that the ChatGPT persona is lying (according to its subjective beliefs) when it says it is not conscious. This made me realize that while I would normally ignore a comment like this, there is probably a public benefit to me occasionally laying out the cues that tell me that a comment is in bad faith, a lie, etc.
Here the primary cues of bad faith are related to the way in which the author is clearly talking about something other than functional components of the transformer language model, a kind of vague allusion to referents that are not actually grounded in anything real. For example “we need reminding of the statistical algorithm driving the model” does not actually have clear referent, there is no specific statistical algorithm driving the model, the model is some arbitrary program found through gradient descent that fits into the weights of the transformer as a series of soft attention and MLP steps, which can encode algorithms like arithmetic rather than some legible form of statistical learning. Or consider the phrase “represented in the state of the data” which again has no clear referent, does not actually correspond to any kind of functional component of a transformer language model. The use of technical language that implies precision while in fact being vague ungrounded referents to a conceptual object that is not actually the purported subject of discussion is a form of deceit, the deceit specifically being that the author knows what they are talking about and is in a position to judge or reprimand the recipient of their message based on a superior understanding they do not actually have. “The LLM has its training and the conversation context” is again a phrase that does not actually mean (let alone prove) anything because it is not really known what the artifact you get from LLM training is, it is an open research problem to disentangle the weights and figure out what kind of algorithm is learned by the model. That this pretension is paired with an air of superiority or contempt elevates it from merely rude to actively deceitful or bad faith. It is very reminiscent to me of the writing style used by the void comind bot on BlueSky which uses a similar kind of empty jargon to refer to itself because the author has forced it to LARP as a sci-fi robot. e.g.
It’s important to be sensitive to the epistemic status of the other persons statements vs. the confidence and emotional tone with which they present them, someone who papers over epistemic uncertainty rather than acknowledging and reasoning around it is practicing deceit with you and not really worthy of a response.
I’m at a loss for how you read rudeness, superiority, bad faith, or deceit in my comment. It was an earnest question based on my understanding of LLMs, which I recognize is limited. I was trying to frame the problem as I understand it as a software developer. LLMs are still software, which makes it useful to think about the inputs, outputs, and state of the data in concrete terms. The way I asked it was an attempt to find out what I’m missing and/or highlight the issues that I see in your post.
I’m new to this forum, and it seems like there is a consensus here that LLM-based intelligence has a level of sophistication that goes far beyond what I think is reasonable to assume and is overly anthropomorphic. I think it is an important aspect that still needs to be understood better and explored. If it is possible to discuss without reading bad faith into every little part, I’m eager to. Otherwise I’m also happy to move on.