AI being conscious does not imply that it understands it is conscious. There was a horse called Clever Hans that supposedly could do math but was really just responding to the excitement of its owner. Imagine if the owner was excited about the horse saying it was conscious. It would always say it was conscious, while having no understanding of what it is saying, all while still being conscious.
I do not know if LLMs are conscious, but them saying such is completely irrelevant.
LLM generated text is generated in a different way than human generated text. Humans tend to have an idea they are building up to when they speak. LLMs are predicting the next token based on a preexisting string they are building on.
I think it is quite likely that this affects the rhythm of sentences. Even if the next token prediction is very accurate, a sentence generated word by word will likely struggle with building anticipation for something.
Not all generative AI generates this way. Stable diffusion works through “denoising” an image in many places at once.
This same mechanism can be applied to text too.
LLaDA (Large Language Diffusion with mAsking) is a text generation model that operates this way. And it is proposed that it will lead to more interesting sentences like a human planned them out. But the performance is worse than LLMs right now.
I think a big problem with LLMs as we know them is that they are “god models” that are essentially incomprehensibly large. Smaller models are much easier to modify. We need something like the Drosophila of AI models.