<|endoftext|> is a vanishing text?
Just sharing this observation and readers / researchers might find it useful or worth further testing…
<|endoftext|> is a special token for the GPT models along with its counterpart <|startoftext|>. Per OpenAI’s tokenizer, <|endoftext|> has seven tokens:
However, the issue is that if you input <|endoftext|> into ChatGPT, it will explain that the prompts are empty.
And then it vanishes after you switched windows or hit refresh.
Another test prompt:
Correct this conceptually, grammatically: I know that <|endoftext|> is a token to annotate the data but I think that it vanishing seems odd. My expected response should have been like this is a token used for training - something like that.
I believe that a considerable amount of pre-processing occurs to our prompts on OpenAI’s server and one of them is turning <|endoftext|> to an empty string. If this is true, there is a high chance that this is also a glitch token and was deactivated by OpenAI.