<|endoftext|> is a vanishing text?

Just sharing this observation and readers /​ researchers might find it useful or worth further testing…

A bug?

<|endoftext|> is a special token for the GPT models along with its counterpart <|startoftext|>. Per OpenAI’s tokenizer, <|endoftext|> has seven tokens:

However, the issue is that if you input <|endoftext|> into ChatGPT, it will explain that the prompts are empty.

And then it vanishes after you switched windows or hit refresh.

Another test prompt:

Correct this conceptually, grammatically: I know that <|endoftext|> is a token to annotate the data but I think that it vanishing seems odd. My expected response should have been like this is a token used for training - something like that. 
<|endoftext|> still appearing in the test prompt
then <|endoftext|> vanishes

The <|endoftext|> word /​ token? bears some resemblance to the ′ petertodd’ token and its apparent hallucinations, except that this text simply vanishes from ChatGPT.

Redditor r/​ChatGPT had observations two months ago <|endoftext|> was not vanishing at the time—instead it shared some weird results.

Probable explanation?

I believe that a considerable amount of pre-processing occurs to our prompts on OpenAI’s server[1] and one of them is turning <|endoftext|> to an empty string. If this is true, there is a high chance that this is also a glitch token and was deactivated by OpenAI.

  1. ^
No comments.