Unfortunately your validation loss of 0.022 nats isn’t plausible, as it’s below the intrinsic entropy of the text (estimated to be about 1 bit/char or 0.69 nats for english text). So there is almost certainly a bug somewhere, probably having to do with masking or data leakage. One way to test this would be to train on uniform random sequences and check that your loss converges to, and isn’t below, the base entropy of log(vocab_size) for both models.
Yes, thanks for catching that, that graph is from an earlier version that had a data leakage bug. The current model and benchmark are in the linked repo. I would really appreciate if you could replicate the results. The actual result is 1.596 nats, not 0.022.
Unfortunately your validation loss of 0.022 nats isn’t plausible, as it’s below the intrinsic entropy of the text (estimated to be about 1 bit/char or 0.69 nats for english text). So there is almost certainly a bug somewhere, probably having to do with masking or data leakage. One way to test this would be to train on uniform random sequences and check that your loss converges to, and isn’t below, the base entropy of log(vocab_size) for both models.
Yes, thanks for catching that, that graph is from an earlier version that had a data leakage bug. The current model and benchmark are in the linked repo. I would really appreciate if you could replicate the results. The actual result is 1.596 nats, not 0.022.