This is really fantastic work and I hope to comment more intelligently after I reread it a few times but in case I don’t get to it, kudos!
Joseph Eisner
Karma: 7
Interesting! Currently, you are deciding the increment to the next token… using which activations? Post transformer? Post MLP?
It seems like maybe what you’re looking for is, for each layer, determine the increment from the previous token after the previous layer’s MLP (for the current token). Since layer 0 you have no previous layer to reference, maybe it just uses standard RoPE, or maybe you do something similar to what you’re doing now where you determine the layer 0 increment based on the end representation of the previous token.
This is really interesting!
I’m curious what happens if you relax the accumulating sum, allowing the model to “reposition” the tokens as it desires. Due to the causal mask the model already has access to the ordering (which is why NoPE is an effective positional encoding), but this might allow it to move related words near eachother, even if they are not adjacent in the sentence.
Moreover, you could provide multiple dimensions of “position” with which to do this, by say having a rotary encoding on the first half of the latent vectors and then a separate independently learned encoding on the second half.
I wonder how (or if) the model would use this to group tokens at various layers.
Did you use a learning rate schedule? Cosine anneal? If not: probably should. If so: the loss/perplexity/bpc would always appear to plateau as you finish the schedule, it doesn’t imply more training wouldn’t be beneficial...
It seems like there are two unrelated ideas here, can you help bridge them?
(1) Posting new techniques online means those techniques get incorporated into the next gen of frontier LLM (or do you mean that the technique is in the training data so the next gen frontier LLM is merely aware of the technique?).
(2) Agents have dramatically different safe agent-hours, and this doesn’t correspond with how “good” the models are.