I’m not sure how interesting this is, but I relaxed the positive-increment constraint and removed the sigmoid, and the model only learned one negative increment, specifically the last byte of the Chinese possessive modifier (的), although ’s in English sometimes got slightly-negative increments.
There’s definitely aspects of this that make me think it’s constrainted by the inability to change its mind, since s’ in English (plural possessive) gets a larger increment, presumably because it’s too late to change “s” and the model likes to separate on punctuation. I’m guessing if we can figure out a way to let the model decide on spacing after seeing the whole sentence it would do something more interesting.
Interesting! Currently, you are deciding the increment to the next token… using which activations? Post transformer? Post MLP?
It seems like maybe what you’re looking for is, for each layer, determine the increment from the previous token after the previous layer’s MLP (for the current token). Since layer 0 you have no previous layer to reference, maybe it just uses standard RoPE, or maybe you do something similar to what you’re doing now where you determine the layer 0 increment based on the end representation of the previous token.
The DeltaMLP blocks consume the incoming residual stream (or embeddings) for the current token between the transformer blocks, so we always have one (although advancing the position before position 0′s is meaningless since only relative position matters). For layer 0, we use the token’s embedding, so the position is fully static per-token.
From the attention section’s perspective, we’re feeding in the embedding or residual the same way we normally would, and the only difference is that Q and K are also rotated based on the sum of all calculated position increments instead of the position count.
I’m not sure how interesting this is, but I relaxed the positive-increment constraint and removed the sigmoid, and the model only learned one negative increment, specifically the last byte of the Chinese possessive modifier (的), although ’s in English sometimes got slightly-negative increments.
There’s definitely aspects of this that make me think it’s constrainted by the inability to change its mind, since s’ in English (plural possessive) gets a larger increment, presumably because it’s too late to change “s” and the model likes to separate on punctuation. I’m guessing if we can figure out a way to let the model decide on spacing after seeing the whole sentence it would do something more interesting.
Interesting! Currently, you are deciding the increment to the next token… using which activations? Post transformer? Post MLP?
It seems like maybe what you’re looking for is, for each layer, determine the increment from the previous token after the previous layer’s MLP (for the current token). Since layer 0 you have no previous layer to reference, maybe it just uses standard RoPE, or maybe you do something similar to what you’re doing now where you determine the layer 0 increment based on the end representation of the previous token.
The DeltaMLP blocks consume the incoming residual stream (or embeddings) for the current token between the transformer blocks, so we always have one (although advancing the position before position 0′s is meaningless since only relative position matters). For layer 0, we use the token’s embedding, so the position is fully static per-token.
From the attention section’s perspective, we’re feeding in the embedding or residual the same way we normally would, and the only difference is that Q and K are also rotated based on the sum of all calculated position increments instead of the position count.