I think the requirement that increments be positive probably isn’t necessary, although the problem where the model can’t look forward prevents a lot of uses for this (it might work decently with BPE though).
For retroactively changing positions, I think training would be unstable and finnicky, but it would be interesting if you could get it to work.
Something related that I’m interested in is BLT/H-Net style models that do dynamic chunking instead of tokenization. H-Net does this at multiple scales, so it could theoretically group words in the first layer and concepts in the second. I haven’t actually tried to train or inspect on yet though.
I think the requirement that increments be positive probably isn’t necessary, although the problem where the model can’t look forward prevents a lot of uses for this (it might work decently with BPE though).
For retroactively changing positions, I think training would be unstable and finnicky, but it would be interesting if you could get it to work.
Something related that I’m interested in is BLT/H-Net style models that do dynamic chunking instead of tokenization. H-Net does this at multiple scales, so it could theoretically group words in the first layer and concepts in the second. I haven’t actually tried to train or inspect on yet though.