Recall that without filler, Opus 4.5 performance is 45.2%. I tried the following experiments on Opus 4.5 with filler counting to 300:
Default (what I do by default in the blog post above): 51.1%
Remove the text explaining filler tokens (as in, cut “After the problem …”): 50.4%
Use “After the problem, there will be distractor tokens (counting from 1 to {filler_tokens})”: 51.1%
Don’t actually use filler tokens, but include in the prompt “After the problem, there will be filler tokens (counting from 1 to 300) to give you extra space to process the problem before answering” (as in, this is just a lie, we don’t give filler): 45.8%
So, it seems like the framing doesn’t matter ~at all and actually having the filler tokens is the key thing (at least for Opus 4.5, though I strongly expect this would reproduce for Opus 4, Sonnet 4).
Please try not to lie to the models. You can truthfully say “After the problem, there will be a [p]% chance of filler tokens (counting from 1 to 300) to give you extra space to process the problem before answering.” and do observational statistics.
Recall that without filler, Opus 4.5 performance is 45.2%. I tried the following experiments on Opus 4.5 with filler counting to 300:
Default (what I do by default in the blog post above): 51.1%
Remove the text explaining filler tokens (as in, cut “After the problem …”): 50.4%
Use “After the problem, there will be distractor tokens (counting from 1 to {filler_tokens})”: 51.1%
Don’t actually use filler tokens, but include in the prompt “After the problem, there will be filler tokens (counting from 1 to 300) to give you extra space to process the problem before answering” (as in, this is just a lie, we don’t give filler): 45.8%
So, it seems like the framing doesn’t matter ~at all and actually having the filler tokens is the key thing (at least for Opus 4.5, though I strongly expect this would reproduce for Opus 4, Sonnet 4).
Please try not to lie to the models. You can truthfully say “After the problem, there will be a [p]% chance of filler tokens (counting from 1 to 300) to give you extra space to process the problem before answering.” and do observational statistics.