It is in a single forward pass, just with additional fixed irrelevant tokens after. I think this still counts as “in a single forward pass” for the typical usage of the term. (It just doesn’t know the answer until somewhat later tokens.)
Separately, worth noting that the model doesn’t do that much worse without filler: performance only drops to 46%, 18%.
My choice to not mention the filler yet was because I didn’t want to introduce too much complexity yet and I think this number is the most representative number from a misalignment risk perspective; models will typically have a bunch of tokens somewhere they can use to do opaque reasoning.
It is in a single forward pass, just with additional fixed irrelevant tokens after. I think this still counts as “in a single forward pass” for the typical usage of the term. (It just doesn’t know the answer until somewhat later tokens.)
Separately, worth noting that the model doesn’t do that much worse without filler: performance only drops to 46%, 18%.
My choice to not mention the filler yet was because I didn’t want to introduce too much complexity yet and I think this number is the most representative number from a misalignment risk perspective; models will typically have a bunch of tokens somewhere they can use to do opaque reasoning.