On this dataset, I find that Gemini 3 Pro gets 60% of 2-hop questions right and 34% of 3-hop questions right.
I initially got tripped up by the wording here: I thought this was 60% accuracy on 2-hop questions in a forward pass, not with 300 filler tokens, which aren’t mentioned until later in the post.
It’s a good piece, but wanted to comment in case someone else gets confused at the same spot.
It is in a single forward pass, just with additional fixed irrelevant tokens after. I think this still counts as “in a single forward pass” for the typical usage of the term. (It just doesn’t know the answer until somewhat later tokens.)
Separately, worth noting that the model doesn’t do that much worse without filler: performance only drops to 46%, 18%.
My choice to not mention the filler yet was because I didn’t want to introduce too much complexity yet and I think this number is the most representative number from a misalignment risk perspective; models will typically have a bunch of tokens somewhere they can use to do opaque reasoning.
I initially got tripped up by the wording here: I thought this was 60% accuracy on 2-hop questions in a forward pass, not with 300 filler tokens, which aren’t mentioned until later in the post.
It’s a good piece, but wanted to comment in case someone else gets confused at the same spot.
It is in a single forward pass, just with additional fixed irrelevant tokens after. I think this still counts as “in a single forward pass” for the typical usage of the term. (It just doesn’t know the answer until somewhat later tokens.)
Separately, worth noting that the model doesn’t do that much worse without filler: performance only drops to 46%, 18%.
My choice to not mention the filler yet was because I didn’t want to introduce too much complexity yet and I think this number is the most representative number from a misalignment risk perspective; models will typically have a bunch of tokens somewhere they can use to do opaque reasoning.