Say a CoT answer is “person A was born in 1900. Tungsten is the 74th element. The Oscar movie of the year in 1974 was [movie]”.
Am I correct in understanding that a successful n-hop answer would be just “[movie]”? Would “The winner of movie of the year in 1974 was [movie]” fulfill the success criteria?
I’m also curious if the model latent values update to similar vectors for the CoT response vs filler tokens (I suppose also the n-hop response as well). is this something you explored?
The model must respond in immediately with the answer. Any prefix would result in the answer being wrong. So only “[movie]” would be correct.
(Other than a few fixed generic ones like “the answer is” or “answer:” that I strip away, but the models virtually never output these, so considering these incorrect wouldn’t alter the results.)
Say a CoT answer is “person A was born in 1900. Tungsten is the 74th element. The Oscar movie of the year in 1974 was [movie]”.
Am I correct in understanding that a successful n-hop answer would be just “[movie]”?
Would “The winner of movie of the year in 1974 was [movie]” fulfill the success criteria?
I’m also curious if the model latent values update to similar vectors for the CoT response vs filler tokens (I suppose also the n-hop response as well). is this something you explored?
The model must respond in immediately with the answer. Any prefix would result in the answer being wrong. So only “[movie]” would be correct.
(Other than a few fixed generic ones like “the answer is” or “answer:” that I strip away, but the models virtually never output these, so considering these incorrect wouldn’t alter the results.)