A benchmark is a large set of somewhat similar tasks, so I’m thinking of them the same as families of tasks. I agree that some kinds of task will have larger or smaller λ (e.g. some should be very close to 0 and very close to 1) and characterising this would be useful. It would be especially useful to know if there are any with λ > 1.
I think we’ll mainly see inference compute going into longer chain of thought except where:
speed is very valuable
or λ is close to 1
or so much compute has been expended that we are at or near the plateau
A benchmark is a large set of somewhat similar tasks, so I’m thinking of them the same as families of tasks. I agree that some kinds of task will have larger or smaller λ (e.g. some should be very close to 0 and very close to 1) and characterising this would be useful. It would be especially useful to know if there are any with λ > 1.
I think we’ll mainly see inference compute going into longer chain of thought except where:
speed is very valuable
or λ is close to 1
or so much compute has been expended that we are at or near the plateau