log-linear scaling of x with pre-training compute will be worth it as the k-step success rate will improve near-linearly
I don’t follow. The k-step success is polynomial in x, not exponential (it’s xk, not kx).
Although if we fix some cutoff c for the k-step success probability, and then look at the value of k for which xk=c , then we get k=log(c)/log(x)∼1/log(x). This is super-linear in x over the interval from 0 to 1, so linearly growing improvements in x cause this “highest feasible k” to grow faster-than-linearly. (Is this what you meant? Note that this is similar to how METR computes time horizons.)
METR found that horizon lengths are growing exponentially in time, not linearly.
(One-step success probabilities have been growing at least linearly with time, I would think – due to super-linear growth in inputs like dataset size, etc. – so we should expect horizon lengths to grow super-linearly due to what I said in the previous paragraph.)
(N.B. I expect it will be easier to conduct this kind of analysis in terms of logit(x) instead of x.)
I don’t follow. The k-step success is polynomial in x, not exponential (it’s xk, not kx).
Although if we fix some cutoff c for the k-step success probability, and then look at the value of k for which xk=c , then we get k=log(c)/log(x)∼1/log(x). This is super-linear in x over the interval from 0 to 1, so linearly growing improvements in x cause this “highest feasible k” to grow faster-than-linearly. (Is this what you meant? Note that this is similar to how METR computes time horizons.)
METR found that horizon lengths are growing exponentially in time, not linearly.
(One-step success probabilities have been growing at least linearly with time, I would think – due to super-linear growth in inputs like dataset size, etc. – so we should expect horizon lengths to grow super-linearly due to what I said in the previous paragraph.)
(N.B. I expect it will be easier to conduct this kind of analysis in terms of logit(x) instead of x.)
Thanks! I fixed the last paragraph accordingly. I indeed wanted to say faster-than-linearly for the highest feasible k.