Hey, I am a Data Science undergraduate and recently I’ve gotten involved in AIS. I read this post and ended up wanting to build the measurement-error layer you flagged (and that Alexander Barry mentioned in his note). Apparently the raw per-run data is already on the same METR repo.
Having done that I have found at least three things you’ll find interesting:
1. You suggested trying a Weibull instead of log-normal for the human timing distribution. What I got is that real within-task residuals are right-skewed with heavy tails (skew +1.09, excess kurtosis 8.2) while a Weibull log is fixed left-skewed regardless of its shape parameter so it can’t possibly match this data. PSIS-LOO on the baseline runs: Student-t −1097.5, log-normal −1164.9, Weibull −1219.3 for comparison.
Baseline-duration residuals: observed data vs the log-normal, Student-t, and Weibull-implied densities
2. You also noted a visible kink in the trajectory (pre-Sonnet-3.5, in the appendix) and said you wanted to try a piecewise linear near o1 but skipped it because of compute. I fit it: the breakpoint lands at early 2024 actually, and the slope goes from 0.41 to 3.14 after the bend, and it gets 64% of the Bayesian stacking weight over linear though honestly the two fit almost equally well, so call it a lean, not a clear win.
Horizon vs release date, all four trend shapes with credible bands, forecast to 2029
3. I also used Barry’s comment below directly, ~60% of estimate-only annotations land within 3x of the real baseline, which implies a sigma_est prior around 1.25 log-minutes rather than the 0.8 I’d been using but refitting that doesn’t change a lot.
Hey, I am a Data Science undergraduate and recently I’ve gotten involved in AIS. I read this post and ended up wanting to build the measurement-error layer you flagged (and that Alexander Barry mentioned in his note). Apparently the raw per-run data is already on the same METR repo.
Having done that I have found at least three things you’ll find interesting:
1.
You suggested trying a Weibull instead of log-normal for the human timing distribution. What I got is that real within-task residuals are right-skewed with heavy tails (skew +1.09, excess kurtosis 8.2) while a Weibull log is fixed left-skewed regardless of its shape parameter so it can’t possibly match this data. PSIS-LOO on the baseline runs: Student-t −1097.5, log-normal −1164.9, Weibull −1219.3 for comparison.
Baseline-duration residuals: observed data vs the log-normal, Student-t, and Weibull-implied densities2.
You also noted a visible kink in the trajectory (pre-Sonnet-3.5, in the appendix) and said you wanted to try a piecewise linear near o1 but skipped it because of compute. I fit it: the breakpoint lands at early 2024 actually, and the slope goes from 0.41 to 3.14 after the bend, and it gets 64% of the Bayesian stacking weight over linear though honestly the two fit almost equally well, so call it a lean, not a clear win.
Horizon vs release date, all four trend shapes with credible bands, forecast to 20293.
I also used Barry’s comment below directly, ~60% of estimate-only annotations land within 3x of the real baseline, which implies a sigma_est prior around 1.25 log-minutes rather than the 0.8 I’d been using but refitting that doesn’t change a lot.
Here I leave you the full code implementation:
https://github.com/korentomas/metr-measurement-error