How do we know in this model if the identity of failed subtasks is constant, that there is a subset of subtask such that P percent of subtasks are passed with probability zero, vs all subtasks are uniform, but the hazard rate comes from a chance of unrecoverable failure on each individual task step.
Remember that the main graph is 50% pass rate over resampling, so it would be the correlation of supertasks to resampling that would tell you of few hard subtasks vs lots of not so hard ones with nonzero hazard to every task.
The correct test is both whether model performance on individual tasks correlates, and whether tasks have run to run pass rates other than zero and 1. Doing that analysis suggests that there is a subset of hard tasks, but that subset is pass sometimes as well as pass never.
I think that’s right—there are some subtasks (and hence some tasks) which sit at the threshold of being occasionally doable (see fn 9). METR’s data will have this per-task success rate but I haven’t looked whether they expose that anywhere for analysis. When I asked Claude, it said a U shape isn’t uncommon for evals individual tasks success rates, though not across the board and I don’t know how much weight to put on that. Would want to look properly.
A U-shape for individual task success rates is a natural consequence of success being logistic-like in nature or contingent on many logistic tasks at once. Suppose that depends on the LLM’s abilities as where is the difference between the LLM’s capability level on a frontier and the problem’s difficulty on the frontier. Then and Since is distributed rather uniformly, the U-shape isn’t surprising.
How do we know in this model if the identity of failed subtasks is constant, that there is a subset of subtask such that P percent of subtasks are passed with probability zero, vs all subtasks are uniform, but the hazard rate comes from a chance of unrecoverable failure on each individual task step.
Remember that the main graph is 50% pass rate over resampling, so it would be the correlation of supertasks to resampling that would tell you of few hard subtasks vs lots of not so hard ones with nonzero hazard to every task.
The correct test is both whether model performance on individual tasks correlates, and whether tasks have run to run pass rates other than zero and 1. Doing that analysis suggests that there is a subset of hard tasks, but that subset is pass sometimes as well as pass never.
I think that’s right—there are some subtasks (and hence some tasks) which sit at the threshold of being occasionally doable (see fn 9). METR’s data will have this per-task success rate but I haven’t looked whether they expose that anywhere for analysis. When I asked Claude, it said a U shape isn’t uncommon for evals individual tasks success rates, though not across the board and I don’t know how much weight to put on that. Would want to look properly.
A U-shape for individual task success rates is a natural consequence of success being logistic-like in nature or contingent on many logistic tasks at once. Suppose that
P.S. Could you take a look at my model involving macrostrategy levels?