Thinking about whether Fig 2 makes sense seems like a really good point.
When I was playing with what I could make of the data in a different comment I plotted the different tasks. One can observe that there is only a single task (SHADE-Arena Monitor TPR @ 1% FPR) is in this >0.5hr regime. I think this should feel pretty suspect. Anything you see on this right side of the figure 2 plots is coming entirely from just one task, with a metric that might think could be pretty spikey / non-linear. This one task absolutely does not represent the universe of >0.5hr imprtant tasks.
Seems pretty important, unless I’m confused where there are other tasks here.
Thinking about whether Fig 2 makes sense seems like a really good point.
When I was playing with what I could make of the data in a different comment I plotted the different tasks. One can observe that there is only a single task (SHADE-Arena Monitor TPR @ 1% FPR) is in this >0.5hr regime. I think this should feel pretty suspect. Anything you see on this right side of the figure 2 plots is coming entirely from just one task, with a metric that might think could be pretty spikey / non-linear. This one task absolutely does not represent the universe of >0.5hr imprtant tasks.
Seems pretty important, unless I’m confused where there are other tasks here.
thanks! some discussion trying to condense in other thread.