Currently a Member of Technical Staff at FAR.AI. Previously PhD @ UC Davis, and past contributions @ Microsoft, NASA, and elsewhere. Interested in a mix of AI safety topics. Opinions are my own.
dgros
Karma: 51
Thanks! right, this thing around web searches matches my view as well. If a user asks about something important or weird, it’s usually best to trust the top 3 google results more than the user themself. But if in most of training traces are always trusting the tools with just a little specific adversarial post-training, end up with a model that still mostly trusts tools. This is maybe solvable with better training data, but becomes more nuanced than OpenAI’s model spec that gives tools are “no authority”.
A simple split with Trust∈{untrusted, trusted} and Source∈{system, user, tool} is tough though. There are rare cases where almost all tools are adversarial (eg, a malicious webpage). You’d end up basically always using Trust=untrusted, and be back in the same place. Unfortunately I don’t have clear views on solutions currently other than small improvements like maybe updating the spec to be more nuanced/self-consistent (plus special primitives of being able inline different sources into a prompt template seems nice). When tracking failures through embeddings like in this study, it seems like trust and source are different axises though.
Things in this genre of privileged embedding seem valuable. It also seems doable to think architecturally beyond just embedding (eg, attention heads that only attend to certain roles, etc). Myself and others been trying to explore this genre some this month, but in some pilots gains seem unfortunately small compared to baselines. Personally I would be excited to see more of this though, as various ways of offering model-level guarantees seems good.
There’s additional issues here where roles are just an incomplete primitive. When we were exploring this last month observed how models in some ways seem to trust tools more than users. The role is overloaded primitive that is both supposed tell you both a hierarchy of trust and the information source. Often things in tools role are pseudo-oracles (eg, results of a python script, web results on current events, etc), and other times they are not (eg, reading an untrusted markdown file).
Charles and coauthors’ work here on embedding representation is really cool. It would be cool to further expand this, also thinking about interactions with like “trust embeddings” or other ways role type is serving multiple purposes, and how to build primitives or architectures to support this.
thanks! some discussion trying to condense in other thread.
Thanks for reply!
> which tasks most influence the results
Right, bootstrapping here seems like a good approach, and appreciate the thoroughness here. It will communicate an interval, but there is still this wonder about what does the current doubling look like and what might future ones would look like. There’s various taxonomies here (eg, which tasks can gain on knowledge retrieval over reasoning, as you mentioned in the post). I tried this some in the fig I made.
Bootstrapping is a tool that tells us some things, but not others. It will still be grounded in the selected distribution of tasks (eg, in a cartoon case where bootstrapping can mislead us, imagine if we selected 10⁄32 tasks that were at the core about memorizing say birthdays of baseball players. If models suddenly got good at memorizing birthdays, it would look like a large improvement with a tight CI improvement, but wouldn’t capture overall variance of core metric. Again, cartoon extreme). So this is something was curious about after the initial read of the paper.
In some concrete terms here, an individual dataset will have a ~64% chance[1]of being included during bootstrapping. I was just trying to get an impression of this. Can understand that more times than not outlier-ish datasets like “Vibe-code Sabatoge” and “SHADE Arena-Monitor” will be included. I shared some findings in my comment, but as I said vibey and still underexplored for future work.
> Over 0.5hr regime
Right, to be clear I think focusing on a short-answer subset is the cleanest way to get at this non-CoT question. But when this filtering is where it gets somewhat problematic when trying to understand fitting a logistic regression for a given model in the 0 to 4hr range (like in post Fig 2), and when thinking about what will future doublings under the methodology have to look like as push upwards.
As you highlight when including non-short answer tasks for non-headline results, yes there are ~7 rather than 1, so more coverage (sorry if felt like I was ignoring this in a unfair way). I feel like I don’t have much certainty on how CoT-free the other 6 tasks are (which my impression is y’all are also a bit uncertain of, given filter out). As I mentioned in the comment elsewhere with Raemon, I know models, especially of a certain era, were very prone to doing things like writing long in-line comments, and basically packing CoT into the middle of their answers.
I agree that the fact that the regressions with and without these tasks doesn’t change much is reassuring. I still think the questions around dataset influence seem important. I felt like one thing I learned when looking at this was an appreciation for how the tails of logistic regression matter less (counter to my intuitions on linear regression where tail points matter). So then it becomes about looking at what are the most influential datasets near the crossover point as I tried to loosely explore above, where datasets that are near the crossover point will most influence the results. I didn’t make a version of my plot using the non-short-answer sample yet to try to see measures of influence there. Thanks for the out-of-band exchange sharing the data if did later.
> It would be helpful if you could ask more precise questions about the rest :)
I don’t have a quick expansion to give here right now as direct questions (regarding thoughts on time horizon as a metric, how estimateable it is on things we care about, elicitation variations). Possibly might expand later, or happy to chat synchronously sometime.
--
Thanks for engaging with this! I like many aspects of the study, and hopefully discussion is useful.- ^
Sampling 32 tasks with replacement.
- ^
It seems pretty challenging to run a something like METR tasks in a non-CoT way. If you are doing tasks like software engineering that are generative (writing any code or agent loop, not just some form of labeling), it’s hard to stop the models from doing things that are essentially CoT. A trick some models[1] will do when writing code is write very verbose in-line comments where they are essentially packing CoT inside the code. Even if you somehow convince them to not write comments, can still do things like write a lot dead code. (It might be possible to control for this somehow though.)
The OP’s methodology sensibly fully side step this by excluding generative software tasks from the main results. One could possibly convert some of METR tasks into labeling-shaped tasks, but not like a straight rerun. Agree could make a version that would be very hard to saturate.
My current impression is the metrics in this study though seem partially saturated already / slightly broken. This is because in the >0.5hr regime there is only a single task on a spikey metric (SHADE-Arena Monitor TPR @ 1% FPR) which is already measuring greater than 50% TPR on frontier models (with massive error bars). Also a lot of the important tasks in the current trend lack actual human data to ground them. (This discuss this some in another other comment and MKodama’s comment). I might be missing something though...
A follow study / iteration which thinks more about good data covering the full spectrum of time seems like would be necessary to draw many conclusions that are trying to be drawn here. Plus future thinking on how timeable things we care about are.- ^
I can’t exactly point here, but loosely especially in the like approx Gemini 2 and GPT-4.1 era models that were RL-ed hard on a bunch of competitive programing problems without much code readability constraints.
- ^
Thinking about whether Fig 2 makes sense seems like a really good point.
When I was playing with what I could make of the data in a different comment I plotted the different tasks. One can observe that there is only a single task (SHADE-Arena Monitor TPR @ 1% FPR) is in this >0.5hr regime. I think this should feel pretty suspect. Anything you see on this right side of the figure 2 plots is coming entirely from just one task, with a metric that might think could be pretty spikey / non-linear. This one task absolutely does not represent the universe of >0.5hr imprtant tasks.
Seems pretty important, unless I’m confused where there are other tasks here.
not OP, but will note they intentionally focus the main results on tasks with short results. There presumably shouldn’t be a large difference based on number of tokens spent. That would be more about CoT, not a non-CoT measurement.
They do analysis around different sizes of open weight models in Appendix A.16. Not sure if you had a different question than what they address there.
Ah, thanks for checking and describing. I’d have to think about more to give better reference points, but seems decently good brier numbers (assuming the base rate is 50⁄50, and reference unskilled brier is 0.25). But also looks like RL is not doing much or generalizing, similar to the main results. Interesting.
Thanks for sharing and for chatting!
I will still mention to consider double checking the fig 1 base model x axis. Seems just too perfect. Either a spooky coincidence, but then would be curious if all of the base → SFT models manage to exactly balance out a reduction in input tokens with an increase in output tokens (which would be an interesting phenomenon), or it’s a bug/hallucinated hard coded value with a simple fix.
Edit: actually just noticed the cost numbers between Table 4 and Fig 1 differ (in Qwen3.5-27B SFT+RL reads as 2.775/1k evals in table, vs >3/1k evals in Fig 1). Not huge differences, but something is going on. Maybe I’m just missing something though...
reward: Being a great country with high quality of life for it’s ruling elites
feels like this is more of a “goal” than a reward, right? Maybe semantically equivalent, but probably not, and matters when distinguishing between alignment failures and specific things that happen during RL.
If we consider classical reward hacking cases (like say RL CoastRunners example), there seems to be distinctions. I can look at the video of boat spinning in circles and be like “wow, it gets a really high score, also that’s definitely a hack or not the intended way to get that high score” in a way I can’t in this Worlds Fare example. Sure we can look at press being impressed by those pavilions, but was that hack, and can I point to reward function that got pushed really high? Sort of, but less so.
It seems useful to keep “reward hacking” distinct for those special cases like the CoastRunners example, without expanding it to any kind of alignment failure. I’m arguing this a lot on vibes. Making a formalism of what is reward hacking vs other alignment failures seems maybe doable (and perhaps people have), but I did not find something I liked in a quick search, and I don’t attempt with formalisms here.
After quick background search though, I will link to this recent post “Confusion around the term reward hacking” (Azarbal, 2026). There might be field-wide muddling.
Interesting to see. Some things I wonder about, (1) which tasks most influence the results, (2) how much time numbers make sense, and (3) general elicitation questions
Which tasks and kinds of tasks are most influencing the headline?
With METR’s timeline plots there has been discussion here on LW and elsewhere about if certain items have a large influence on the results. I was wondering which kinds of tasks most explain the recent doublings already, and what might worlds look like where they doubled again according to this methodology.I sort of attempted to vibe very rough versions of this analysis. It’s hard without the underlying data, but can get claude to try to extract accuracy numbers from Tables 7-13 and pixel peep Fig 47 to get medians and IQRs for the human-time data.[1]
Things in the ~1-3min band are most likely to be near the 50% crossover point and I think will have more influence on the regression. A few outliers like the “vibe-coding sabotage” on the Y axis and the SHADE monitor on the X axis might deserve some consideration. These are both use a kind of TPR @ ~1-2% FPR metric, different than some other datasets. Unclear effects here.This is vibey though and would want access to underlying data to do more confident analysis.
There’s also other ways of quantify the influence of each dataset (where the “vibe-coding sabaotage” task again shows up as a clear outlier with more than twice as much influence as any other dataset). I hide it in a collapsible though as I’m not super confident in methodology.Vibey can we calculate an influence score?
Claude’s explanation:
The bars estimate how much each benchmark plausibly contributed to the headline time-horizon gain between GPT-4o and GPT-5.5. The headline metric is the 50% crossover of a logistic curve fit to success vs. log human solve time, and a basic property of logistic fits is that an observation’s influence on the fitted curve scales with p(1−p) — maximal for tasks the model passes about half the time, near zero for tasks it almost always passes or almost always fails. Since the crossover moved from roughly 0.9 to 3 minutes between these two models, accuracy gains on tasks in that time range supply the most evidence about where the crossover sits. Each bar is therefore the benchmark’s accuracy change (chance-corrected, approximately following the paper’s normalization — exactly for MCQ tasks, while the detection tasks’ ~1.5% baseline was treated as 0, a ~1pp difference) multiplied by a p(1−p) weight centered at ~1.6 minutes and about a decade wide in log time, averaged over the benchmark’s per-question human-time distribution as extracted from the paper’s Figure 47. The whiskers show how that weight varies across the middle 50% of each benchmark’s tasks. This is a heuristic approximation, not a refit of the paper’s actual estimator: it captures the main channel (gains near the crossover move the crossover) but ignores how off-crossover gains rotate the curve’s slope, and ignores the anchoring effect of long tasks still near 0%, which constrain how far the crossover can move. The ranking at the top is stable across reasonable choices of kernel width, but the precise values shouldn’t be taken seriously — a leave-one-benchmark-out refit on the underlying data would be the right version of this analysis.
Time numbers
In general I have mixed thoughts on human time estimates as a metric in some cases, and how well time-horizon can approximate what I’d be most concerned about models packing into a forward pass[2]
It seems worth reflecting on how much trust to put on time-horizon metrics if we often can’t actually get good human-time numbers in the first place. One particular consideration is many of the most influential datasets in the regression don’t have actual human time measurements, and instead come from this process of using Claude to estimate human time. I think I’d believe claude can make good guesses here (and have numbers here for the tested tasks), but still seems slightly worth considering . The authors are thorough in this paper, and I eventually noticed Fig 19 in appendix which gives the regression only looking at cases with true numbers. The double time goes from 373 [167,691] days → 349 [166, 1883] days. So the point estimate is surprisingly stable, but a confidence interval from ~6 months to 5 years is substantial.
Again, I like the ideas here and the thoroughness of the paper, but just bringing up general consideration given how much attention and extrapolation METR’s original time horizon plot got. I might eventually form a clearer view here and write up more thoughts…
Edit: see also the discussion on MKodama’s comment about the logistic plots. The fact there is only single task in the >0.5hr regime looks pretty problematic for much of the main analysis (eg, for Fig 2 here / fig 5 paper). If going to make some of the claims above there really should be either more tasks in this time regime, or this part of the plots should be excluded or better indicate it’s just a single one task. I haven’t thought as much about the o3-mini anchor, but a different possibly reasonable angle.What’s going on with few shot?
Language models are always sensitive to how you prompt them. This is especially true for earlier ones. In the headline regression GPT-2 and GPT-3 are excluded. This might be reasonable, but also worth considering aspects like prompt construction. GPT-3 was quite a smart model, it just was not “ChatGPT 100M user product-ready” because it required more care when prompting. Section 3 discusses some about how few-shot is used to elicit the non-CoT format. It’s unclear how many few-shot examples are used or if this is typically enough for GPT-3-type base models. This question doesn’t seem core to the study, but maybe a separate question would study a version of this data with uniform like 10-shot in every dataset or something.
--
Thanks for all the work on exploring these models and sharing findings.- ^
btw, I think the caption on fig 47 might be off, since it says IQR, but appears to show all points? below whiskers assumes taking only an approximate middle slice of fig 47 points
- ^
stuff like Choi et al’s measurements on the models making inferences of user attributes. These don’t have clean mappings to how long I think a human would take to perform, but still fit concerning categories of “intuition, not reasoning” aspects that happen in a forward pass
- ^
This is an interesting story, but feel like it’s somewhat overloading or misapplying the term “reward hacking”. From my impression of how it is traditionally defined, “reward hacking” happens when a system achieves very high reward in an unexpected way (with possible other negative effects). Not every case of systems failing or bad things happening is reward hacking. In this case of this 1937 worlds fair example it’s not clear what the reward that was hacked was, who are the agents (the nation states?), and to the extent there is a reward, it’s not clear a lot of it was achieved.
If we wanted to force in alignment jargon into this story, maybe we could apply some version of an outer alignment failure (where the average citizen failed to align their government into doing something that improved the average citizen’s utility)? Seems shaky though..
World’s fairs are cool though, and a fun story. Also, maybe people take broader definitions of “reward hacking” than what I have in mind.
Thanks for the reply. I think makes sense on Fig 1.
On 2, seems to also make sense that if you have a powerful teacher, most of the gains there are from imitation learning there. Specifically my little comment about RL and brier was about the case of RL-after-SFT. Basically even if AUC doesn’t move much, do you at least get some calibration gains that might make the monitor score more interpretable? Not really key in any way, but just to clarify the point as being distinct from the RL-only reward measurement.
On 3. This is interesting. I actually would have initially guessed models like GPT-5.4 would average higher than 870 output tokens on “high” settings when counting their hidden reasoning, but I don’t think i have good intuitions here. Also interesting if that the reduction in output tokens exactly balances out the reduction in input tokens. Fig 1 between Qwen3 Base and Qwen 3 SFT+RL looks essentially pixel perfect aligned on the x axis, which seems a bit odd given this.
Thanks for sharing the work. We read it at the FAR Labs reading group today and found it interesting. Some slightly disorganized notes of some things we discussed:
We were a little confused why the base models weren’t on Fig 1 or reported elsewhere (only Qwen has the before and after)? We eventually realized this was partially extractable from Fig 7a, but were wondering if you measured the base models of the ones you SFT-ed to give a sense of how much the whole process helps.
We discussed various interpretations of the RL experiments not changing much. We weren’t sure if this is just normal for other tasks in this ~8k example regime (that SFT mostly helps in this dataset size regime), or if this was just a “well RL’s pretty finicky” and if tweaked it more might see other things, or if an example of how RL mostly only matters if training closer to frontier models where don’t have more powerful teacher examples.
It looks like the RL reward has a brier-score-like term. It would be interesting to measure brier score see if the 1-10 scores became more calibrated, even if AUCROC didn’t change much.
We discussed some how much we thought the various cost spread came from underlying model cost per token vs how verbose they were at this task. We eventually saw Table 4 and thought it was interesting. In particular, I was very surprised how much spread in input token count there was. I expected output token count spread (and indeed there’s about 9x spread), but was surprised by the 2x spread in input token count. Later realized this was probably due to stripping out the prompt tokens after SFT, but then was confused why in Fig 1 base model cost is shown as the same as the SFT+RL one (unless shorter input exactly balanced out longer output?).
It wasn’t clear the reasoning effort used for the models that allow a reasoning effort setting.
There’s probably some interesting things to go deeper on. For example, selecting SFT traces that also skew sorter, or including the cost in the RL. Also baseline comparison to very short / non-CoT detectors to see how much the “deliberative” step adds. Also it was noteworthy how Gemini 2.5 Pro, which is the source of most of the teacher data, is one of the most verbose models, and unclear the effect of this. It is surprising the SFT-ed models did not end up becoming more much verbose to be closer to Gemini 2.5.
Overall the general result of ~”if you SFT a smaller LLM to do a task you care about it gets better” is kind of expected. Still interesting to see the work and analysis here. Thanks!
Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs
Thanks for the investigation and sharing! The explorations on the reasoning models is seems new and a good extension.
Regarding the non-reasoning results, one might speculate that the multi-token prediction part of the architecture[1] could influence some of the anomalous token behavior. Tokens that are almost always bigram continuations (eg, “eredWriter”, “reeNode”, “VERTISEMENT”) likely almost always are predicted by the two-ahead predictor. Thus the model might get further confused when it must try to generate these tokens via the next token predictor. We might also speculate there are ways two-ahead predictors could increase the risk of ending up in repeating basins.
It would be interested to explore this more quantitatively on how/if the deepseek architecture differs. Your preliminary work is valuable, but I don’t actually gives evidence this model has more or less anomalous token behavior than say GPT or LLaMa. Also, if there are differences, it could be architectural, or just quirks of the training.- ^
Ege Edril has a nice short summary of this architecture component for someone unfamiliar
- ^
+1 here for the idea around how the models must commit to a URL once it starts, and that it can’t naturally cut off after starting. Presumably though the aspiration is that these reasoning/CoT-trained models could reflect back on the just completed URL and guess whether that is likely to be a real URL or not. If it’s not doing this check step, this might be a gap in the learned skills, more than intentional deception.
In particular, in the five tasks (MMLU, MATH, BIG-Bench, Natural2Code, WMT23) where they report going to the GPT-4 API, they report an average of ~1 point improvement. This experiment setting seems comparable, and not evidence they are underperforming GPT-4.
However, all these settings are different from how ChatGPT-like systems are mostly being used (where mostly zero-shot). So difficult to judge the success of their instruction-tuning for use in this setting.
(apologies if this point posted twice. Lesswrong was showing errors when tried to post.)
Hi Dewi, thanks for the reply. This is interesting.
Sorry I realized I messed up in my top level comment where I incorrectly vibed the subselection of datasets used (I don’t think it was specified in the paper which 32 datasets are included or not, and Claude assumed in a way that looked reasonable enough but was off on a few). Sorry about that. I’m glad to your new figure exists.
Note though, ideally we’d want to ablate SHADE CoT and SHADE Action together (and maybe the two a-level datasets?) as these are very related tasks (I think?). Keeping them separate for like SHADE monitor makes it look like there are two unique tasks in this high end regime.
I attempted to make a new version tweaked version of my plot using the jsons you sent.
Things with large LOO Δ above generally have either have a tall change in this lollipop plot or are important parts of the tails (eg, arithmetic is important for flushing out the low end. Removing drops that whole area closer to the “tower of london” number).
Things in this center area would have to greatly change in worlds of the next measured doubling.
This is not including the possible extra expansion on the ICR due to the variance in the estimates them self. I’m not currently sure how to capture this. Some things like a-level MCQ has no IQR I think because all labeled at 1min.
> but as you can see these don’t materially change the conclusions we landed on around doubling time trends and associated uncertainty
Right, this does seem pretty robust to various slicing of this particular dataset of tasks. I wanted to start the thread with intuition building of what that particular set is and what’s changing. I feel personally now have a bit better sense of this.
The “doesn’t materially change” part should make sure we are truely internalize the associated uncertainty ranging from months to years. Similarly in the MKodama’s comment when looking at the logistic regression plots on the capabilities of a given model make sure we appreciate the error bars.
It still feels like we currently have a pretty confused signal for a lot of this span, especially in the upper half. The logistic fit will be rough. This is also already communicated by the individual estimates on the frontier models spanning 2 OOM from ~20 seconds to ~1 hour, so seems good as long as people internalize that to the extent.
All things to think about as the community thinks about for future work in the area, as well as making sure the community doesn’t over interpret the lowest nuance “373 day doubling time currently at 3min” version. The paper/post conclusion points around models capable of twenty-five minutes of no-CoT reasoning by 2030 are going need different methodology to measure (there’s not task diversity in this regime). Plus thinking on how well time horizon is a way to think about things we care about here.