Francis Rhys Ward
Karma: 883
Perspectives on Continual Learning: Survey Results and Forecasts
Quick response:
> (1) which tasks most influence the resultsThe mainline trends and uncertainty estimates are computed from bootstraps which resample many times over different benchmarks, meaning that the analysis is robust to leaving out any particular (subset of) benchmarks. This is true for both the main doubling trends and the individual model TH estimates. From the paper:
>The fact there is only single task in the >0.5hr regime looks pretty problematic
We have lots of tasks that are >0.5 hr. In the main trend we only include a subset of (mostly shorter) tasks, but including all tasks doesn’t change the trend that much (the overall doubling times reduces a bit because frontier models can do some long horizon SWE tasks without CoT, e.g., just outputting tool calls at each step).
From the paper:
> Figure 22: Comparison of TH trend lines using all tasks including short-answer, generation, and multi-turn agentic. The most significant impact of including the multi-turn agentic tasks is an increase in the point-estimate THs for the latest frontier models which perform very well on these tasks.It would be helpful if you could ask more precise questions about the rest :)
Angles of attack for continual learning safety
We don’t have privileged access from OpenAI. Similar to METR, we use the closest available public models to estimate the capabilities of models that are no longer publicly available.
How might continual learning affect safety and alignment?
What’s Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
Implications of Continual Learning for LLM Agents: Introduction
>I think ablations/knockouts (e.g. helpful-only models, RLVR-only models, models without X piece of post-training) should also be counted here.
I would count these as “natural”—where the definitive feature is to understand training pipelines and their safety properties or failure modes.
One difference is: Worst-case MOs are supposed to upper-bound the difficulty of some problem, like eliciting hidden goals, they need not exhibit super realistic behaviours or mechanisms. Constructed MOs are supposed to behave similarly to the real-life case so you can learn about the real situation, but they need not be a difficult case for safety measures.
Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models
Please interpret me as saying that the hope with the methodology for worst-case MOs is that we can have good reason to believe that the problem is strictly harder than the real case, rather than the methodology itself being to cross your fingers and hope that the MO is strictly harder without good reason.
Three types of model organism
Bayesian epistemology typically works in the framework of an existing hypothesis space, with a prior over that space, which is then updated. In addition to updating your credences about the possibilities in the space, you can also reformulate your hypothesis space itself, e.g., because you become aware of new possibilities (like the existence of scammers), or because you want to carve the world into different concepts due to some ontological shift. I think the Bayesian should just be allowed to reformulate their hypothesis space and reform their prior to get out of this.
[Paper] How does information access affect LLM monitors’ ability to detect sabotage?
The Elicitation Game: Evaluating capability elicitation techniques
Why care about AI personhood?
[Paper] AI Sandbagging: Language Models can Strategically Underperform on Evaluations
Nathan’s suggestion is that adding noise to a sandbagging model might increase performance, rather than decrease it as usual for a non-sandbagging model. It’s an interesting idea!
We do this because it’s a sharper distinct meaning of no-CoT, i.e., in the main plot we restrict to tasks which only require very few forward passes. See the paper and above comments showing that including longer tasks, including generation and agent long-horizon tasks, doesn’t change the trends that much.