Model + scaffolding provides a good source of uplift compared to model only and this continues to be true
Any uplift that comes via scaffolding can be in principle rolled directly into the model weights
My cruxes are:
What are the capabilities advantages of keeping things in the scaffolding rather than pushing them into the weights? I could imagine that keeping things as tool calls is good for interpretability reasons. But I also could imagine that all else being equal, putting it in the weights makes it a more flexible / general capability.
I also expect that there will be substantial leakage, i.e. capabilities are not cleanly described as (model-only) or (scaffolding-only), but rather will be split across both.
My rough takes are like this:
Model + scaffolding provides a good source of uplift compared to model only and this continues to be true
Any uplift that comes via scaffolding can be in principle rolled directly into the model weights
My cruxes are:
What are the capabilities advantages of keeping things in the scaffolding rather than pushing them into the weights? I could imagine that keeping things as tool calls is good for interpretability reasons. But I also could imagine that all else being equal, putting it in the weights makes it a more flexible / general capability.
I also expect that there will be substantial leakage, i.e. capabilities are not cleanly described as (model-only) or (scaffolding-only), but rather will be split across both.