We both think general-purpose long-horizon agency (my term) / staying-on-track-across-large-inferential-distances (your term, maybe not equivalent to mine but at least heavily correlated with mine?)
They’re equivalent, I think. “Staying on track across inferential distances” is a phrasing I use to convey a more gears-level mental picture of what I think is going on, but I’d term the external behavior associated with it “general-purpose long-horizon agency” as well.
Correct?
Basically, yes. I do expect some transfer learning from RL, but I expect it’d still only lead to a “fixed-horizon” agency, and may end up more brittle than people hope.
To draw an analogy: Intuitively, I would’ve expected reasoning models to grok some simple compact “reasoning algorithm” that would’ve let them productively reason for arbitrarily long. Instead, they seem to end up with a fixed “reasoning horizon”, and scaling o1 → o3 → … is required to extend it.
I expect the same of “agent” models. With more training, they’d be able to operate on ever-so-slightly longer horizons. But extending the horizon would require steeply (exponentially?) growing amounts of compute, and the models would never quite grok the “compact generator” of arbitrary-horizon arbitrary-domain agency.
They’re equivalent, I think. “Staying on track across inferential distances” is a phrasing I use to convey a more gears-level mental picture of what I think is going on, but I’d term the external behavior associated with it “general-purpose long-horizon agency” as well.
Basically, yes. I do expect some transfer learning from RL, but I expect it’d still only lead to a “fixed-horizon” agency, and may end up more brittle than people hope.
To draw an analogy: Intuitively, I would’ve expected reasoning models to grok some simple compact “reasoning algorithm” that would’ve let them productively reason for arbitrarily long. Instead, they seem to end up with a fixed “reasoning horizon”, and scaling o1 → o3 → … is required to extend it.
I expect the same of “agent” models. With more training, they’d be able to operate on ever-so-slightly longer horizons. But extending the horizon would require steeply (exponentially?) growing amounts of compute, and the models would never quite grok the “compact generator” of arbitrary-horizon arbitrary-domain agency.