Very complicated ways of facilitating agency seem feasible. There’s Imbue doing some CoEm-sounding things (debuggable planning in natural language that’s possible to inspect and intervene on), without a clear stance on extinction risk implications. It might turn out that there is a whole operating system’s worth of engineering effort that’s useful for turning basic LLM-style capabilities into coherent autonomous reasoning.
This is mostly irrelevant for capabilities if scaling gets to AGI on its own, but if that doesn’t happen in the next few years, extremely complicated agency engineering efforts might become more important than further scaling, giving CoEm-like or even CAIS-like systems an opportunity to determine the safety properties of first AGIs.
Very complicated ways of facilitating agency seem feasible. There’s Imbue doing some CoEm-sounding things (debuggable planning in natural language that’s possible to inspect and intervene on), without a clear stance on extinction risk implications. It might turn out that there is a whole operating system’s worth of engineering effort that’s useful for turning basic LLM-style capabilities into coherent autonomous reasoning.
This is mostly irrelevant for capabilities if scaling gets to AGI on its own, but if that doesn’t happen in the next few years, extremely complicated agency engineering efforts might become more important than further scaling, giving CoEm-like or even CAIS-like systems an opportunity to determine the safety properties of first AGIs.