Thanks. (I have read this with interest, but don’t have much of an overall reply.) A couple comments:
Humanity should count as actual AGI, but it remains slow.
It’s a weird middle ground, where it is fooming but pretty slowly. (I mean, as a human, I would think that, i.e. I would experience time on a scale that’s somewhat faster than humanity’s foom.) And it kinda only half has the A in AGI. If it fully had the A (or I guess, ISGI, in silico general intelligence) then it would foom much faster.
Similarly, I think LLMs that do prosaic RSI of automatically building the next model (crucially including formulation of new RL tasks/environments/graders) will be actual AGI in the sense that they can (on their own, without humanity’s input) eventually generate and accumulate as much crystallized intelligence as humanity would.
Maybe kinda. But my guess is that this would be kind of like calling [hominid evolution by natural selection on genetic variation] an AGI, or calling [the ecosystem of the Earth throughout all of time] an AGI. I mean it would probably be extremely faster & scarier than evolution, but still.
Pretraining very sample inefficiently reconstructs cognitive skills that left evidence about their nature in the text.
It reconstructs a lot of them to a significant extent, but very much misses a lot of them to a significant extent, I think.
But this is very likely enough for a base model of 2031 to be sufficiently teachable (using RL tasks) to end up learning how to formulate new RL tasks/environments/graders on its own, starting the slow-learning prosaic RSI process of building the next model automatically.
This seems plausible-ish. I would weakly expect a lot of plateaus, e.g. due to highly correlated taste, but not strongly and I haven’t thought about it much. (Maybe you’re pricing that in to “slow”.)
Thanks. (I have read this with interest, but don’t have much of an overall reply.) A couple comments:
It’s a weird middle ground, where it is fooming but pretty slowly. (I mean, as a human, I would think that, i.e. I would experience time on a scale that’s somewhat faster than humanity’s foom.) And it kinda only half has the A in AGI. If it fully had the A (or I guess, ISGI, in silico general intelligence) then it would foom much faster.
Maybe kinda. But my guess is that this would be kind of like calling [hominid evolution by natural selection on genetic variation] an AGI, or calling [the ecosystem of the Earth throughout all of time] an AGI. I mean it would probably be extremely faster & scarier than evolution, but still.
It reconstructs a lot of them to a significant extent, but very much misses a lot of them to a significant extent, I think.
This seems plausible-ish. I would weakly expect a lot of plateaus, e.g. due to highly correlated taste, but not strongly and I haven’t thought about it much. (Maybe you’re pricing that in to “slow”.)