However, I’m split on whether LLM-based agents, if and when they start working well, could make this a reality. There are a few reasons why I was afraid, and I’m less afraid now, which I will go over in this post. Note that this doesn’t tell us anything about the chance of loss of control from non-LLM (or vastly improved LLM—italics mine—S.K.) agents, such as the brain in a box in a basement scenario. The latter is now a large source of my p(doom) probability mass.
To me this looks like the “no true Scotsman” fallacy. The AI-2027 forecast relies on Agents since Agent-3 thinking in neuralese and becoming undetectably misaligned. Similarly, LLM agents’ intelligence arguably didn’t max out at “approximately smart human level”, it will max out at a spiked capabilities profile which is likely already known to include the ability to receive a gold medal on the IMO 2025 and to assist with research, but not known to include the ability to handle long contexts as well as humans.
Regarding the context, I made twocomments explaining my reasons to think that current-state LLMs don’thave enough attention to handle long contexts. The context is in fact distilled into a vector of few numbers, then the vector is processed and the next token is added wherever the LLM decides.
I agree AI intelligence is and likely will remain spiky and some spikes are above human-level (of course a calculator also spikes above human-level). But I’m as of yet not convinced that the whole LLM-based intelligence spectrum will max out above takeover-level. But I’d be open for arguments.
To me this looks like the “no true Scotsman” fallacy. The AI-2027 forecast relies on Agents since Agent-3 thinking in neuralese and becoming undetectably misaligned. Similarly, LLM agents’ intelligence arguably didn’t max out at “approximately smart human level”, it will max out at a spiked capabilities profile which is likely already known to include the ability to receive a gold medal on the IMO 2025 and to assist with research, but not known to include the ability to handle long contexts as well as humans.
Regarding the context, I made two comments explaining my reasons to think that current-state LLMs don’t have enough attention to handle long contexts. The context is in fact distilled into a vector of few numbers, then the vector is processed and the next token is added wherever the LLM decides.
I agree AI intelligence is and likely will remain spiky and some spikes are above human-level (of course a calculator also spikes above human-level). But I’m as of yet not convinced that the whole LLM-based intelligence spectrum will max out above takeover-level. But I’d be open for arguments.