One oddity that stands out is Yudkowsky and Soares ongoing contempt for large language models and hypothetical agents based on them. Again for a book which is explicitly premised on the idea that urgent action is necessary because AI might become superintelligent in just a few years it is bizarre that the authors don’t feel comfortable making more reference to the particulars of the existing AI systems which hypothetical near-future agents would be based on.
Except that non-LLM AI agents have yet to be ruled out. Quoting Otto Barten,
Note that this doesn’t tell us anything about the chance of loss of control from non-LLM (or vastly improved LLM (sic! -- S.K.)) agents, such as the brain in a box in a basement scenario. The latter is now a large source of my p(doom) probability mass.
Alas, as I remarked in a comment, Barten’s mentioning of vastly improved LLM agents makes Barten’s optimism resemble the “No True Scotsman” fallacy.
Okay but they don’t have to be ruled out for you to say things like “One way this could work is bla bla bla” and have that be sane in the context of what already exists. Again if you think something is a near term concern it’s not unreasonable to make reference to how the existing things could evolve in the near future. I think what’s actually going on here is that rather than non-LLM agents being “not ruled out” (which I agree with, they are by no means ruled out) Yudkowsky and Soares find LLM agents an implausible architecture but don’t want to say this explicitly because they think saying that too loudly would speed up timelines. I think they’re actually wrong about the viability of LLM agents, but it does contribute to a sort of odd abstract tone it otherwise would have less of.
Except that non-LLM AI agents have yet to be ruled out. Quoting Otto Barten,
Alas, as I remarked in a comment, Barten’s mentioning of vastly improved LLM agents makes Barten’s optimism resemble the “No True Scotsman” fallacy.
Okay but they don’t have to be ruled out for you to say things like “One way this could work is bla bla bla” and have that be sane in the context of what already exists. Again if you think something is a near term concern it’s not unreasonable to make reference to how the existing things could evolve in the near future. I think what’s actually going on here is that rather than non-LLM agents being “not ruled out” (which I agree with, they are by no means ruled out) Yudkowsky and Soares find LLM agents an implausible architecture but don’t want to say this explicitly because they think saying that too loudly would speed up timelines. I think they’re actually wrong about the viability of LLM agents, but it does contribute to a sort of odd abstract tone it otherwise would have less of.