One key point here is reminiscent of Amdahl’s law in computing: if you are trying to make some computation faster by parallelizing it, and a fraction x of it is stuff you don’t have a way to parallelize, then no amount of extra hardware will make the speedup bigger than 1/(1-x).
Similarly, if you are trying to make some discovery / invention / optimization faster by doing it more intelligently, and a fraction x of the work is something other than pure thinking, then no amount of better thinking will make the speedup bigger than 1/(1-x).
I think the usual rejoinder on the “AI go foom” side is that we are likely to overestimate x by underestimating what really effective thinking can do; Eliezer’s story “That Alien Message” is intended as an intuition pump in that direction. (Along with his non-fictional comments in the same post, suggesting e.g. that something sufficiently smart looking at webcam footage of a falling apple would be contemplating GR as a plausible hypothesis by the third frame.) I don’t find that my own intuition is very compellingly pumped by this, for what it’s worth.
I think the usual rejoinder on the “AI go foom” side is that we are likely to overestimate x by underestimating what really effective thinking can do
Well, yeah, and on the whole, it’s the kind of assumption that one can’t scientifically prove or disprove. It’s something that can’t be observed yet and that we’ll see play out (hopefully) this century.
I guess the main issue I see with this stance is not that it’s unfounded, but that its likely cause is something like <childhood indoctrination to hold having good grade, analytical thinking, etc as the highest value in life>, as in, it would perfectly explain why it seems to be readily believed by anyone that stumbles upon less wrong, whereas few/no other beliefs (that don’t have a real-world observation to prove/disprove them) are so widely shared here (as well as more generally in a lot of nerdy communities).
Granted, I can’t “prove” this one way or another, but I think it helps to have some frameworks of thinking that are able to persuade people that start from an “intelligence is supreme” perspective towards the centre, much like the alien story might persuade people that start from an “intelligence can’t accomplish much” perspective.
One key point here is reminiscent of Amdahl’s law in computing: if you are trying to make some computation faster by parallelizing it, and a fraction x of it is stuff you don’t have a way to parallelize, then no amount of extra hardware will make the speedup bigger than 1/(1-x).
Similarly, if you are trying to make some discovery / invention / optimization faster by doing it more intelligently, and a fraction x of the work is something other than pure thinking, then no amount of better thinking will make the speedup bigger than 1/(1-x).
I think the usual rejoinder on the “AI go foom” side is that we are likely to overestimate x by underestimating what really effective thinking can do; Eliezer’s story “That Alien Message” is intended as an intuition pump in that direction. (Along with his non-fictional comments in the same post, suggesting e.g. that something sufficiently smart looking at webcam footage of a falling apple would be contemplating GR as a plausible hypothesis by the third frame.) I don’t find that my own intuition is very compellingly pumped by this, for what it’s worth.
Well, yeah, and on the whole, it’s the kind of assumption that one can’t scientifically prove or disprove. It’s something that can’t be observed yet and that we’ll see play out (hopefully) this century.
I guess the main issue I see with this stance is not that it’s unfounded, but that its likely cause is something like <childhood indoctrination to hold having good grade, analytical thinking, etc as the highest value in life>, as in, it would perfectly explain why it seems to be readily believed by anyone that stumbles upon less wrong, whereas few/no other beliefs (that don’t have a real-world observation to prove/disprove them) are so widely shared here (as well as more generally in a lot of nerdy communities).
Granted, I can’t “prove” this one way or another, but I think it helps to have some frameworks of thinking that are able to persuade people that start from an “intelligence is supreme” perspective towards the centre, much like the alien story might persuade people that start from an “intelligence can’t accomplish much” perspective.