This one. The argument on LW goes as “you can’t define distinction between tool and agent, so we’re right”.
Now, to those with knowledge of the field, it is akin to some supposedly engineer claiming you can’t define distinction between a bolt and a screw, as a way to defy the statement that “you can avoid splitting the brittle wood if you drill a hole and use a bolt, rather than use a screw”, which was a rebuttal to “a threaded fastener would split the brittle wood piece”. The only things it demonstrates is ignorance, incompetence, and lack of work towards actually fulfilling the stated mission.
For this ‘oracle’ link, it clearly illustrates the mechanism of generation of strings employed to talk about the AI risks. You start with the scary idea, then you progress to necessity for each type of AI to be shown scary, then you proceed to each subtype, then you make more and more detailed strings designed to approximate the strings that result from entirely different process of starting from basics (and study of the field) and proceeding upwards to risk estimate.
That task is aided by fact that it is impossible to define a provably safe AI in English (or in technobabble) due to vagueness/ambiguousity, and due to fact that language predominantly attributes real world desires when describing anything that seems animate. That is, when you have a system that takes in sequences and generates functions that approximate the sequences (thus allowing prediction of next element in sequences, without over-training on noise), you can describe it as predictor in English and now you got ‘implicit’ goal of changing the world to match the prediction. Followed by “everyone give us money or it is going to kill us all, we’re the only ones whom understand this implied desire! [everyone else’s more wrong because we call ourselves less wrong seem to be implied]”. Speaking of which use of language is a powerful irrationality technique.
Meanwhile, in the practice, such stuff is not only not implicit, it is incredibly difficult to implement even if you wanted to implement it. Ultimately, many of the ‘implied’ qualities that are very hard to avoid in English descriptions of AI are, also, incredibly difficult to introduce when programming. We have predictor-type algorithms, which can be strongly superhuman if given enough computing power—and none of them would exhibit a trace of ‘implicit’ desire to change the world.
There’s the notion that anything which doesn’t ‘understand’ your implied, is not powerful enough (not powerful enough for what?), that’s just rationalization, and is not otherwise substantiated or even defined. Or even relevant. Let’s make example in other field. Clearly, any space propulsion we know is possible, is not powerful enough to get to faster than speed of light. Very true. Shouldn’t be used to imply that we’ll have faster than light travel.
This one. The argument on LW goes as “you can’t define distinction between tool and agent, so we’re right”.
Now, to those with knowledge of the field, it is akin to some supposedly engineer claiming you can’t define distinction between a bolt and a screw, as a way to defy the statement that “you can avoid splitting the brittle wood if you drill a hole and use a bolt, rather than use a screw”, which was a rebuttal to “a threaded fastener would split the brittle wood piece”. The only things it demonstrates is ignorance, incompetence, and lack of work towards actually fulfilling the stated mission.
For this ‘oracle’ link, it clearly illustrates the mechanism of generation of strings employed to talk about the AI risks. You start with the scary idea, then you progress to necessity for each type of AI to be shown scary, then you proceed to each subtype, then you make more and more detailed strings designed to approximate the strings that result from entirely different process of starting from basics (and study of the field) and proceeding upwards to risk estimate.
That task is aided by fact that it is impossible to define a provably safe AI in English (or in technobabble) due to vagueness/ambiguousity, and due to fact that language predominantly attributes real world desires when describing anything that seems animate. That is, when you have a system that takes in sequences and generates functions that approximate the sequences (thus allowing prediction of next element in sequences, without over-training on noise), you can describe it as predictor in English and now you got ‘implicit’ goal of changing the world to match the prediction. Followed by “everyone give us money or it is going to kill us all, we’re the only ones whom understand this implied desire! [everyone else’s more wrong because we call ourselves less wrong seem to be implied]”. Speaking of which use of language is a powerful irrationality technique.
Meanwhile, in the practice, such stuff is not only not implicit, it is incredibly difficult to implement even if you wanted to implement it. Ultimately, many of the ‘implied’ qualities that are very hard to avoid in English descriptions of AI are, also, incredibly difficult to introduce when programming. We have predictor-type algorithms, which can be strongly superhuman if given enough computing power—and none of them would exhibit a trace of ‘implicit’ desire to change the world.
There’s the notion that anything which doesn’t ‘understand’ your implied, is not powerful enough (not powerful enough for what?), that’s just rationalization, and is not otherwise substantiated or even defined. Or even relevant. Let’s make example in other field. Clearly, any space propulsion we know is possible, is not powerful enough to get to faster than speed of light. Very true. Shouldn’t be used to imply that we’ll have faster than light travel.