There’s so much that can go wrong with such reasoning, given that intelligence (even at the size of a galaxy of Dyson spheres) is not a perfect God, as to render such arguments irrelevant and entirely worthless. Furthermore there’s enough ways how the non-orthogonality can hold, such as e.g. almost all intelligences with wrong moral systems crashing or failing to improve, that are not covered by ‘converges’.
meta: Tendency to talk seriously about products of very bad reasoning really puts an upper bracket on the sanity of newcomers to LW. As is the idea that very bad argument trumps authority (when it comes to the whole topic).
Would you take criticism if it is not ‘positive’ and doesn’t give you alternative method to use for talking about same topic? Faulty reasoning has unlimited domain of application—you can ‘reason’ about purpose of the universe, number of angels that fit on a tip of a pin, of what superintelligences would do, etc. In those areas, non-faulty reasoning can not compete in terms of providing a sort of pleasure from reasoning, or in terms of interesting sounding ‘results’ that can be obtained with little effort and knowledge.
You can reason what particular cognitive architecture can do on a given task given N operations; you can reason what the best computational process can do in N operations. But that will involve actually using mathematics, and results will not be useful for unintelligent debates in the way in which your original statement is useful (I imagine you could use it to reply to someone who believes in absolute morality, as a soundbite; i really don’t see how it could have any predictive power what so ever about the superintelligence though).
I am interested in anything that allows better reasoning about these topics.
Mathematics has a somewhat limited use when discussing the orthogonality thesis. AIXI, and some calculations about the strength of optimisation processes and stuff like that. But when answering the question “is it likely that humans will build AIs with certain types of goals”, we need to look beyond mathematics.
I won’t pretend the argument in this post is strong—it’s just, to use the technical term, “kinda neat” and I’d never seen it presented this way before.
What would you consider reasonable reasoning on questions like the orthogonality thesis in practice?
That’s how religions were created, you know—they could not actually answer why lightning is thundering, why sun is moving through the sky, etc. So they did look way ‘beyond’ the non-faulty reasoning, in search for answers now (being inpatient), and got answers that were much much worse than no answers at all. I feel LW is doing precisely same thing with AIs. Ultimately, when you can’t compute the right answer in the given time, you will either have no answer or compute a wrong one.
On the orthogonality thesis, it is the case that you can’t answer this question given limited knowledge and time (got to know AI’s architecture first), and any reasonable reasoning tells you this, while LW’s pseudo-rationality keeps giving you wrong answers (that aren’t any less wrong than anyone else including the mormon church and any other weird religious group), I don’t quite sure what you guys are doing wrong; maybe the focus on biases and conflation of biases with stupidity did lead to a fallacy that lack of (known) biases will lead to non stupidity, i.e. smartness, and if only you won’t be biased you’ll have a good answer. It doesn’t work like this. It leads to another wrongness.
It was definitely important to make animals come, or to make it rain, tens thousands years ago. I’m getting a feeling that as I tell you that your rain making method doesn’t work, you aren’t going to give up trying if I don’t provide you with an airplane, a supply of silver iodide, flight training, runway, fuel, and so on (and even then the method will only be applicable to some days, while the pray for rain is applicable any time).
As for the best guess, if you suddenly need a best guess on a topic because someone told you of something and you couldn’t really see a major flaw in vague reasoning of the sort that can arrive at anything via a minor flaw on every step, that’s a backdoor other agents will exploit to take your money (those agents will likely also opt to modify their own beliefs somewhat, because, hell, it feels a lot better to be saving mankind than to be scamming people). What is actually important to you, is your utility, and the best reasoning here is strategic: do not leave backdoors open.
Not a relevant answer. You have given me no tools to estimate the risks or lack thereof in AI development. What methods do you use to reach conclusions on these issues? If they are good, I’d like to know them.
If you want to maximize your win, it is a relevant answer.
For the risk estimate per se, I think one needs not so much methods as a better understanding of the topic, which is attained by studying the field of artificial intelligence—in non cherry picked manner—and takes a long time. If you want easier estimate right now, you could try to estimate how privileged is the hypothesis that there is the risk. (There is no method that would let you calculate the wave from spin down and collision of orbiting black holes without spending a lot of time studying GR, applied mathematics, and computer science. Why do you think there’s a method for you to use to tackle even harder problem from first principles?)
Best yet, ban of thinking of it as risk (we have introduced, for instrumental reasons, the burden of proof on those whom say there is no risk, when it comes to new drugs etc, and we did so solely because introduction of random chemicals into a well evolved system is much more often harmful than beneficial. In general there is no reason to put burden of proof on those whom say there is no wolf, especially not when people screaming wolf get candy for doing so), and think of it as prediction of what happens in 100 years. Clearly, you would not listen to philosophers whom use ideals for predictions.
Thank you for your answer. I don’t think the methods you describe are much good for predictions. On the other hand, few methods are much good for predictions anyway.
I’ve already picked up a few online AI courses to get some background; emotionally this has made me feel that AI is likely to be somewhat less powerful than anticipated, but that it’s motivations are more certain to be more alien than I’d thought. Not sure how much weight to put on these intuitions.
Ultimately, when you can’t compute the right answer in the given time, you will either have no answer or compute a wrong one.
But if the question is possibly important and you have to make a decision now, you have to make a best guess. How do you think we should do that?
How do you know that you have to make a decision now? You don’t know when AGI is going to be invented. You don’t know if it will be a quick transition from expert systems towards general reasoning capabilities or if AGI will be constructed piecewise over a longer period of time. You don’t know if all that you currently believe to know will be rendered moot in future. You don’t know if the resources that you currently spend on researching friendly AI are a wasted opportunity because all that you could possible come up with will be much easier to come by in future.
All that you really know at this time is that smarter than human intelligence is likely possible and that something that is smarter is hard to control.
There’s so much that can go wrong with such reasoning, given that intelligence (even at the size of a galaxy of Dyson spheres) is not a perfect God, as to render such arguments irrelevant and entirely worthless. Furthermore there’s enough ways how the non-orthogonality can hold, such as e.g. almost all intelligences with wrong moral systems crashing or failing to improve, that are not covered by ‘converges’.
meta: Tendency to talk seriously about products of very bad reasoning really puts an upper bracket on the sanity of newcomers to LW. As is the idea that very bad argument trumps authority (when it comes to the whole topic).
What type of reasoning would you prefer to be used when talking about superintelligences?
Would you take criticism if it is not ‘positive’ and doesn’t give you alternative method to use for talking about same topic? Faulty reasoning has unlimited domain of application—you can ‘reason’ about purpose of the universe, number of angels that fit on a tip of a pin, of what superintelligences would do, etc. In those areas, non-faulty reasoning can not compete in terms of providing a sort of pleasure from reasoning, or in terms of interesting sounding ‘results’ that can be obtained with little effort and knowledge.
You can reason what particular cognitive architecture can do on a given task given N operations; you can reason what the best computational process can do in N operations. But that will involve actually using mathematics, and results will not be useful for unintelligent debates in the way in which your original statement is useful (I imagine you could use it to reply to someone who believes in absolute morality, as a soundbite; i really don’t see how it could have any predictive power what so ever about the superintelligence though).
I am interested in anything that allows better reasoning about these topics.
Mathematics has a somewhat limited use when discussing the orthogonality thesis. AIXI, and some calculations about the strength of optimisation processes and stuff like that. But when answering the question “is it likely that humans will build AIs with certain types of goals”, we need to look beyond mathematics.
I won’t pretend the argument in this post is strong—it’s just, to use the technical term, “kinda neat” and I’d never seen it presented this way before.
What would you consider reasonable reasoning on questions like the orthogonality thesis in practice?
That’s how religions were created, you know—they could not actually answer why lightning is thundering, why sun is moving through the sky, etc. So they did look way ‘beyond’ the non-faulty reasoning, in search for answers now (being inpatient), and got answers that were much much worse than no answers at all. I feel LW is doing precisely same thing with AIs. Ultimately, when you can’t compute the right answer in the given time, you will either have no answer or compute a wrong one.
On the orthogonality thesis, it is the case that you can’t answer this question given limited knowledge and time (got to know AI’s architecture first), and any reasonable reasoning tells you this, while LW’s pseudo-rationality keeps giving you wrong answers (that aren’t any less wrong than anyone else including the mormon church and any other weird religious group), I don’t quite sure what you guys are doing wrong; maybe the focus on biases and conflation of biases with stupidity did lead to a fallacy that lack of (known) biases will lead to non stupidity, i.e. smartness, and if only you won’t be biased you’ll have a good answer. It doesn’t work like this. It leads to another wrongness.
But if the question is possibly important and you have to make a decision now, you have to make a best guess. How do you think we should do that?
It was definitely important to make animals come, or to make it rain, tens thousands years ago. I’m getting a feeling that as I tell you that your rain making method doesn’t work, you aren’t going to give up trying if I don’t provide you with an airplane, a supply of silver iodide, flight training, runway, fuel, and so on (and even then the method will only be applicable to some days, while the pray for rain is applicable any time).
As for the best guess, if you suddenly need a best guess on a topic because someone told you of something and you couldn’t really see a major flaw in vague reasoning of the sort that can arrive at anything via a minor flaw on every step, that’s a backdoor other agents will exploit to take your money (those agents will likely also opt to modify their own beliefs somewhat, because, hell, it feels a lot better to be saving mankind than to be scamming people). What is actually important to you, is your utility, and the best reasoning here is strategic: do not leave backdoors open.
Not a relevant answer. You have given me no tools to estimate the risks or lack thereof in AI development. What methods do you use to reach conclusions on these issues? If they are good, I’d like to know them.
If you want to maximize your win, it is a relevant answer.
For the risk estimate per se, I think one needs not so much methods as a better understanding of the topic, which is attained by studying the field of artificial intelligence—in non cherry picked manner—and takes a long time. If you want easier estimate right now, you could try to estimate how privileged is the hypothesis that there is the risk. (There is no method that would let you calculate the wave from spin down and collision of orbiting black holes without spending a lot of time studying GR, applied mathematics, and computer science. Why do you think there’s a method for you to use to tackle even harder problem from first principles?)
Best yet, ban of thinking of it as risk (we have introduced, for instrumental reasons, the burden of proof on those whom say there is no risk, when it comes to new drugs etc, and we did so solely because introduction of random chemicals into a well evolved system is much more often harmful than beneficial. In general there is no reason to put burden of proof on those whom say there is no wolf, especially not when people screaming wolf get candy for doing so), and think of it as prediction of what happens in 100 years. Clearly, you would not listen to philosophers whom use ideals for predictions.
Thank you for your answer. I don’t think the methods you describe are much good for predictions. On the other hand, few methods are much good for predictions anyway.
I’ve already picked up a few online AI courses to get some background; emotionally this has made me feel that AI is likely to be somewhat less powerful than anticipated, but that it’s motivations are more certain to be more alien than I’d thought. Not sure how much weight to put on these intuitions.
How do you know that you have to make a decision now? You don’t know when AGI is going to be invented. You don’t know if it will be a quick transition from expert systems towards general reasoning capabilities or if AGI will be constructed piecewise over a longer period of time. You don’t know if all that you currently believe to know will be rendered moot in future. You don’t know if the resources that you currently spend on researching friendly AI are a wasted opportunity because all that you could possible come up with will be much easier to come by in future.
All that you really know at this time is that smarter than human intelligence is likely possible and that something that is smarter is hard to control.
How do you know we don’t? Figuring out whether there is urgency or not is one of those questions whose solution we need to estimate… somehow.