It’s probably not entirely fair to compare my case to yours because I started reading the Sequences before I was part of this community, and so I was much less familiar with the idea of Friendly AI than you are. But to answer your questions:
Before reading the Sequences, I assumed unfriendly AI was one more crazy speculative idea about the future, around the level of “We’ll discover psionics and merge into a single cosmic consciousness” and not really worthy of any more consideration.
I think you believe that superintelligent AI may not be possible, that it’s unlikely to “go foom”, and that in general it’s not a great use of our time to worry about it.
That’s a good question. Looking over the post list I’m surprised that I can’t find any that look like the sort of thing that would do that directly (there’s a lot about how it’s important to build a Friendly AI as opposed to just throw one together and assume it will be Friendly, but if I understand you right we don’t disagree there). It could have been an indirect effect of realizing that the person who wrote these was very smart and he believed in it. It could have been that they taught me enough rationality to realize I might be wrong about this and should consider changing my mind. And it could have been just very gradual worldview change. You said you were reading the debate with Robin, and that seems like a good starting point. The two dependency thingies labelled “Five Sources of Discontinuity” and “Optimization and the Singularity” here also give me vague memories of being good. But I guess that either I was wrong about the Sequences being full of brilliant pro-Singularity arguments, or they’re more complicated than I thought. Maybe someone else who’s read them more recently than I have can answer this better?
...which shouldn’t discourage you from reading the Sequences. They’re really good. Really. They might or might not directly help you on this question, but they’ll be indirectly helpful on this and many other things. It’s a really good use of your time (debating with me isn’t; I don’t claim any special insight on this issue beyond what I’ve picked up from the Sequences and elsewhere, and I don’t think I’ve ever posted any articles on AI simply because I wouldn’t even meet this community’s lax standards for expertise).
Here is why I believe that reading the Sequences might not be worth the effort:
1) According to your survey, 38.5% of all people have read at least 75% of the Sequences yet only 16.5% think that unfriendly AI is the most fearsome existential risk.
2) The following (smart) people have read the Sequences, and more, but do not agree about risks from AI:
According to your survey, 38.5% of all people have read at least 75% of the Sequences yet only 16.5% think that unfriendly AI is the most fearsome existential risk.
So what? I’m not even sure that Eliezer himself considers uFAI the most likely source of extinction. It’s just that Friendly AI would help save us from most the other possible sources of extinction too (not just from uFAI), and from several other sources of suffering too (not just extinction), so it kills multiple birds with one stone to figure it out.
As a point of note, I myself didn’t place uFAI as the most likely existential risk in that survey. That doesn’t mean I share your attitude.
No, but in the light of an expected utility calculation. Why would I read the Sequences?
Assuming you continue to write posts authoritatively about subjects related to said sequences—including criticisms of the contents therein—having read the sequences may reduce the frequency of you humiliating yourself.
They contain many insights unrelated to AI (looking at the sequences wiki page, it seems that most AI-ish things are concentrated in the second half). And many people had fun reading them. I think it would be a better use of time than trying to generically improve your math education that you speak of elsewhere (I don’t think it makes sense to learn math as an instrumental goal without a specific application in mind—unless you simply like math, in which case knock yourself out).
From a theoretical standpoint, you should never expect that observing something will shift your beliefs in some particular direction (and, guess what, there’s a post about that). This doesn’t work for humans—we can be convinced of things and we can expect to be convinced even if we don’t want to. But then, the fact that the sequences fail to convince many people shouldn’t be an argument against reading them. At least now you can be sure that they’re safe to read and won’t brainwash you.
I assumed unfriendly AI was one more crazy speculative idea about the future, around the level of “We’ll discover psionics and merge into a single cosmic consciousness” and not really worthy of any more consideration.
I do not believe that it is that speculative and I am really happy that there are people like Eliezer Yudkowsky who think about it.
In most of my submissions I try to show that there are a lot of possibilities of how the idea of superhuman AI could fail to be a risk.
Why do I do that? The reason for doing so are due to the main points of disagreement with Eliezer Yudkowsky and others who believe that “this is crunch time”. The points being that 1) I believe that they are overconfident when it comes to risks from AI, that the evidence simply doesn’t allow you to dramatize the case the way they do, and 2) I believe that they are overconfident when it comes to their methods of reasoning.
I would never have critized them if they had said, 1) “AI might pose a risk. We should think about it and evaluate the risk carefully.” and 2) “Here are some logical implications of AI being a risk. We don’t know if AI is a risk so those implications are secondary and should be discounted accordingly.”
But that is not what is happening. They portray friendly AI as a moral imperative and use the full weight of all logical implications of risks from AI to blow up its expected utility.
And that’s where my saying that I “found no flaws but feel that there are flaws” comes into play.
I understand that P(Y|X) ≈ 1, then P(X∧Y) ≈ P(X). The problem is that, as muflax put it, I don’t see how you can believe in the implied invisible and remain even remotely sane. It does not work out. Even though on an intellectual level I completely agree with it, my intuition is yelling that something is very wrong here. It is ear-battering. I can’t ignore it. Call it irrational or just sad, I can’t help it.
I think you believe that superintelligent AI may not be possible, that it’s unlikely to “go foom”, and that in general it’s not a great use of our time to worry about it.
It is fascinating. If I could work directly at it then I would do it. But giving away my money? Here we get to point #1, mentioned above.
Is there enough evidence that my money would make a difference? This question is deep. The question is not just about the likelihood of a negative Singularity, but also the expected utility of contributing any amount of money to friendly AI research. I am seriously unable to calculate that. I don’t even know if I should get an MRI to check for unruptured brain aneurysms.
Another problem is that I am not really altruistic. I’d love to see everybody happy. But that’s it. But then I also don’t really care about myself that much. I only care if I might suffer, but not about being dead. That’s what makes the cryonics question pretty easy for me. I just don’t care enough.
It could have been an indirect effect of realizing that the person who wrote these was very smart and he believed in it.
This is one of the things I don’t understand. I don’t think Eliezer is that smart. But even if he was, I don’t think that increases the probability of him being right about some extraordinary ideas very much. Especially since I have chatted with other people that are equally smart who told me that he is wrong.
There are many incredible smart people who hold really absurd ideas.
The biggest problem is that he hasn’t achieved much. All he did was putting some of the work of other people together, especially in the field of rationality and heuristics and biases. And he wrote a popular fanfic. That’s it.
Yeah, he got some rich people to give him money. But the same people also support other crazy ideas with the same amount of money. That’s little evidence.
It could have been that they taught me enough rationality to realize I might be wrong about this and should consider changing my mind.
Sure, I am very likely wrong. But that argument cuts both ways.
You said you were reading the debate with Robin, and that seems like a good starting point.
I will try. Right now I am very off-put by Eliezer’s style of writing. I have a hard time to understand what he is saying while Robin is very clear and I agree about like everything he says.
But I will try to continue and research everything I don’t understand.
...which shouldn’t discourage you from reading the Sequences. They’re really good. Really.
In what respect? Those posts that I have read were quite interesting. But I even enjoy reading a calculus book right now. And just as I expect to never actually benefit from learning calculus I don’t think that it is instrumentally useful to read the Sequences. It is not like that I am raving mad. I have enough rationality to live a good life without the Sequences.
If you mean that they are good in convincing you of risks from AI, then I also ask you how sure you are that they are not only convincing but actually factually right? Do you believe that you have the expertise that is necessary to discern a good argument about artificial intelligence from one that is not even wrong?
It’s a really good use of your time (debating with me isn’t;
Just one last question if you allow. What are you doing against risks from AI? Do you pursue a carrier where you can earn a lot of money to contribute it to SIAI?
What’s your numerical probability estimate that, assuming no one puts much work into stopping it, Unfriendly AI will seriously damage or destroy human civilization in the next few centuries?
Mine is...hmm...I don’t know. Maybe 50%? I’m not sure. I do know that if there were an asteroid nearby with the same probability of impacting Earth, I’d be running up to people and shaking them and shouting “WHY AREN’T WE BUILDING MORE ASTEROID DEFLECTORS?! WHAT’S WRONG WITH YOU? PEOPLE!” I don’t know if I believe in unconditional moral imperatives, but if there were a 50% chance of an asteroid striking Earth soon, or even a 10% chance, and no one was doing anything about it, I would at least feel an imperative conditional on some of my other beliefs to try to help stop it.
So maybe part of what the Sequences did for me was help calibrate my brain well enough so that I noticed the similarity between the asteroid and the AI case.
The remaining sane part seems to be a matter of personal psychology. My undergrad philosophy prof once told a story of how a classmate of hers had to be committed to an institution after reading a book on nihilism: he just started doubting everything and went mad. My prof read the same book on nihilism, thought it made some interesting arguments that she couldn’t immediately refute, went back to her everyday life, and a few years later reconsidered it and found some possible refutations.
I have always sympathized more with my professor’s point of view: I can read arguments which if taken seriously would be nightmarish or imply total doubt, admit the arguments seem plausible, and then go back to making pizza or doing homework or whatever. I’m not sure what the difference is between people like myself and my professor, and people like you and my professor’s classmate. Maybe you’re more rational, deep down? Or maybe you’re just naturally depressed and anxious, and your brain latches onto this as an excuse? I don’t know. In any case, I don’t at all think of being able to resist terrifying implications as a rationalist skill and I’m not sure what I would do in your position.
(in my own case, I have pretty much decided to live a normal life but give a reasonable amount of what I make to SIAI and associated charities, probably, volunteer for them if they need it, and leave it at that. Are the algorithms that produced this plan optimal? No. Are they a heck of a lot better than going insane and resisting the idea of friendly AI with all my might because if I accepted it I would have to go insane and give away all my money? Yes.)
Another problem is that I am not really altruistic. I’d love to see everybody happy. But that’s it. But then I also don’t really care about myself that much. I only care if I might suffer, but not about being dead. That’s what makes the cryonics question pretty easy for me. I just don’t care enough.
...I would describe this as being altruistic; also, I share your intuitions about death and cryonics.
There are many incredible smart people who hold really absurd ideas.
Okay, point. I guess I got the impression Eliezer was both smart and rational; that he was smart in exactly the way that ought to prevent him from holding absurd ideas. This is an intuition, so I can’t really justify it.
If you mean that they are good in convincing you of risks from AI, then I also ask you how sure you are that they are not only convincing but actually factually right? Do you believe that you have the expertise that is necessary to discern a good argument about artificial intelligence from one that is not even wrong?
This seems like a fully general counterargument. “Sure, the evidence for evolution sounds convincing; but how do you know it’s actually true and you aren’t just being tricked?”
Just one last question if you allow. What are you doing against risks from AI? Do you pursue a carrier where you can earn a lot of money to contribute it to SIAI?
I’m pursuing a career as a doctor. Despite a recent major setback, I’m still hoping to get there within a year or so. After that, yeah, I do intend to donate a lot to SIAI—albeit, as I said before, I don’t claim I’ll be anywhere near perfect.
Maybe 50%? I’m not sure. I do know that if there were an asteroid nearby with the same probability of impacting Earth, I’d be running up to people and shaking them and shouting “WHY AREN’T WE BUILDING MORE ASTEROID DEFLECTORS?! WHAT’S WRONG WITH YOU? PEOPLE!”
That would be more than enough to devote a big chunk of the world’s resources on friendly AI research, given the associated utility. But you can’t just make up completely unfounded conjectures, then claim that we don’t have evidence either way (50% chance) but that the utility associated with a negative outcome is huge and we should therefore take it seriously. Because that reasoning will ultimately make you privilege random high-utility outcomes over theories based on empirical evidence.
This seems like a fully general counterargument. “Sure, the evidence for evolution sounds convincing; but how do you know it’s actually true and you aren’t just being tricked?”
You can’t really compare that. The arguments for evolution are pretty easy to understand and the evidence is overwhelming. But Eliezer Yudkowsky could tell me anything about AI and I would have no way to tell if he was right or not even wrong.
After that, yeah, I do intend to donate a lot to SIAI—albeit, as I said before, I don’t claim I’ll be anywhere near perfect.
I see. That makes me take you much more seriously.
But Eliezer Yudkowsky could tell me anything about AI and I would have no way to tell if he was right or not even wrong.
You know, one upside of logic is that, if someone tells you proposition x is true, gives you the data, and shows their steps of reasoning, you can tell whether they’re lying or not. I’m not a hundred percent onboard with Yudkowsky’s AI risk views, but I can at least tell that his line of reasoning is correct as far as it goes. He may be making some unjustified assumptions about AI architecture, but he’s not wrong about there being a threat. If he’s making a mistake of logic, it’s not one I can find. A big, big chunk of mindspace is hostile-by-default.
I’m not rich. My gross annual salary is lower than Eliezer Yudkowsky’s or Nick Bostrom’s. (mentioned since you keep using me as your example commenter for donations)
It’s probably not entirely fair to compare my case to yours because I started reading the Sequences before I was part of this community, and so I was much less familiar with the idea of Friendly AI than you are. But to answer your questions:
Before reading the Sequences, I assumed unfriendly AI was one more crazy speculative idea about the future, around the level of “We’ll discover psionics and merge into a single cosmic consciousness” and not really worthy of any more consideration.
I think you believe that superintelligent AI may not be possible, that it’s unlikely to “go foom”, and that in general it’s not a great use of our time to worry about it.
That’s a good question. Looking over the post list I’m surprised that I can’t find any that look like the sort of thing that would do that directly (there’s a lot about how it’s important to build a Friendly AI as opposed to just throw one together and assume it will be Friendly, but if I understand you right we don’t disagree there). It could have been an indirect effect of realizing that the person who wrote these was very smart and he believed in it. It could have been that they taught me enough rationality to realize I might be wrong about this and should consider changing my mind. And it could have been just very gradual worldview change. You said you were reading the debate with Robin, and that seems like a good starting point. The two dependency thingies labelled “Five Sources of Discontinuity” and “Optimization and the Singularity” here also give me vague memories of being good. But I guess that either I was wrong about the Sequences being full of brilliant pro-Singularity arguments, or they’re more complicated than I thought. Maybe someone else who’s read them more recently than I have can answer this better?
...which shouldn’t discourage you from reading the Sequences. They’re really good. Really. They might or might not directly help you on this question, but they’ll be indirectly helpful on this and many other things. It’s a really good use of your time (debating with me isn’t; I don’t claim any special insight on this issue beyond what I’ve picked up from the Sequences and elsewhere, and I don’t think I’ve ever posted any articles on AI simply because I wouldn’t even meet this community’s lax standards for expertise).
(Addendum to my other comment)
Here is why I believe that reading the Sequences might not be worth the effort:
1) According to your survey, 38.5% of all people have read at least 75% of the Sequences yet only 16.5% think that unfriendly AI is the most fearsome existential risk.
2) The following (smart) people have read the Sequences, and more, but do not agree about risks from AI:
Robin Hanson
Katja Grace (who has been a visiting fellow)
John Baez (who interviews Eliezer Yudkowsky)
Holden Karnofsky
Ben Goertzel
So what? I’m not even sure that Eliezer himself considers uFAI the most likely source of extinction. It’s just that Friendly AI would help save us from most the other possible sources of extinction too (not just from uFAI), and from several other sources of suffering too (not just extinction), so it kills multiple birds with one stone to figure it out.
As a point of note, I myself didn’t place uFAI as the most likely existential risk in that survey. That doesn’t mean I share your attitude.
I hope I didn’t claim the Sequences, or any argument were 100% effective in changing the mind of every single person who read them.
Also, Ben Goertzel has read all the Sequences? That makes that recent conversation with Luke kind of sad.
No, but in the light of an expected utility calculation. Why would I read the Sequences?
Assuming you continue to write posts authoritatively about subjects related to said sequences—including criticisms of the contents therein—having read the sequences may reduce the frequency of you humiliating yourself.
They contain many insights unrelated to AI (looking at the sequences wiki page, it seems that most AI-ish things are concentrated in the second half). And many people had fun reading them. I think it would be a better use of time than trying to generically improve your math education that you speak of elsewhere (I don’t think it makes sense to learn math as an instrumental goal without a specific application in mind—unless you simply like math, in which case knock yourself out).
From a theoretical standpoint, you should never expect that observing something will shift your beliefs in some particular direction (and, guess what, there’s a post about that). This doesn’t work for humans—we can be convinced of things and we can expect to be convinced even if we don’t want to. But then, the fact that the sequences fail to convince many people shouldn’t be an argument against reading them. At least now you can be sure that they’re safe to read and won’t brainwash you.
I do not believe that it is that speculative and I am really happy that there are people like Eliezer Yudkowsky who think about it.
In most of my submissions I try to show that there are a lot of possibilities of how the idea of superhuman AI could fail to be a risk.
Why do I do that? The reason for doing so are due to the main points of disagreement with Eliezer Yudkowsky and others who believe that “this is crunch time”. The points being that 1) I believe that they are overconfident when it comes to risks from AI, that the evidence simply doesn’t allow you to dramatize the case the way they do, and 2) I believe that they are overconfident when it comes to their methods of reasoning.
I would never have critized them if they had said, 1) “AI might pose a risk. We should think about it and evaluate the risk carefully.” and 2) “Here are some logical implications of AI being a risk. We don’t know if AI is a risk so those implications are secondary and should be discounted accordingly.”
But that is not what is happening. They portray friendly AI as a moral imperative and use the full weight of all logical implications of risks from AI to blow up its expected utility.
And that’s where my saying that I “found no flaws but feel that there are flaws” comes into play.
I understand that P(Y|X) ≈ 1, then P(X∧Y) ≈ P(X). The problem is that, as muflax put it, I don’t see how you can believe in the implied invisible and remain even remotely sane. It does not work out. Even though on an intellectual level I completely agree with it, my intuition is yelling that something is very wrong here. It is ear-battering. I can’t ignore it. Call it irrational or just sad, I can’t help it.
It is fascinating. If I could work directly at it then I would do it. But giving away my money? Here we get to point #1, mentioned above.
Is there enough evidence that my money would make a difference? This question is deep. The question is not just about the likelihood of a negative Singularity, but also the expected utility of contributing any amount of money to friendly AI research. I am seriously unable to calculate that. I don’t even know if I should get an MRI to check for unruptured brain aneurysms.
Another problem is that I am not really altruistic. I’d love to see everybody happy. But that’s it. But then I also don’t really care about myself that much. I only care if I might suffer, but not about being dead. That’s what makes the cryonics question pretty easy for me. I just don’t care enough.
This is one of the things I don’t understand. I don’t think Eliezer is that smart. But even if he was, I don’t think that increases the probability of him being right about some extraordinary ideas very much. Especially since I have chatted with other people that are equally smart who told me that he is wrong.
There are many incredible smart people who hold really absurd ideas.
The biggest problem is that he hasn’t achieved much. All he did was putting some of the work of other people together, especially in the field of rationality and heuristics and biases. And he wrote a popular fanfic. That’s it.
Yeah, he got some rich people to give him money. But the same people also support other crazy ideas with the same amount of money. That’s little evidence.
Sure, I am very likely wrong. But that argument cuts both ways.
I will try. Right now I am very off-put by Eliezer’s style of writing. I have a hard time to understand what he is saying while Robin is very clear and I agree about like everything he says.
But I will try to continue and research everything I don’t understand.
In what respect? Those posts that I have read were quite interesting. But I even enjoy reading a calculus book right now. And just as I expect to never actually benefit from learning calculus I don’t think that it is instrumentally useful to read the Sequences. It is not like that I am raving mad. I have enough rationality to live a good life without the Sequences.
If you mean that they are good in convincing you of risks from AI, then I also ask you how sure you are that they are not only convincing but actually factually right? Do you believe that you have the expertise that is necessary to discern a good argument about artificial intelligence from one that is not even wrong?
Just one last question if you allow. What are you doing against risks from AI? Do you pursue a carrier where you can earn a lot of money to contribute it to SIAI?
What’s your numerical probability estimate that, assuming no one puts much work into stopping it, Unfriendly AI will seriously damage or destroy human civilization in the next few centuries?
Mine is...hmm...I don’t know. Maybe 50%? I’m not sure. I do know that if there were an asteroid nearby with the same probability of impacting Earth, I’d be running up to people and shaking them and shouting “WHY AREN’T WE BUILDING MORE ASTEROID DEFLECTORS?! WHAT’S WRONG WITH YOU? PEOPLE!” I don’t know if I believe in unconditional moral imperatives, but if there were a 50% chance of an asteroid striking Earth soon, or even a 10% chance, and no one was doing anything about it, I would at least feel an imperative conditional on some of my other beliefs to try to help stop it.
So maybe part of what the Sequences did for me was help calibrate my brain well enough so that I noticed the similarity between the asteroid and the AI case.
The remaining sane part seems to be a matter of personal psychology. My undergrad philosophy prof once told a story of how a classmate of hers had to be committed to an institution after reading a book on nihilism: he just started doubting everything and went mad. My prof read the same book on nihilism, thought it made some interesting arguments that she couldn’t immediately refute, went back to her everyday life, and a few years later reconsidered it and found some possible refutations.
I have always sympathized more with my professor’s point of view: I can read arguments which if taken seriously would be nightmarish or imply total doubt, admit the arguments seem plausible, and then go back to making pizza or doing homework or whatever. I’m not sure what the difference is between people like myself and my professor, and people like you and my professor’s classmate. Maybe you’re more rational, deep down? Or maybe you’re just naturally depressed and anxious, and your brain latches onto this as an excuse? I don’t know. In any case, I don’t at all think of being able to resist terrifying implications as a rationalist skill and I’m not sure what I would do in your position.
(in my own case, I have pretty much decided to live a normal life but give a reasonable amount of what I make to SIAI and associated charities, probably, volunteer for them if they need it, and leave it at that. Are the algorithms that produced this plan optimal? No. Are they a heck of a lot better than going insane and resisting the idea of friendly AI with all my might because if I accepted it I would have to go insane and give away all my money? Yes.)
...I would describe this as being altruistic; also, I share your intuitions about death and cryonics.
Okay, point. I guess I got the impression Eliezer was both smart and rational; that he was smart in exactly the way that ought to prevent him from holding absurd ideas. This is an intuition, so I can’t really justify it.
This seems like a fully general counterargument. “Sure, the evidence for evolution sounds convincing; but how do you know it’s actually true and you aren’t just being tricked?”
I’m pursuing a career as a doctor. Despite a recent major setback, I’m still hoping to get there within a year or so. After that, yeah, I do intend to donate a lot to SIAI—albeit, as I said before, I don’t claim I’ll be anywhere near perfect.
That would be more than enough to devote a big chunk of the world’s resources on friendly AI research, given the associated utility. But you can’t just make up completely unfounded conjectures, then claim that we don’t have evidence either way (50% chance) but that the utility associated with a negative outcome is huge and we should therefore take it seriously. Because that reasoning will ultimately make you privilege random high-utility outcomes over theories based on empirical evidence.
You can’t really compare that. The arguments for evolution are pretty easy to understand and the evidence is overwhelming. But Eliezer Yudkowsky could tell me anything about AI and I would have no way to tell if he was right or not even wrong.
I see. That makes me take you much more seriously.
You know, one upside of logic is that, if someone tells you proposition x is true, gives you the data, and shows their steps of reasoning, you can tell whether they’re lying or not. I’m not a hundred percent onboard with Yudkowsky’s AI risk views, but I can at least tell that his line of reasoning is correct as far as it goes. He may be making some unjustified assumptions about AI architecture, but he’s not wrong about there being a threat. If he’s making a mistake of logic, it’s not one I can find. A big, big chunk of mindspace is hostile-by-default.
I’m not rich. My gross annual salary is lower than Eliezer Yudkowsky’s or Nick Bostrom’s. (mentioned since you keep using me as your example commenter for donations)