I really hope that John Baez is going to explain himself and argue for why he is more concerned with global warming than risks from AI.
Since I was interviewing Yudkowsky rather than the other way around, I didn’t explain my views—I was getting him to explain his. But the last part of this interview will touch on global warming, and if you want to ask me questions, that would be a great time to do it.
(Week 311 is just the first part of a multi-part interview.)
For now, you might be interested to read about Gregory Benford’s assessment of the short-term future, which somewhat resembles my own.
Tim Tyler wrote:
It looks like a conventional “confused environmentalist” prioritisation to me.
I’m probably confused (who isn’t?), but I doubt I’m conventional. If I were, I probably wouldn’t be so eager to solicit the views of Benford, Yudkowsky and Drexler on my blog. A big problem is that different communities of intelligent people have very different views on which threats and opportunities are most important, and these communities don’t talk to each other enough and think clearly enough to come to agreement, even on factual issues. I’d like to make a dent in that problem.
The list you cite is not the explanation that XiXiDu seeks.
Since I was interviewing Yudkowsky rather than the other way around, I didn’t explain my views—I was getting him to explain his.
Would you be willing to write a blog post reviewing his arguments and explaining why you either reject them, don’t understand them or accept them and start working to mitigate risks from AI? It would be valuable to have someone like you, who is not deeply involved with the SIAI (Singularity Institute) or LessWrong.com, to write a critique on their arguments and objectives. I myself don’t have the education (yet) to do so and welcome any reassurance that would help me to take action.
If you don’t have the time to write a blog post, maybe you can answer just the following question. If someone was going to donate $100k and you could pick the charity, would you choose the SIAI? Yes/No answer if you’re too busy, a short explanation if you’ve the time. Thank you!
For now, you might be interested to read about Gregory Benford’s assessment of the short-term future, which somewhat resembles my own.
You mean, “before we take on the galaxy, let’s do a smaller problem”? So you don’t think that we’ll have to face risks from AI before climate change takes a larger toll? You don’t think that working on AGI means working on the best possible solution to the problem of climate change? And even if we had to start taking active measures against climate change in the 2020s, you don’t think we should rather spend that time on AI because we can survive a warmer world but no runaway AI? Gregory Benford writes that “we still barely glimpse the horrors we could be visiting on our children and their grandchildren’s grandchildren”. That sounds to me like he assumes that there will be grandchildren, which might not be the case if some kind of AGI doesn’t take care of a lot of other problems we’ll have to face soon.
A big problem is that different communities of intelligent people have very different views on which threats and opportunities are most important, and these communities don’t talk to each other enough and think clearly enough to come to agreement, even on factual issues.
I tell you that all you have to do is to read the LessWrong Sequences and the publications written by the SIAI to agree that working on AI is much more important than climate change, are you going to take the time and do it?
Since XiXiDu also asked this question on my blog, I answered over there.
I tell you that all you have to do is to read the LessWrong Sequences and the publications written by the SIAI to agree that working on AI is much more important than climate change, are you going to take the time and do it?
I have read most of those things, and indeed I’ve been interested in AI and the possibility of a singularity at least since college (say, 1980). That’s why I interviewed Yudkowsky.
I have read most of those things, and indeed I’ve been interested in AI and the possibility of a singularity at least since college (say, 1980).
That answers my questions. There are only two options, either there is no strong case for risks from AI or a world-class mathematician like you didn’t manage to understand the arguments after trying for 30 years. For me that means that I can only hope to be much smarter than you (to understand the evidence myself) or to conclude that Yudkowsky et al. are less intelligent than you are. No offense, but what other option is there?
I should also state how I would answer my question. My answer would be No. The SIAI deserves funding but since it currently receives $500,000 per year I would not recommend someone to donate another $100,000 right now. The reason is that I think that there are valid arguments that justify the existence of such an organisation. But there are no reasons to expect that they currently need more money. The SIAI does publish no progress report and does not disclose how it uses the money it gets. There are various other issues to decide that the SIAI does currently not deserve more donations. That is not to say that the problem of risks from AI may not deserve more funding, but differently. My current uncertainty about how urgent and substantive the risks are also does contribute to the decision that the SIAI is at this time well-funded.
I’m asking people like you to assess how likely it is that I am wrong about my judgement and if I should make it a priority to seek more information right now or concentrate on other projects.
Just a minor correction: this cannot be a true statement to make of an American 501(c)3 charity because it would be illegal of them to not disclose what they’re spending money on in their Form 990. Hence, it’s easy to examine SIAI/MIRI, Girl Scouts, Edge Foundation, Lifeboat Foundation, JSTOR, ALCOR… Really, all the information is there for anyone who wants to know it, you can download for free, one just has to just not be lazy and not assume that it doesn’t exist.
In 2009, they received 432,139 of “gifts”, made 194,686 by putting on a conference—and paid Yudkowsky 95,550 and Vassar 52,083. Yudkowsky probably also got a fair bit of the 83,934 spent on project 4c. 400,000 was also spent on the things described at the end of the document. Figures are all USD.
“Probably?” According to what priors? Do not make stuff up. As of 2013, MIRI has never paid anyone more than $99K in one year, and IIRC the $95K shown there was due to an error by the payroll service we were using which accidentally shifted one month of my salary backward by one year (paid on Dec 31 instead of Jan 1).
I confirm the payroll error part; I remember speaking to Amy about it a couple times, though it happened shortly before my time. I also suspect MIRI has never paid anymore than $99k in one year, but I haven’t looked it up.
XiXiDu wrote:
Since I was interviewing Yudkowsky rather than the other way around, I didn’t explain my views—I was getting him to explain his. But the last part of this interview will touch on global warming, and if you want to ask me questions, that would be a great time to do it.
(Week 311 is just the first part of a multi-part interview.)
For now, you might be interested to read about Gregory Benford’s assessment of the short-term future, which somewhat resembles my own.
Tim Tyler wrote:
I’m probably confused (who isn’t?), but I doubt I’m conventional. If I were, I probably wouldn’t be so eager to solicit the views of Benford, Yudkowsky and Drexler on my blog. A big problem is that different communities of intelligent people have very different views on which threats and opportunities are most important, and these communities don’t talk to each other enough and think clearly enough to come to agreement, even on factual issues. I’d like to make a dent in that problem.
The list you cite is not the explanation that XiXiDu seeks.
Would you be willing to write a blog post reviewing his arguments and explaining why you either reject them, don’t understand them or accept them and start working to mitigate risks from AI? It would be valuable to have someone like you, who is not deeply involved with the SIAI (Singularity Institute) or LessWrong.com, to write a critique on their arguments and objectives. I myself don’t have the education (yet) to do so and welcome any reassurance that would help me to take action.
If you don’t have the time to write a blog post, maybe you can answer just the following question. If someone was going to donate $100k and you could pick the charity, would you choose the SIAI? Yes/No answer if you’re too busy, a short explanation if you’ve the time. Thank you!
You mean, “before we take on the galaxy, let’s do a smaller problem”? So you don’t think that we’ll have to face risks from AI before climate change takes a larger toll? You don’t think that working on AGI means working on the best possible solution to the problem of climate change? And even if we had to start taking active measures against climate change in the 2020s, you don’t think we should rather spend that time on AI because we can survive a warmer world but no runaway AI? Gregory Benford writes that “we still barely glimpse the horrors we could be visiting on our children and their grandchildren’s grandchildren”. That sounds to me like he assumes that there will be grandchildren, which might not be the case if some kind of AGI doesn’t take care of a lot of other problems we’ll have to face soon.
I tell you that all you have to do is to read the LessWrong Sequences and the publications written by the SIAI to agree that working on AI is much more important than climate change, are you going to take the time and do it?
Since XiXiDu also asked this question on my blog, I answered over there.
I have read most of those things, and indeed I’ve been interested in AI and the possibility of a singularity at least since college (say, 1980). That’s why I interviewed Yudkowsky.
That answers my questions. There are only two options, either there is no strong case for risks from AI or a world-class mathematician like you didn’t manage to understand the arguments after trying for 30 years. For me that means that I can only hope to be much smarter than you (to understand the evidence myself) or to conclude that Yudkowsky et al. are less intelligent than you are. No offense, but what other option is there?
Understanding of the singularity is not a monotonically increasing function of intelligence.
I should also state how I would answer my question. My answer would be No. The SIAI deserves funding but since it currently receives $500,000 per year I would not recommend someone to donate another $100,000 right now. The reason is that I think that there are valid arguments that justify the existence of such an organisation. But there are no reasons to expect that they currently need more money. The SIAI does publish no progress report and does not disclose how it uses the money it gets. There are various other issues to decide that the SIAI does currently not deserve more donations. That is not to say that the problem of risks from AI may not deserve more funding, but differently. My current uncertainty about how urgent and substantive the risks are also does contribute to the decision that the SIAI is at this time well-funded.
I’m asking people like you to assess how likely it is that I am wrong about my judgement and if I should make it a priority to seek more information right now or concentrate on other projects.
Just a minor correction: this cannot be a true statement to make of an American 501(c)3 charity because it would be illegal of them to not disclose what they’re spending money on in their Form 990. Hence, it’s easy to examine SIAI/MIRI, Girl Scouts, Edge Foundation, Lifeboat Foundation, JSTOR, ALCOR… Really, all the information is there for anyone who wants to know it, you can download for free, one just has to just not be lazy and not assume that it doesn’t exist.
They do do various things like that from time to time—e.g.; http://singinst.org/achievements
Up to 2008, almost half of it went into paying their own salaries, IIRC.
The SIAI accounts are on Guidestar. You have to register, though.
In 2009, they received 432,139 of “gifts”, made 194,686 by putting on a conference—and paid Yudkowsky 95,550 and Vassar 52,083. Yudkowsky probably also got a fair bit of the 83,934 spent on project 4c. 400,000 was also spent on the things described at the end of the document. Figures are all USD.
“Probably?” According to what priors? Do not make stuff up. As of 2013, MIRI has never paid anyone more than $99K in one year, and IIRC the $95K shown there was due to an error by the payroll service we were using which accidentally shifted one month of my salary backward by one year (paid on Dec 31 instead of Jan 1).
I confirm the payroll error part; I remember speaking to Amy about it a couple times, though it happened shortly before my time. I also suspect MIRI has never paid anymore than $99k in one year, but I haven’t looked it up.
They do do various things like that from time to time—e.g.; http://singinst.org/achievements