People differ in their estimates within MIRI. Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.
For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization’s future (and a high probability that the future will be very powerfully shaped by artificial intelligence in a way that can be affected by initial conditions). But the impact of MIRI in particular has to be a far smaller subset of the expected impact of the cause as a whole, in light of its limited scale and capabilities relative to the relevant universes (total AI research, governments, etc), the probability that AI is not close enough for MIRI to be very relevant, the probability that MIRI’s approach turns out irrelevant, uncertainty over the sign of effects due to contributions to AI progress, future AI risk efforts/replaceability, and various other drag factors.
ETA: To be clear, I think that MIRI’s existence, relative to the counterfactual in which it never existed, has been a good thing and reduced x-risk in my opinion, despite not averting a “medium probability,” e.g. 10%, of x-risk.
ETA2: Probabilities matter because there are alternative uses of donations and human capital.
I have just spent a month in England interacting extensively with the EA movement here. Donors concerned with impact on the long-run future are considering donations to all of the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-university):
80,000 Hours, the Center for Effective Altruism and other organizations that are helping altruists to improve their careers, coordination, information, and do movement building; some specifically mention the Center For Applied Rationality; these organizations also improve non-charity options, e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes
AMF/GiveWell’s other recommended charities to keep GiveWell and the EA movement growing (GiveWell’s growth in particular has been meteoric, with less extreme but still rapid growth in other EA institutions such as Giving What We can and CEA), while actors like GiveWell Labs, Paul Christiano, and Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the interventions and organizations identified by such processes as most effective in terms of their far future impact
Finding ways to fund such evaluation with RFMF, e.g. by paying for FHI or CEA hires to work on them
A donor-advised fund investing the returns until such evaluations or more promising opportunities present themselves or are elicited by the fund, including both known options for which no organization with RFMF or adequate quality exists, and unknown future options; some possible applications include, e.g. convening panels of independent scientific experts to evaluate key technical claims about future technologies, extensions of the DAGGRE forecasting methods, a Bayesian aggregation algorithm that greatly improves extraction of scientific expert opinion or science courts that could mobilize much more talent and resources to neglected problems with good cases, some key steps in biotech enhancement, AI safety research when AI is better understood, and more
This Paul Christiano post discusses the virtues of the donor-advised fund/”Fund for the Future” approach; Giving What We Can has already set up a charitable trust to act as a donor-advised fund in the UK, with one coming soon in the US, and Fidelity already offers a standardized donor-advised fund in America (DAFs allow one to claim tax benefits of donation immediately and then allow the donation to compound); there was much discussion this month about the details of setting up a DAF dedicated to far future causes (the main logistical difficulties are setting up the decision criteria, credibility, and maximum protection from taxation and disruption)
I’m eager to see Eliezer’s planned reply to your “ETA2”, but in the meantime, here are a few of my own thoughts on this...
My guess is that movement-building and learning are still the best things to do right now for AI risk reduction. CEA, CFAR, and GiveWell are doing good movement-building, though the GiveWell crowd tends to be less interested in x-risk mitigation. GiveWell is doing a large share of the EA-learning, and might eventually (via GiveWell labs) do some of the x-risk learning (right now GiveWell has a lot of catching up to do on x-risk).
The largest share of the “explicit” x-risk learning is happening at or near FHI & MIRI, including e.g. Christiano. Lots of “implicit” x-risk learning is happening at e.g. NASA where it’s not clear that EA-sourced funding can have much marginal effect relative to the effect it could have on tiny organizations like MIRI and FHI.
My impression, which could be wrong, is that GiveWell’s ability to hire more researchers is not funding-limited but rather limited by management’s preference to offer lower salaries than necessary to ensure cause loyalty. (I would prefer GiveWell raise salaries and grow its research staff faster.) AMF could be fully funded relatively easily by Good Ventures or the Gates Foundation but maybe they’re holding back because this would be discouraging to the EA movement: small-scale donors requiring the high-evidence threshold met by GiveWell’s top charities would say “Well, I guess there’s nothing for little ’ol me to do here.” (There are other reasons they may be holding back, too.)
I think accelerating learning is more important right now than a DAF. Getting high-quality evidence about which x-risk mitigation efforts are worthwhile requires lots of work, but one thing we’ve learned in the past decade is that causes with high-quality evidence for their effectiveness tend to get funded, and this trend is probably increasing. The sooner we do enough learning to have high-quality evidence for the goodness of particular x-risk mitigation efforts, the sooner large funders will fund those efforts. Or, as Christiano writes:
To me it currently looks like the value of getting information faster is significantly higher than the value of money, and on the current margin I think most of these learning activities are underfunded.
And:
A relatively small set of activities seems to be responsible for most learning that is occurring (for example, much of GiveWell’s work, some work within the Centre for Effective Altruism, some strategy work within MIRI, hopefully parts of this blog, and a great number of other activities that can’t be so easily sliced up)
However, Paul thinks there are serious RFMF problems here:
A more serious concern is that there seems to currently be a significant deficit of human capital specialized for this problem and willing to work on it (without already being committed to work on it), so barring some new recruitment strategies (e.g. paying market wages for non-EAs to do EA strategy research) there are significant issues with room for more funding.
In contrast, I think there is plenty of room for more funding here, even without resorting to “paying market wages for non-EAs to do EA strategy research”:
MIRI could run more workshops and hire some able and willing FAI researchers, which I think is quite valuable for x-risk mitigation strategy learning apart from the object-level FAI progress it might produce. But even excluding this...
With more cash, FHI and CSER could host strategy-relevant conferences and workshops, and get people like Stuart Russell and Richard Posner to participate.
I have plenty of EAs capable of doing the labor-intensive data-gathering work needed for much of the strategy work, e.g. collecting data on how fast different parts of AI are progressing, how much money has gone into AI R&D each decade since the 60s, how ripple effects have worked historically, more IEM-relevant data like Katja’s tech report, etc. I just don’t have the money to pay them to do it.
FHI has lots more researcher-hours it could purchase if it had more cash.
Finally, a clarification: If I think movement-building and learning are most important right now, why is MIRI focused on math research this year? My views on this have shifted even since our 2013 strategy post, and I should note that Eliezer’s reasons for focusing on math research are probably somewhat different from mine.
In my estimation, MIRI’s focus on math research offers the following benefits to movement-building and learning:
Math research has better traction than strategic research with the world’s top cognitive ability. And once top talent is engaged by the math research, some of these top thinkers turn their attention to the strategic issues, too. (Historically true, not just speculation.)
Without an object-level research program on the most important problem (beneficent superintelligence), many of the best people just “bounce off” because there’s nothing for them to engage directly. (Historically true, not just speculation.)
And of course, FAI research tells us some things about how hard FAI research is, which lines of inquiry are tractable now, etc.
Your reasons for focusing on math research at MIRI seem sound, but I take it you’ve noticed the warning sign of finding that what you already decided to do turns out to be a good idea for different reasons than you originally thought?
Yes, though these reasons are pretty similar to the reasons that made me switch positions on strategy back when I thought a focus on strategic research would be best for MIRI in 2013.
My impression, which could be wrong, is that GiveWell’s ability to hire more researchers is not funding-limited but rather limited by management’s preference to offer lower salaries than necessary to ensure cause loyalty.
It’s clearly not funding-limited, as they have plenty of funding for operations. I’m less confident of the salaries explanation as to their difficulties hiring: it is quite plausible as a significant factor.
And the “give more money via GiveWell’s top charities to accelerate their approach to their limits of growth” rationale gets worse every year, as the contribution of a dollar to their money moved falls by half, and the number of potential doublings remaining falls. So I would not see direct donations to GiveWell or its top charities as competitive, although other interventions that bolstered it more effectively could.
Two plausible examples: 80,000 hours might deliver a number of good new hires to GiveWell, or the Effective Fundraising experimental project, inspired by discussion like this 80,000 hours blog post, may succeed in efficiently mobilizing non-EA funds to support GiveWell’s top charities.
Getting high-quality evidence about which x-risk mitigation efforts are worthwhile requires lots of work, but one thing we’ve learned in the past decade is that causes with high-quality evidence for their effectiveness tend to get funded, and this trend is probably increasing. The sooner we do enough learning to have high-quality evidence for the goodness of particular x-risk mitigation efforts, the sooner large funders will fund those efforts.
Yes.
I think accelerating learning is more important right now than a DAF.
One of the biggest virtues of a large “fund for the future,” IMHO, is that it would make it easier to start up new projects in the field separate from existing organizations if they could meet the (transparently announced) standards of the fund, with the process as transparent and productive of information as practicable, GiveWell style.
And it could serve those who think the existing organizations in the field are deficient in some remediable way (rather than having some general objection to all work in the area).
In contrast, I think there is plenty of room for more funding here, even without resorting to “paying market wages for non-EAs to do EA strategy research...[FHI RFMF to hire more academics/free up grant-related time from existing staff...MIRI math workshops]...[hiring people to collect and publish data on past AI predictions, past AI progress, past inputs into the AI field, etc...]
Good points that I’m largely on board with, qualitatively, although one needs to make more of a case to show they meet the bar of beating existing alternatives, or waiting for others to enter the field and do things better.
One of the biggest virtues of a large “fund for the future,” IMHO, is that it would make it easier to start up new projects in the field separate from existing organizations if they could meet the (transparently announced) standards of the fund, with the process as transparent and productive of information as practicable, GiveWell style.
And it could serve those who think the existing organizations in the field are deficient in some remediable way (rather than having some general objection to all work in the area).
That does sound good. Is there any ongoing progress on figuring out what those transparently announced standards could be, and how one might set up such a DAF? Are there such standards in place for the one in the UK?
The one in the UK is mainly functioning as a short term DAF along the lines of “direct the money as you intend, with trust of GWWC as a backstop” which is fine if you don’t want to delay disbursement until after you die.
Is there any ongoing progress on figuring out what those transparently announced standards could be, and how one might set up such a DAF?
Not yet, so far mainly discussions, e.g. with Paul, Rob Wiblin, Nick Beckstead, et al. I expect more from CEA on this (not wholly independently of my own actions).
My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015.
Hasn’t Eliezer said, on every occasion since the beginning of LW when the opportunity has arisen, that Eliezer-in-1999 was disastrously wrong and confused about lots of important things?
(I don’t know whether present-day-Eliezer thinks 18-years-ago-Eliezer was wrong about this particular thing, but I would be cautious about taking things he said that long ago as strongly indicative of his present opinions.)
Yes, I am aware that this is what Eliezer has said, and I wasn’t implying that those early statements reflect Eliezer’s current thinking. There is a clear difference between “Eliezer believed this in the past, so he must believe it at present” and “Eliezer made some wrong predictions in the past, so we must treat his current predictions with caution”. Eliezer is entitled to ask his readers not to assume that his past beliefs reflect those of his present self, but he is not entitled to ask them not to hold him responsible for having once said stuff that some may think was ill-judged.
I hadn’t realised anyone was arguing for not treating Eliezer’s current predictions with caution. I can’t imagine why anyone wouldn’t treat anyone’s predictions with caution in this field.
My point is that these early pronouncements are (limited) evidence that we should treat Eliezer’s predictions with more caution than we would otherwise.
OK, I guess. I have to say that the main impression I’m getting from this exchange is that you wanted to say “boo Eliezer”; it seems like if you wanted to make an actual usefully constructive point you’d have been somewhat more explicit in your original comment. (“Eliezer wrote this in 1999: [...]. I know that Eliezer has since repudiated a lot of his opinions and thought processes of that period, but if his opinions were that badly wrong in 1999 then we shouldn’t take them too seriously now either.” or whatever.)
I will vigorously defend anyone’s right to say “boo Eliezer” or “yay Eliezer”, but don’t have much optimism about getting a useful outcome from a conversation that begins that way, and will accordingly drop it now.
Well, a nanowar is just a conflict on a very, very small scale—like many orders of magnitude less serious than your average barfight. Perhaps we had one before 2015 and nobody noticed! Now we just have to wait until 2020 for the seed AI to transcend.
Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.
Are these available? Are they the standard stuff (i.e., “Evidence and Import”)?
~
For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization’s future
How do you arrive at that conclusion? I’m less skeptical of the cause-specific claim than the organization-specific claim, but it’s worth digging deeper into.
Eliezer...Are these available? Are they the standard stuff (i.e., “Evidence and Import”)?
Yes, and his posts about intelligence explosion on Overcoming Bias, this, this, and unfortunately comments scattered around Less Wrong or various interveiws that would take some work to find and gather in one place.
How do you arrive at that conclusion? I’m less skeptical of the cause-specific claim than the organization-specific claim, but it’s worth digging deeper into.
Nick Bostrom’s book on superintelligence probably provides the best single treatment now, having synthesized most pre-existing work. It is moving towards publication, but you might ask him if you can read the draft.
Most pre-existing work? I would’ve said “having synthesized ~5% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI.”
Good news! Having now read the near-finished draft, my new guess is that Bostrom’s book synthesizes more like 20% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI. A lot has been added to the book since April. It’s really killing me that the book won’t be published until mid 2014.
One can delve indefinitely into any subtopic, but with diminishing returns. Do you think that it doesn’t address most of the higher-level topic areas, if not all of the issues arising therein?
No, I think it does a pretty good job of that. I’m not arguing that the book should be different than it is. I’m just saying that it definitely doesn’t synthesize “most” pre-existing work.
People differ in their estimates within MIRI. Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.
For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization’s future (and a high probability that the future will be very powerfully shaped by artificial intelligence in a way that can be affected by initial conditions). But the impact of MIRI in particular has to be a far smaller subset of the expected impact of the cause as a whole, in light of its limited scale and capabilities relative to the relevant universes (total AI research, governments, etc), the probability that AI is not close enough for MIRI to be very relevant, the probability that MIRI’s approach turns out irrelevant, uncertainty over the sign of effects due to contributions to AI progress, future AI risk efforts/replaceability, and various other drag factors.
ETA: To be clear, I think that MIRI’s existence, relative to the counterfactual in which it never existed, has been a good thing and reduced x-risk in my opinion, despite not averting a “medium probability,” e.g. 10%, of x-risk.
ETA2: Probabilities matter because there are alternative uses of donations and human capital.
I have just spent a month in England interacting extensively with the EA movement here. Donors concerned with impact on the long-run future are considering donations to all of the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-university):
80,000 Hours, the Center for Effective Altruism and other organizations that are helping altruists to improve their careers, coordination, information, and do movement building; some specifically mention the Center For Applied Rationality; these organizations also improve non-charity options, e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes
AMF/GiveWell’s other recommended charities to keep GiveWell and the EA movement growing (GiveWell’s growth in particular has been meteoric, with less extreme but still rapid growth in other EA institutions such as Giving What We can and CEA), while actors like GiveWell Labs, Paul Christiano, and Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the interventions and organizations identified by such processes as most effective in terms of their far future impact
Finding ways to fund such evaluation with RFMF, e.g. by paying for FHI or CEA hires to work on them
The FHI’s other work
A donor-advised fund investing the returns until such evaluations or more promising opportunities present themselves or are elicited by the fund, including both known options for which no organization with RFMF or adequate quality exists, and unknown future options; some possible applications include, e.g. convening panels of independent scientific experts to evaluate key technical claims about future technologies, extensions of the DAGGRE forecasting methods, a Bayesian aggregation algorithm that greatly improves extraction of scientific expert opinion or science courts that could mobilize much more talent and resources to neglected problems with good cases, some key steps in biotech enhancement, AI safety research when AI is better understood, and more
This Paul Christiano post discusses the virtues of the donor-advised fund/”Fund for the Future” approach; Giving What We Can has already set up a charitable trust to act as a donor-advised fund in the UK, with one coming soon in the US, and Fidelity already offers a standardized donor-advised fund in America (DAFs allow one to claim tax benefits of donation immediately and then allow the donation to compound); there was much discussion this month about the details of setting up a DAF dedicated to far future causes (the main logistical difficulties are setting up the decision criteria, credibility, and maximum protection from taxation and disruption)
I’m eager to see Eliezer’s planned reply to your “ETA2”, but in the meantime, here are a few of my own thoughts on this...
My guess is that movement-building and learning are still the best things to do right now for AI risk reduction. CEA, CFAR, and GiveWell are doing good movement-building, though the GiveWell crowd tends to be less interested in x-risk mitigation. GiveWell is doing a large share of the EA-learning, and might eventually (via GiveWell labs) do some of the x-risk learning (right now GiveWell has a lot of catching up to do on x-risk).
The largest share of the “explicit” x-risk learning is happening at or near FHI & MIRI, including e.g. Christiano. Lots of “implicit” x-risk learning is happening at e.g. NASA where it’s not clear that EA-sourced funding can have much marginal effect relative to the effect it could have on tiny organizations like MIRI and FHI.
My impression, which could be wrong, is that GiveWell’s ability to hire more researchers is not funding-limited but rather limited by management’s preference to offer lower salaries than necessary to ensure cause loyalty. (I would prefer GiveWell raise salaries and grow its research staff faster.) AMF could be fully funded relatively easily by Good Ventures or the Gates Foundation but maybe they’re holding back because this would be discouraging to the EA movement: small-scale donors requiring the high-evidence threshold met by GiveWell’s top charities would say “Well, I guess there’s nothing for little ’ol me to do here.” (There are other reasons they may be holding back, too.)
I think accelerating learning is more important right now than a DAF. Getting high-quality evidence about which x-risk mitigation efforts are worthwhile requires lots of work, but one thing we’ve learned in the past decade is that causes with high-quality evidence for their effectiveness tend to get funded, and this trend is probably increasing. The sooner we do enough learning to have high-quality evidence for the goodness of particular x-risk mitigation efforts, the sooner large funders will fund those efforts. Or, as Christiano writes:
And:
However, Paul thinks there are serious RFMF problems here:
In contrast, I think there is plenty of room for more funding here, even without resorting to “paying market wages for non-EAs to do EA strategy research”:
MIRI could run more workshops and hire some able and willing FAI researchers, which I think is quite valuable for x-risk mitigation strategy learning apart from the object-level FAI progress it might produce. But even excluding this...
With more cash, FHI and CSER could host strategy-relevant conferences and workshops, and get people like Stuart Russell and Richard Posner to participate.
I have plenty of EAs capable of doing the labor-intensive data-gathering work needed for much of the strategy work, e.g. collecting data on how fast different parts of AI are progressing, how much money has gone into AI R&D each decade since the 60s, how ripple effects have worked historically, more IEM-relevant data like Katja’s tech report, etc. I just don’t have the money to pay them to do it.
FHI has lots more researcher-hours it could purchase if it had more cash.
Finally, a clarification: If I think movement-building and learning are most important right now, why is MIRI focused on math research this year? My views on this have shifted even since our 2013 strategy post, and I should note that Eliezer’s reasons for focusing on math research are probably somewhat different from mine.
In my estimation, MIRI’s focus on math research offers the following benefits to movement-building and learning:
Math research has better traction than strategic research with the world’s top cognitive ability. And once top talent is engaged by the math research, some of these top thinkers turn their attention to the strategic issues, too. (Historically true, not just speculation.)
Without an object-level research program on the most important problem (beneficent superintelligence), many of the best people just “bounce off” because there’s nothing for them to engage directly. (Historically true, not just speculation.)
And of course, FAI research tells us some things about how hard FAI research is, which lines of inquiry are tractable now, etc.
Your reasons for focusing on math research at MIRI seem sound, but I take it you’ve noticed the warning sign of finding that what you already decided to do turns out to be a good idea for different reasons than you originally thought?
Yes, though these reasons are pretty similar to the reasons that made me switch positions on strategy back when I thought a focus on strategic research would be best for MIRI in 2013.
It’s clearly not funding-limited, as they have plenty of funding for operations. I’m less confident of the salaries explanation as to their difficulties hiring: it is quite plausible as a significant factor.
And the “give more money via GiveWell’s top charities to accelerate their approach to their limits of growth” rationale gets worse every year, as the contribution of a dollar to their money moved falls by half, and the number of potential doublings remaining falls. So I would not see direct donations to GiveWell or its top charities as competitive, although other interventions that bolstered it more effectively could.
Two plausible examples: 80,000 hours might deliver a number of good new hires to GiveWell, or the Effective Fundraising experimental project, inspired by discussion like this 80,000 hours blog post, may succeed in efficiently mobilizing non-EA funds to support GiveWell’s top charities.
Yes.
One of the biggest virtues of a large “fund for the future,” IMHO, is that it would make it easier to start up new projects in the field separate from existing organizations if they could meet the (transparently announced) standards of the fund, with the process as transparent and productive of information as practicable, GiveWell style.
And it could serve those who think the existing organizations in the field are deficient in some remediable way (rather than having some general objection to all work in the area).
Good points that I’m largely on board with, qualitatively, although one needs to make more of a case to show they meet the bar of beating existing alternatives, or waiting for others to enter the field and do things better.
Also, I should mention the Global Catastrophic Risks Institute, even if no one at the EA events in England mentioned it while I was there.
That does sound good. Is there any ongoing progress on figuring out what those transparently announced standards could be, and how one might set up such a DAF? Are there such standards in place for the one in the UK?
The one in the UK is mainly functioning as a short term DAF along the lines of “direct the money as you intend, with trust of GWWC as a backstop” which is fine if you don’t want to delay disbursement until after you die.
Not yet, so far mainly discussions, e.g. with Paul, Rob Wiblin, Nick Beckstead, et al. I expect more from CEA on this (not wholly independently of my own actions).
Eliezer wrote this in 1999:
Hasn’t Eliezer said, on every occasion since the beginning of LW when the opportunity has arisen, that Eliezer-in-1999 was disastrously wrong and confused about lots of important things?
(I don’t know whether present-day-Eliezer thinks 18-years-ago-Eliezer was wrong about this particular thing, but I would be cautious about taking things he said that long ago as strongly indicative of his present opinions.)
Yes, I am aware that this is what Eliezer has said, and I wasn’t implying that those early statements reflect Eliezer’s current thinking. There is a clear difference between “Eliezer believed this in the past, so he must believe it at present” and “Eliezer made some wrong predictions in the past, so we must treat his current predictions with caution”. Eliezer is entitled to ask his readers not to assume that his past beliefs reflect those of his present self, but he is not entitled to ask them not to hold him responsible for having once said stuff that some may think was ill-judged.
I hadn’t realised anyone was arguing for not treating Eliezer’s current predictions with caution. I can’t imagine why anyone wouldn’t treat anyone’s predictions with caution in this field.
My point is that these early pronouncements are (limited) evidence that we should treat Eliezer’s predictions with more caution than we would otherwise.
OK, I guess. I have to say that the main impression I’m getting from this exchange is that you wanted to say “boo Eliezer”; it seems like if you wanted to make an actual usefully constructive point you’d have been somewhat more explicit in your original comment. (“Eliezer wrote this in 1999: [...]. I know that Eliezer has since repudiated a lot of his opinions and thought processes of that period, but if his opinions were that badly wrong in 1999 then we shouldn’t take them too seriously now either.” or whatever.)
I will vigorously defend anyone’s right to say “boo Eliezer” or “yay Eliezer”, but don’t have much optimism about getting a useful outcome from a conversation that begins that way, and will accordingly drop it now.
Thanks for the feedback. I agree that a comment worded in the manner you suggest would have communicated my point more effectively.
Yudkowsky has changed his views a lot over the last 18 years though. A lot of his earlier writing is extremely optimistic about AI and it’s timeline.
Well, a nanowar is just a conflict on a very, very small scale—like many orders of magnitude less serious than your average barfight. Perhaps we had one before 2015 and nobody noticed! Now we just have to wait until 2020 for the seed AI to transcend.
Are these available? Are they the standard stuff (i.e., “Evidence and Import”)?
~
How do you arrive at that conclusion? I’m less skeptical of the cause-specific claim than the organization-specific claim, but it’s worth digging deeper into.
Yes, and his posts about intelligence explosion on Overcoming Bias, this, this, and unfortunately comments scattered around Less Wrong or various interveiws that would take some work to find and gather in one place.
Nick Bostrom’s book on superintelligence probably provides the best single treatment now, having synthesized most pre-existing work. It is moving towards publication, but you might ask him if you can read the draft.
Most pre-existing work? I would’ve said “having synthesized ~5% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI.”
Good news! Having now read the near-finished draft, my new guess is that Bostrom’s book synthesizes more like 20% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI. A lot has been added to the book since April. It’s really killing me that the book won’t be published until mid 2014.
One can delve indefinitely into any subtopic, but with diminishing returns. Do you think that it doesn’t address most of the higher-level topic areas, if not all of the issues arising therein?
No, I think it does a pretty good job of that. I’m not arguing that the book should be different than it is. I’m just saying that it definitely doesn’t synthesize “most” pre-existing work.