Suppose some new rich sponsor wanted to donate a lot to MIRI, subject to an independent outside group of experts evaluating the merits of some of its core claims, like that AGI is a near-term (under 100 years) x-risk and that MIRI has non-negligible odds (say, a few percent or more) of mitigating it. Who would you suggest s/he would engage for review?
like that AGI is a near-term (under 100 years) x-risk
FHI sent a survey the top 100 most cited authors in AI and got a response rate of ~1/3, and the median estimates backed this (although this needs to be checked for response bias). Results will be published in September at PT-AI.
x-risk and that MIRI has non-negligible odds (say, a few percent or more) of mitigating it.
I.e. a probability of a few percent that there is AI risk, MIRI solves it, and otherwise it wouldn’t have been solved and existential catastrophe would have resulted? That would not happen with non-gerrymandered criteria for the expert group.
But if a credible such group did deliver that result believably, then one could go to Gates or Buffett (who has spent hundreds of millions on nuclear risk efforts with much lower probability of averting nuclear war) or national governments and get billions in funding. All the work in that scenario is coming from the independent panel concluding the thing is many orders of magnitude better than almost any alternative use of spending, way past the threshold for funding.
The rich guy who says he would donate based on it is an irrelevancy in the hypo.
Damned if I know. Oddly enough, anyone chooses to spend a bunch of their life becoming an expert on these issues tends to be sympathetic to the claims, and most random others tend to make up crap on the spot and stick with it. If they could manage to pay Peter Norvig enough money to spend a lot of time working through these issues I’d be pretty optimistic, but Peter Norvig works for Google and would be hard to pay sufficiently.
I agree with Eliezer that the main difficulty is in getting top-quality, relatively rational people to spend hundreds of hours being educated, working through the arguments, etc.
Jaan has done a surprising amount of that and also read most or all of the Sequences. Thiel has not yet decided to put in that kind of time.
Here’s a list of people I’d want on that committee if they were willing to put in hundreds of hours catching up and working through the arguments with us: Scott Aaronson, Peter Norvig, Stuart Russell, Michael Nielsen.
I’d probably be able to add lots more names to that list if I could afford to spend more time becoming familiar with the epistemic standards and philosophical sophistication of more high-status CS people. I would trust Carl Shulman, Paul Christiano, Jacob Steinhardt, and a short list of others to add to my list with relatively little personal double-checking from me.
But yeah; the main problem seems to me that I don’t know how to get 400 hours of Andrew Ng’s time.
Although with Ng in particular it might not take 400 hours. When Louie and I met with him in Nov. ’12 he seemed to think AI was almost certainly a century or more away, but by May ’13 (after getting to do his deep learning work on Google’s massive server clusters for a few months) he changed his tune, saying “It gives me hope –- no, more than hope –- that we might be able to [build AGI]… We clearly don’t have the right algorithms yet. It’s going to take decades. This is not going to be an easy one, but I think there’s hope.” (On the other hand, maybe he just made himself sound more optimistic than he anticipates inside because he was giving a public interview on behalf of pro-AI Google.)
This is a great answer but actually a little tangential to my question, sorry for being vague. Mine was actually about the part of shminux’s proposal that involved finding potential mega donors. Relatedly, how much convincing do you think it would take to get Tallinn or thiel to increase their donations by an order of magnitude, something they could easily afford? This seems like a relatively high leverage plan if you can swing it. With x million dollars you can afford to actually pay to hire people like google can, if on a much smaller scale.
Suppose some new rich sponsor wanted to donate a lot to MIRI, subject to an independent outside group of experts evaluating the merits of some of its core claims, like that AGI is a near-term (under 100 years) x-risk and that MIRI has non-negligible odds (say, a few percent or more) of mitigating it. Who would you suggest s/he would engage for review?
FHI sent a survey the top 100 most cited authors in AI and got a response rate of ~1/3, and the median estimates backed this (although this needs to be checked for response bias). Results will be published in September at PT-AI.
I.e. a probability of a few percent that there is AI risk, MIRI solves it, and otherwise it wouldn’t have been solved and existential catastrophe would have resulted? That would not happen with non-gerrymandered criteria for the expert group.
But if a credible such group did deliver that result believably, then one could go to Gates or Buffett (who has spent hundreds of millions on nuclear risk efforts with much lower probability of averting nuclear war) or national governments and get billions in funding. All the work in that scenario is coming from the independent panel concluding the thing is many orders of magnitude better than almost any alternative use of spending, way past the threshold for funding.
The rich guy who says he would donate based on it is an irrelevancy in the hypo.
Damned if I know. Oddly enough, anyone chooses to spend a bunch of their life becoming an expert on these issues tends to be sympathetic to the claims, and most random others tend to make up crap on the spot and stick with it. If they could manage to pay Peter Norvig enough money to spend a lot of time working through these issues I’d be pretty optimistic, but Peter Norvig works for Google and would be hard to pay sufficiently.
Do you guys deliberately go out of your way to evangeliz to Jaan tallin and thiel or is that source of funds a lucky break?
I agree with Eliezer that the main difficulty is in getting top-quality, relatively rational people to spend hundreds of hours being educated, working through the arguments, etc.
Jaan has done a surprising amount of that and also read most or all of the Sequences. Thiel has not yet decided to put in that kind of time.
Here’s a list of people I’d want on that committee if they were willing to put in hundreds of hours catching up and working through the arguments with us: Scott Aaronson, Peter Norvig, Stuart Russell, Michael Nielsen.
I’d probably be able to add lots more names to that list if I could afford to spend more time becoming familiar with the epistemic standards and philosophical sophistication of more high-status CS people. I would trust Carl Shulman, Paul Christiano, Jacob Steinhardt, and a short list of others to add to my list with relatively little personal double-checking from me.
But yeah; the main problem seems to me that I don’t know how to get 400 hours of Andrew Ng’s time.
Although with Ng in particular it might not take 400 hours. When Louie and I met with him in Nov. ’12 he seemed to think AI was almost certainly a century or more away, but by May ’13 (after getting to do his deep learning work on Google’s massive server clusters for a few months) he changed his tune, saying “It gives me hope –- no, more than hope –- that we might be able to [build AGI]… We clearly don’t have the right algorithms yet. It’s going to take decades. This is not going to be an easy one, but I think there’s hope.” (On the other hand, maybe he just made himself sound more optimistic than he anticipates inside because he was giving a public interview on behalf of pro-AI Google.)
This is a great answer but actually a little tangential to my question, sorry for being vague. Mine was actually about the part of shminux’s proposal that involved finding potential mega donors. Relatedly, how much convincing do you think it would take to get Tallinn or thiel to increase their donations by an order of magnitude, something they could easily afford? This seems like a relatively high leverage plan if you can swing it. With x million dollars you can afford to actually pay to hire people like google can, if on a much smaller scale.