I imagine this isn’t your intention, but this does read a lot like “I think external review like AI experts would be good, but if we do that review, and don’t liek the results, it’s because we picked the wrong AI experts.”
MIRI wants to learn more about whether its basic assumptions and their apparent strategic implications are sound.
Outsiders of various kinds want to learn more about whether MIRI is a reasonable thing to support and emulate.
These two groups (MIRI & outsiders) have wildly different information about risks from AI, and are thus in different positions with respect to how they should respond to external reviews of MIRI’s core assumptions.
To illustrate the point, consider two actual events of direct or indirect “external review” of MIRI’s core assumptions.
#1: Someone as intelligent and broadly informed as Russ Roberts was apparently relieved of some of his worry about AI risk upon hearing Kevin Kelly’s reasoning on the matter. Given my state of information, I can immediately see that Kelly’s point about the Church-Turing thesis isn’t relevant to whether we should worry about AI (Ctrl+F on this page for “The basis of my non-worry”). But maybe Russ Roberts isn’t in a position to do so, because he knows so little about AI and theory of computation that he can’t tell whether Kelly’s stated reasons for complacency are sound or not. So the result of this indirect “external review” of MIRI’s assumptions is that Roberts has to say to himself “Well, I don’t know what to think. Robin Hanson says we should worry but Kevin Kelly says we shouldn’t, and they both have fairly good normal credentials on the issue of long-term tech forecasting relative to almost everyone.” And I, meanwhile, can see that I haven’t been presented with much reason to re-evaluate my core assumptions about AI risk.
#2: In mid-2012, Paul Christiano, who at the time had just finished his undergraduate degree, gave (w/ help from Carl Shulman) such a detailed, informed, and reasonable critique of Eliezer’s standard case for high confidence in hard takeoff that both Eliezer and I have since mentally re-tagged “AI takeoff speed” as an “open question” rather than a “moderately closed question,” prompting Eliezer to write up intelligence explosion microeconomics as his “Friendly AI Open Problem #1.” Eliezer and I were in a position to make a fairly significant update from Paul’s argument, but unfortunately Paul has far fewer normal credentials as an “expert worth listening to” than Kevin Kelly or Robin Hanson do, so outsiders probably aren’t in a position to be moved much either way by Paul stating his opinion and his reasons on the issue, due to (fairly appropriate) epistemic learned helplessness.
Also note that if 10 of the world’s top AI experts spent two weeks with MIRI and FHI trying to understand our arguments, and their conclusion was that AI wasn’t much of a risk, and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we’re on the right track. (I don’t expect this to actually happen; this is just another illustration of the situation we’re in with regard to external review.)
ETA: Also, I should mention that I don’t gain any confidence in MIRI’s core assumptions form the fact that people like Kevin Kelly or these people give bad arguments against the plausibility of intelligence explosion. As far as I can tell, these people aren’t engaging the arguments in much detail, and are mostly just throwing up whatever rejections first occur to them. I would expect them to do that whether MIRI’s core assumptions were correct or not. That’s why I stipulated that to gain confidence in MIRI’s core assumptions, we’d need to get a bunch of smart, reasonable people to investigate the arguments in detail, and try hard to extract good objections from them, and learn that they can’t come up with good objections even under those circumstances.
Also note that if 10 of the world’s top AI experts spent two weeks with MIRI and FHI trying to understand our arguments, and their conclusion was that AI wasn’t much of a risk, and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we’re on the right track.
Based on what we already know this would require a very unrepresentative sample, and cause wider revisions. And if they published such obviously unconvincing reasons it would lead to similar updates in many casual observers.
And so what we are going to do is, there is really almost no reason to make human-like intelligence because we can do it so easily in 9 months. Untrained workforce.
Yes, this argument is remarkably unconvincing. Human labor is still costly, limited in supply (it’s not 9 months, it’s 20+ years, with feeding, energy costs, unreliable quality and many other restrictions), and so forth.
Based on what we already know this would require a very unrepresentative sample.
There’s too much focus on confirmation—e.g. if it is false, there must be some update in the opposite direction, but in practice one would just say that “those top 10 AI experts took us seriously and engaged out arguments, which boosts our confidence that we are on the right track”.
I’ll also mention that GiveWell — clearly not a den of MIRI-sympathizers — is effectively doing an external review of some of MIRI’s core assumptions, by way of “shallow investigations” of specific catastrophic risks (among other interventions).
So far, they’ve reviewed climate change, asteroids, supervolcanoes, and nuclear war. None of these reviews cite the corresponding chapters in GCR, perhaps because GiveWell wants to do its own review of these issues that is mostly independent from the Bostromian school (which includes MIRI).
So far, GiveWell seems to have come to the same conclusions as the Bostromian school has about these specific risks, with the major caveat that these are shallow investigations that are “not researched and vetted to the same level as [Givewell’s] standard recommendations.”
I’m pretty excited about GiveWell investigating GCRs independently of the Bostromian school, since (1) I admire the quality and self-skepticism of GiveWell’s research so far, (2) I think GCRs are important to study, and (3) this will provide a pretty solid external review of some of MIRI’s core assumptions about the severity of various x-risks.
and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we’re on the right track.
Unpack “top 10 AI experts” as “who we think the top 10 AI experts are” and unpack “their primary reasons were” with “we think their primary reasons were”, and this updating will sound a lot more silly, especially if further conditioned by “we do not have a way to show our superiority in a more objective manner”.
No. It was given over a series of in-person conversations and whiteboard calculations, with only scattered notes taken throughout. Paul does describe his own “mainline AI scenario” here, though.
MIRI wants to learn more about whether its basic assumptions and their apparent strategic implications are sound.
I still think it would be valuable to hear what relevant, independent AI experts think about these basic assumptions and strategic implications, perhaps accompanied with a detailed theory as to why they’ve come to wrong answers and MIRI has more advanced insight.
Well, we can do this for lots of specific cases. E.g. last time I spoke to Peter Norvig, he said his reason for not thinking much about AI risk at this point (despite including a discussion of it in his AI textbook) was that he’s fairly confident AI is hundreds of years away. Unfortunately, I didn’t have time to walk him through the points of When Will AI Be Created? to see exactly why we disagreed on this point.
This will all be a lot easier when Bostrom’s Superintelligence book comes out next year, so that experts can reply to the basic theses of our view when they are organized neatly in one place and explained in some detail with proper references and so on.
I imagine this isn’t your intention, but this does read a lot like “I think external review like AI experts would be good, but if we do that review, and don’t liek the results, it’s because we picked the wrong AI experts.”
Here are two things we want from an external review:
MIRI wants to learn more about whether its basic assumptions and their apparent strategic implications are sound.
Outsiders of various kinds want to learn more about whether MIRI is a reasonable thing to support and emulate.
These two groups (MIRI & outsiders) have wildly different information about risks from AI, and are thus in different positions with respect to how they should respond to external reviews of MIRI’s core assumptions.
To illustrate the point, consider two actual events of direct or indirect “external review” of MIRI’s core assumptions.
#1: Someone as intelligent and broadly informed as Russ Roberts was apparently relieved of some of his worry about AI risk upon hearing Kevin Kelly’s reasoning on the matter. Given my state of information, I can immediately see that Kelly’s point about the Church-Turing thesis isn’t relevant to whether we should worry about AI (Ctrl+F on this page for “The basis of my non-worry”). But maybe Russ Roberts isn’t in a position to do so, because he knows so little about AI and theory of computation that he can’t tell whether Kelly’s stated reasons for complacency are sound or not. So the result of this indirect “external review” of MIRI’s assumptions is that Roberts has to say to himself “Well, I don’t know what to think. Robin Hanson says we should worry but Kevin Kelly says we shouldn’t, and they both have fairly good normal credentials on the issue of long-term tech forecasting relative to almost everyone.” And I, meanwhile, can see that I haven’t been presented with much reason to re-evaluate my core assumptions about AI risk.
#2: In mid-2012, Paul Christiano, who at the time had just finished his undergraduate degree, gave (w/ help from Carl Shulman) such a detailed, informed, and reasonable critique of Eliezer’s standard case for high confidence in hard takeoff that both Eliezer and I have since mentally re-tagged “AI takeoff speed” as an “open question” rather than a “moderately closed question,” prompting Eliezer to write up intelligence explosion microeconomics as his “Friendly AI Open Problem #1.” Eliezer and I were in a position to make a fairly significant update from Paul’s argument, but unfortunately Paul has far fewer normal credentials as an “expert worth listening to” than Kevin Kelly or Robin Hanson do, so outsiders probably aren’t in a position to be moved much either way by Paul stating his opinion and his reasons on the issue, due to (fairly appropriate) epistemic learned helplessness.
Also note that if 10 of the world’s top AI experts spent two weeks with MIRI and FHI trying to understand our arguments, and their conclusion was that AI wasn’t much of a risk, and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we’re on the right track. (I don’t expect this to actually happen; this is just another illustration of the situation we’re in with regard to external review.)
Related: Contrarian Excuses.
ETA: Also, I should mention that I don’t gain any confidence in MIRI’s core assumptions form the fact that people like Kevin Kelly or these people give bad arguments against the plausibility of intelligence explosion. As far as I can tell, these people aren’t engaging the arguments in much detail, and are mostly just throwing up whatever rejections first occur to them. I would expect them to do that whether MIRI’s core assumptions were correct or not. That’s why I stipulated that to gain confidence in MIRI’s core assumptions, we’d need to get a bunch of smart, reasonable people to investigate the arguments in detail, and try hard to extract good objections from them, and learn that they can’t come up with good objections even under those circumstances.
Based on what we already know this would require a very unrepresentative sample, and cause wider revisions. And if they published such obviously unconvincing reasons it would lead to similar updates in many casual observers.
Yes, this argument is remarkably unconvincing. Human labor is still costly, limited in supply (it’s not 9 months, it’s 20+ years, with feeding, energy costs, unreliable quality and many other restrictions), and so forth.
There’s too much focus on confirmation—e.g. if it is false, there must be some update in the opposite direction, but in practice one would just say that “those top 10 AI experts took us seriously and engaged out arguments, which boosts our confidence that we are on the right track”.
I’ll also mention that GiveWell — clearly not a den of MIRI-sympathizers — is effectively doing an external review of some of MIRI’s core assumptions, by way of “shallow investigations” of specific catastrophic risks (among other interventions).
So far, they’ve reviewed climate change, asteroids, supervolcanoes, and nuclear war. None of these reviews cite the corresponding chapters in GCR, perhaps because GiveWell wants to do its own review of these issues that is mostly independent from the Bostromian school (which includes MIRI).
So far, GiveWell seems to have come to the same conclusions as the Bostromian school has about these specific risks, with the major caveat that these are shallow investigations that are “not researched and vetted to the same level as [Givewell’s] standard recommendations.”
I’m pretty excited about GiveWell investigating GCRs independently of the Bostromian school, since (1) I admire the quality and self-skepticism of GiveWell’s research so far, (2) I think GCRs are important to study, and (3) this will provide a pretty solid external review of some of MIRI’s core assumptions about the severity of various x-risks.
Unpack “top 10 AI experts” as “who we think the top 10 AI experts are” and unpack “their primary reasons were” with “we think their primary reasons were”, and this updating will sound a lot more silly, especially if further conditioned by “we do not have a way to show our superiority in a more objective manner”.
Is Paul Christiano’s hard takeoff critique publicly available?
No. It was given over a series of in-person conversations and whiteboard calculations, with only scattered notes taken throughout. Paul does describe his own “mainline AI scenario” here, though.
Ah, thanks.
I still think it would be valuable to hear what relevant, independent AI experts think about these basic assumptions and strategic implications, perhaps accompanied with a detailed theory as to why they’ve come to wrong answers and MIRI has more advanced insight.
Well, we can do this for lots of specific cases. E.g. last time I spoke to Peter Norvig, he said his reason for not thinking much about AI risk at this point (despite including a discussion of it in his AI textbook) was that he’s fairly confident AI is hundreds of years away. Unfortunately, I didn’t have time to walk him through the points of When Will AI Be Created? to see exactly why we disagreed on this point.
This will all be a lot easier when Bostrom’s Superintelligence book comes out next year, so that experts can reply to the basic theses of our view when they are organized neatly in one place and explained in some detail with proper references and so on.