My understanding of Luke’s current strategy for MIRI is that it does not hinge on whether or not MIRI itself eventually builds AI or not.
Correct. I doubt that the most likely winning scenario involves MIRI building most of an FAI itself. (By “MIRI” I mean to include MIRI-descendents.) I helped flip MIRI to a focus on technical research because:
Except to the degree that public FAI progress enables others to build uFAI, humanity is better off with more FAI research on hand (than will be the case without MIRI’s efforts) when it becomes clear AGI is around the corner. And right now MIRI is focused on safely publishable FAI research.
Compared to tech forecasting or philosophy research, technical research is more able to attract the attention of the literally-smartest young people in the world. Once they’re engaged, they sometimes turn their attention to the strategic issues as well, so in the long run a focus on technical research might actually be better for strategy research than a focus on strategy research is, at least now that the basics will be laid out in Bostrom’s Superintelligence book next year. Example: Paul Christiano (an IMO silver medalist who authored a paper on quantum money with Scott Aaronson before he was 20) is now one of the most useful superintelligence strategists, but was initially attracted to the area because he found the issue of cryptographic boxes for AI intellectually stimulating, and then he spent 200+ hours talking to Carl Shulman and became a good strategist.
There are several important strategic questions about which I only expect to get evidence by trying to build Friendly AI. E.g. how hard is FAI relative to AGI? What’s the distribution of eAIs around FAIs in mind design space, and how much extra optimization power do you have to apply to avoid them? How much better can we do on the value-loading problem than Paul’s brain in a box approach? How parallellizable is FAI development? How much FAI research can be done in public? To what degree can we slap Friendliness onto arbitrary AGIs, vs. having to build systems from the ground up for Friendliness? I think we’ll mostly learn about these important questions by making FAI progress.
It’s much easier to get traction with academics with technical research. Technical research is also better for gaining prestige, which helps with outreach and recruitment. E.g. within a few hours of publishing our first math result, John Baez and Timothy Gowers were discussing it on Google+.
The startup founder’s heuristic that you should do some strategy thinking up front, but then you have to get out of your armchair and try to build the thing, and that’s how you’ll learn what’s going to work and what’s not.
The heuristic that ethics & safety thinking is best done by people who know the details of what they’re talking about, i.e. from within a science.
And, well, there’s some chance that MIRI really will need to try to build FAI itself, if nobody bigger and better-funded will. (Right now, better-funded outfits seem lackadaisical about FAI. I expect that to change in a least a few cases, but not particularly quickly. If AGI comes surprisingly soon, say in 20 years, then MIRI+FHI+friends might be our only shot at winning.)
Also note that MIRI has in fact spent most of its history on strategic research and movement-building, and now that those things are also being done pretty well by FHI, CEA, and CFAR, it makes sense for MIRI to focus on (what we think is) the most useful object-level thing (FAI research), especially since we have a comparative advantage there: Eliezer.
I really like both of your comments in this thread, Luke.
Also note that MIRI has in fact spent most of its history on strategic research and movement-building, and now that those things are also being done pretty well by FHI, CEA, and CFAR, it makes sense for MIRI to do (what we think is) the most useful object-level thing (FAI research), especially since we have a comparative advantage there (Eliezer).
I’m glad you mentioned this. I should clarify that most of my uncertainty about continuing to donate to MIRI in the future is uncertainty about donating to MIRI vs. one of these other organizations. To the extent that it’s really important to have people at Google, the DoD, etc. be safety-conscious, it think it’s possible movement building might offer better returns than technical research right now… but I’m not sure about that, and I do think the technical research is valuable.
Right; I think it’s hard to tell whether donations do more good at MIRI, FHI, CEA, or CFAR — but if someone is giving to AMF then I assume they must care only about beings who happen to be living today (a Person-Affecting View), or else they have a very different model of the world than I do, one where the value of the far future is somehow not determined by the intelligence explosion.
Edit: To clarify, this isn’t an exhaustive list. E.g. I think GiveWell’s work is also exciting, though less in need of smaller donors right now because of Good Ventures.
He’s talking specifically about people donating to AMF. There are more things people can do than donate to AMF and donate to one of MIRI, FHI, CEA, and CFAR.
Or simply because the quality of research is positively correlated with ability to secure funding, and thus research that would not be done without your donations generally has the lowest expected value of all research. In case of malaria, we need quantity, in case of AI research, we need quality.
Increasing the quality of the far future. In principle there may be some way to have a lasting impact by making society better off for the indefinite future. I tend to think this is not very unlikely; it would be surprising if a social change (other than a values change or extinction) had an impact lasting for a significant fraction of civilization’s lifespan, and indeed I haven’t seen any plausible examples of such a change.
...
I think the most promising interventions at the moment are:
Increase the profile of effective strategies for decision-making, particularly with respect to policy-making and philanthropy.
They could also reasonably believe that marginal donations to the organizations listed would not reliably influence an intelligence explosion in a way that would have significant positive impact on the value of the far future. They might also believe that AMF donations would have a greater impact on potential intelligence explosions (for example, because an intelligence explosion is so far into the future that the best way to help is to ensure human prosperity up to the point where GAI research actually becomes useful).
They might also believe that AMF donations would have a greater impact on potential intelligence explosions
It is neither probable nor plausible that AMF, a credible maximum of short-term reliable known impact on lives saved valuing all current human lives equally, should happen to also possess a maximum of expected impact on future intelligence explosions. It is as likely as that donating to your local kitten shelter should be the maximum of immediate lives saved. This kind of miraculous excuse just doesn’t happen in real life.
OK. Granted. Even a belief that the AMF is better at affecting intelligence explosions is unlikely to justify the claim that it is the best, and thus not justify the behavior described.
Amazing how even after reading all Eliezer’s posts (many more than once), I can still get surprise, insight and irony at a rate sufficient enough to produce laughter for 1+ minute.
I’m curious as to why you include CEA—my impression was that GWWC and 80k both focus on charities like AMF anyway? Is that wrong, or does CEA do more than it’s component organizations?
Perhaps because GWWC’s founder Toby Ord is part of FHI, and because CEA now shares offices with FHI, CEA is finding / producing new far future focused EAs at a faster clip than, say, GiveWell (as far as I can tell).
Correct. I doubt that the most likely winning scenario involves MIRI building most of an FAI itself. (By “MIRI” I mean to include MIRI-descendents.) I helped flip MIRI to a focus on technical research because:
Except to the degree that public FAI progress enables others to build uFAI, humanity is better off with more FAI research on hand (than will be the case without MIRI’s efforts) when it becomes clear AGI is around the corner. And right now MIRI is focused on safely publishable FAI research.
Compared to tech forecasting or philosophy research, technical research is more able to attract the attention of the literally-smartest young people in the world. Once they’re engaged, they sometimes turn their attention to the strategic issues as well, so in the long run a focus on technical research might actually be better for strategy research than a focus on strategy research is, at least now that the basics will be laid out in Bostrom’s Superintelligence book next year. Example: Paul Christiano (an IMO silver medalist who authored a paper on quantum money with Scott Aaronson before he was 20) is now one of the most useful superintelligence strategists, but was initially attracted to the area because he found the issue of cryptographic boxes for AI intellectually stimulating, and then he spent 200+ hours talking to Carl Shulman and became a good strategist.
There are several important strategic questions about which I only expect to get evidence by trying to build Friendly AI. E.g. how hard is FAI relative to AGI? What’s the distribution of eAIs around FAIs in mind design space, and how much extra optimization power do you have to apply to avoid them? How much better can we do on the value-loading problem than Paul’s brain in a box approach? How parallellizable is FAI development? How much FAI research can be done in public? To what degree can we slap Friendliness onto arbitrary AGIs, vs. having to build systems from the ground up for Friendliness? I think we’ll mostly learn about these important questions by making FAI progress.
It’s much easier to get traction with academics with technical research. Technical research is also better for gaining prestige, which helps with outreach and recruitment. E.g. within a few hours of publishing our first math result, John Baez and Timothy Gowers were discussing it on Google+.
The startup founder’s heuristic that you should do some strategy thinking up front, but then you have to get out of your armchair and try to build the thing, and that’s how you’ll learn what’s going to work and what’s not.
The heuristic that ethics & safety thinking is best done by people who know the details of what they’re talking about, i.e. from within a science.
And, well, there’s some chance that MIRI really will need to try to build FAI itself, if nobody bigger and better-funded will. (Right now, better-funded outfits seem lackadaisical about FAI. I expect that to change in a least a few cases, but not particularly quickly. If AGI comes surprisingly soon, say in 20 years, then MIRI+FHI+friends might be our only shot at winning.)
Also note that MIRI has in fact spent most of its history on strategic research and movement-building, and now that those things are also being done pretty well by FHI, CEA, and CFAR, it makes sense for MIRI to focus on (what we think is) the most useful object-level thing (FAI research), especially since we have a comparative advantage there: Eliezer.
I really like both of your comments in this thread, Luke.
I’m glad you mentioned this. I should clarify that most of my uncertainty about continuing to donate to MIRI in the future is uncertainty about donating to MIRI vs. one of these other organizations. To the extent that it’s really important to have people at Google, the DoD, etc. be safety-conscious, it think it’s possible movement building might offer better returns than technical research right now… but I’m not sure about that, and I do think the technical research is valuable.
Right; I think it’s hard to tell whether donations do more good at MIRI, FHI, CEA, or CFAR — but if someone is giving to AMF then I assume they must care only about beings who happen to be living today (a Person-Affecting View), or else they have a very different model of the world than I do, one where the value of the far future is somehow not determined by the intelligence explosion.
Edit: To clarify, this isn’t an exhaustive list. E.g. I think GiveWell’s work is also exciting, though less in need of smaller donors right now because of Good Ventures.
There is also the possibility that they believe that MIRI/FHI/CEA/CFAR will have no impact on the intelligence explosion or the far future.
He’s talking specifically about people donating to AMF. There are more things people can do than donate to AMF and donate to one of MIRI, FHI, CEA, and CFAR.
Correct.
Or simply because the quality of research is positively correlated with ability to secure funding, and thus research that would not be done without your donations generally has the lowest expected value of all research. In case of malaria, we need quantity, in case of AI research, we need quality.
Given the mention of Christiano above, I want to shout out one of his more important blog posts.
...
They could also reasonably believe that marginal donations to the organizations listed would not reliably influence an intelligence explosion in a way that would have significant positive impact on the value of the far future. They might also believe that AMF donations would have a greater impact on potential intelligence explosions (for example, because an intelligence explosion is so far into the future that the best way to help is to ensure human prosperity up to the point where GAI research actually becomes useful).
It is neither probable nor plausible that AMF, a credible maximum of short-term reliable known impact on lives saved valuing all current human lives equally, should happen to also possess a maximum of expected impact on future intelligence explosions. It is as likely as that donating to your local kitten shelter should be the maximum of immediate lives saved. This kind of miraculous excuse just doesn’t happen in real life.
OK. Granted. Even a belief that the AMF is better at affecting intelligence explosions is unlikely to justify the claim that it is the best, and thus not justify the behavior described.
Amazing how even after reading all Eliezer’s posts (many more than once), I can still get surprise, insight and irony at a rate sufficient enough to produce laughter for 1+ minute.
I’m curious as to why you include CEA—my impression was that GWWC and 80k both focus on charities like AMF anyway? Is that wrong, or does CEA do more than it’s component organizations?
Perhaps because GWWC’s founder Toby Ord is part of FHI, and because CEA now shares offices with FHI, CEA is finding / producing new far future focused EAs at a faster clip than, say, GiveWell (as far as I can tell).
I’m currently donating to FHI for the UK tax advantages, so that’s good to hear.
Bill Gates presents his rationale for attacking Malaria and Polio here.
I can’t make much sense of it personally—but at least he isn’t working on stopping global warming.