It’s not clear that funding SIAI and FHI has positive expected value.
Perhaps I should ask for clarification of what you mean by “not clear that X has positive expected value.” Relative to what alternative?
Also, you shouldn’t say “not clear whether the expected value off X is positive, negative, or zero”—relative to a particular probability distribution, the expected value of something has a definite value.
You might express what you want to say better by saying “I currently think that X has positive expected value, but I expect my beliefs to change very rapidly with incoming evidence”.
This is an example of mistaking instability of a subjective probability with uncertainty about a subjective probability (there is no uncertainty about your subjective probabilities).
Okay, fine: I currently believe that funding SIAI and FHI has expected value near zero but my belief on this matter is unstable and subject to rapid change with incoming evidence.
As I see it, most of current worth of SIAI is in focusing attention on the problem of FAI, and it doesn’t need to produce any actual research on AI to make progress on that goal. The mere presence of this organization allows people like me to (1) recognize the problem of FAI, something you are unlikely to figure out or see as important on your own and (2) see the level of support for the cause, and as a result be more comfortable about seriously devoting time to studying the problem (in particular, extensive discussion by many smart people on Less Wrong and elsewhere gives more confidence that the idea is not a mirage).
Initially, most of the progress in this direction was produced personally by Eliezer, but now SIAI is strong enough to carry on. Publicity causes more people to seriously think about the problem, which will eventually lead to technical progress, if it’s possible at all, regardless of whether current SIAI is capable of making that progress.
This makes current SIAI clearly valuable, because whatever is the truth about possible paths towards FAI, it takes a significant effort to explore them, and SIAI calls attention to that task. If SIAI can make progress on the technical problem as well, more power to them. If other people begin to make technical progress, they now have the option of affiliating with SIAI, which might be a significant improvement over personally trying to fight for funding on FAI research.
Not all publicity is good publicity. The majority of people who I’ve met off of Less Wrong who have heard of SIAI think that the organization is full of crazy people. A lot of these people are smart. Some of these people have Ph.D.’s from top tier universities in sciences.
I think that SIAI should be putting way more emphasis on PR, networking within academic, etc. This is in consonance with a comment by Holden Karnofsky here
To the extent that your activities will require “beating” other organizations (in advocacy, in speed of innovation, etc.), what are the skills and backgrounds of your staffers that are relevant to their ability to do this?
I’m worried that SIAI’s poor ability to make a good public impression may poison the cause of existential risk in the mind of the public and dissuade good researchers from studying existential risk. There are some very smart people who it would be good to have working on Friendly AI who, despite their capabilities, care a lot about their status in broader society. I think that it’s very important that an organization that works toward Friendly AI at least be well regarded by a sizable minority people in the scientific community.
In my experience, academics often cannot distinguish between SIAI and Kurzweil-related activities such as the Singularity University. With its 25k tuition for two months, SU is viewed as some sort of scam, and Kurzweilian ideas of exponential change are seen as naive. People hear about Kurzweil, SU, the Singularity Summit, and the Singularity Institute, and assume that the latter is behind all those crazy singularity things.
We need to make it easier to distinguish the preference and decision theory research program as an attempt to solve a hard problem from the larger cluster of singularity ideas, which, even in the intelligence explosion variety, are not essential.
Agreed. I’m often somewhat embarrassed to mention SIAI’s full name, or the Singularity Summit, because of the term “singularity” which, in many people’s minds—to some extent including my own—is a red flag for “crazy”.
Honestly, even the “Artificial Intelligence” part of the name can misrepresent what SIAI is about. I would describe the organization as just “a philosophy institute researching hugely important fundamental questions.”
Agreed. I’m often somewhat embarrassed to mention SIAI’s full name, or the Singularity Summit, because of the term “singularity” which, in many people’s minds—to some extent including my own—is a red flag for “crazy”.
Agreed; I’ve had similar thoughts. Given recent popular coverage of the various things called “the Singularity”, I think we need to accept that it’s pretty much going to become a connotational dumping ground for every cool-sounding futuristic prediction that anyone can think of, centered primarily around Kurzweil’s predictions.
Honestly, even the “Artificial Intelligence” part of the name can misrepresent what SIAI is about. I would describe the organization as just “a philosophy institute researching hugely important fundamental questions.”
I disagree somewhat there. Its ultimate goal is still to create a Friendly AI, and all of its other activities (general existential risk reduction and forecasting, Less Wrong, the Singularity Summit, etc.) are, at least in principle, being carried out in service of that goal. Its day-to-day activities may not look like what people might imagine when they think of an AI research institute, but that’s because FAI is a very difficult problem with many prerequisites that have to be solved first, and I think it’s fair to describe SIAI as still being fundamentally about FAI (at least to anyone who’s adequately prepared to think about FAI).
Describing it as “a philosophy institute researching hugely important fundamental questions” may give people the wrong impressions, if it’s not quickly followed by more specific explanation. When people think of “philosophy” + “hugely important fundamental questions”, their minds will probably leap to questions which are 1) easily solved by rationalists, and/or 2) actually fairly silly and not hugely important at all. (“Philosophy” is another term I’m inclined toward avoiding these days.) When I’ve had to describe SIAI in one phrase to people who have never heard of it, I’ve been calling it an “artificial intelligence think-tank”. Meanwhile, Michael Vassar’s Twitter describes SIAI as a “decision theory think-tank”. That’s probably a good description if you want to address the current focus of their research; it may be especially good in academic contexts, where “decision theory” already refers to an interesting established field that’s relevant to AI but doesn’t share with “artificial intelligence” the connotations of missed goals, science fiction geekery, anthropomorphism, etc.
Ah, I think I can guess who you are. You work under a professor called Josh and have an umlaut in your surname. Shame that the others in that great research group don’t take you seriously.
I’m pretty sure usable suggestions for improvement are welcome. About ten years ago there was only the irrational version of Eliezer who just recently understood that the problem existed, while right now we have some non-crazy introductory and scholary papers, and a community that understands the problem. The progress seems to be in the right direction.
If you asked the same people about the idea of FAI fifteen years ago, say, they’d label it crazy just the same. SIAI gets labeled automatically, by association with the idea. Perceived craziness is the default we must push the public perception away from, not something initiated by actions of SIAI (you’d need to at least point out specific actions to attempt this argument).
Good point—I will write to SIAI about this matter.
I actually agree that up until this point progress has been in the right direction, I guess my thinking is that the SIAI has attracted a community consisting of a very particular kind of person, may have achieved near-saturation within this population, and that consequently SIAI as presently constituted may have outlived the function that you mention. This is the question of room for more funding
Agree with
Perceived craziness is the default we must push the public perception away from, not something initiated by actions of SIAI (you’d need to at least point out specific actions to attempt this argument).
There are things that I have in mind but I prefer to contact SIAI about them directly before discussing them in public.
I think there are many people who worry about AI in one form or another. They may not do very informed worrying and they may be anthropomorphising, but they still worry and that might be harnessable. See Stephen Hawkings on AI.
SIAIs emphasis on the singularity aspect of the possible dangers of AI is unfortunate as it requires people to get their heads around this. So it alienates the people who just worry about the robot uprising or their jobs being stolen and being outcompeted evolutionarily.
So lets say instead of SIAI you had IRDAI (Institute to research the Dangers of AI). It could look at each potential AI and assess the various risks each architecture posed. It could practice on things like feed forward neural networks and say what types of danger they might pose (job stealing, being rooted and used by a hacker, or going FOOM), based on their information theoretical ability to learn from different information sources, security model and the care being take to make sure human values are embedded in it. In the process of doing that it would have to develop theories of FAI in order to say whether a system was going to have human-like values stably.
The emphasis placed upon very hard take off just makes it less approachable and look more wacky to the casual observer.
Safe robots have nothing whatsoever to do with FAI. Saying otherwise would be incompetent, or a lie. I believe that there need not be an emphasis of hard takeoff, but likely for reasons not related to yours.
Agreed. My dissertation is on moral robots, and one of the early tasks was examining SIAI and FAI and determining that the work was pretty much unrelated (I presented a pretty bad conference paper on the topic).
Apart from they both need a fair amount of computer science to predict their capabilities and dangers?
Call your research institute something like the Institute for the prevention of Advanced Computational Threats, and have separate divisions for robotics and FAI. Gain the trust of the average scientist/technology aware person by doing a good job on robotics and they are more likely to trust you when it comes to FAI.
Apart from they both need a fair amount of computer science to predict their capabilities and dangers?
I recently shifted to believing that pure mathematics is more relevant for FAI than computer science.
Call your research institute something like the Institute for the prevention of Advanced Computational Threats, and have separate divisions for robotics and FAI. Gain the trust of the average scientist/technology aware person by doing a good job on robotics and they are more likely to trust you when it comes to FAI.
In FAI, the central question is what a program wants (which is a certain kind of question about what the program means), and not what a program does.
Computer science will tell lots about what which programs can do how, and how to construct a program that does what you need, but less about what a program means (the sort of computer science that does is already a fair distance towards mathematics). This is also a problem with statistics/machine learning, and the reason they are not particularly useful for FAI: they teach certain tools, and how these tools work, but understanding they provide isn’t portable enough.
Mathematical logic, on the other hand, contains lots of wisdom in the right direction: what kinds of mathematical structures can be defined how, which structures a given definition defines, what concepts are definable, and so on. And to understands the concepts themselves one needs to go further.
Unfortunately, I can’t give a good positive argument for the importance of math; that would require a useful insight (arrived at through use of mathematical tools). At the least, I can attest finding a lot of confusion in my past thinking about FAI as a result of each “level up” in understanding of mathematics, and that counts for something.
Still not good enough. $10000000000 is near zero, for some definition of “near”. Why not just give a dollar value? I think people have a strong fear of being accused of spurious precision, but really, giving a precise number and then saying “unstable with a standard deviation of X per hour of debate” is the only mathematically consistent way of saying what you want to say.
Okay, let’s try again. My current belief is that at present, donations to SIAI are a less cost effective way of accomplishing good than donating to a charity like VillageReach or StopTB which improves health in the developing world.
My internal reasoning is as follows:
Roughly speaking the potential upside of donating to SIAI (whatever research SIAI would get done) is outwieghed by the potential downside (the fact that SIAI could divert funding away from future existential risk organizations). By way of contrast, I’m reasonably confident that there’s some upside to improving health in the developing world (keep in mind that historically, development has been associated with political stability and getting more smart people in the pool of people thinking about worthwhile things) and giving to accountable effectiveness oriented organizations will raise the standard for accountability across the philanthropic world (including existential risk charities).
I wish that there were better donation opportunities than VillageReach and StopTB and I’m moderately optimistic that some will emerge in the near future (e.g. over the next ten years) but I don’t see any at the moment.
So we both agree that a more-accountable set of existential risk organizations would (all else equal) be the best way to spend money, better than third world charity certainly.
The disagreement is about this idea of current existential risk organizations diverting money away from future organizations that are better.
My impression is that existential risk charity is very much unlike third-world aid charity, in that how to deliver third world aid is not a philosophically challenging problem. Everyone has a good intuitive understanding of people, of food and the lack thereof, and at least some understanding of things like incentive problems.
However, something like Friendly AI theory requires a virtually complete re-education of a person (that is if they are very smart to start with. If not, they’ll just never understand it). If it were easy to understand, it would be something for which charity was not required: governments would be doing it, not out of charity, but out of self-interest.
Given this difference, your idea of demanding high levels of accountability might itself need some scrutiny. My personal position is to require nothing in terms of accountability, competence or performance unless and until it is demonstrated that there are, in fact, other groups who want to start an existential risk charity, and to begin the process of competition by funding those other groups, should they in fact arise.
I am currently working for the Lifeboat foundation, by the way, which is such an “other group”, and is, in fact, funded to the tune of $200k. But three is still pretty darn small, and the number of people involved is tiny.
•I think that at the margin a highly accountable existential risk charity would definitely be better than a third world charity. I could imagine that if a huge amount of money were being flooded into the study of existential risk, it would be more cost effective to send money to the developing world.
•I’m very familiar with pure mathematics. My belief is that in pure mathematics the variability in productivity of researchers stretches over many orders of magnitude. By analogy, I would guess that the productivity of Friendly AI researchers will also differ by many orders of magnitude. I suspect that the current SIAI researchers are not at the high end of this range (out of virtue of the fact that the most talented researchers are very rare, very few people are currently thinking about these things, and my belief that the correlation between currently thinking about these things and having talent is weak).
Moreover, I think that if a large community of people who value Friendly AI research emerges, there will be positive network effects that heighten the productivity of the researchers.
For these reasons, I think that the expected value of the research that SIAI is doing is negligible in comparison with the expected value of the publicity that SIAI generates. At the margin, I’m not convinced that SIAI is generating good publicity for the cause of existential risk. I think that SIAI may be generating bad publicity for the cause of existential risk. See my exchange with Vladimir Nesov. Aside from the general issue of it being good to encourage accountability, this is why I don’t think that funding SIAI is a good idea right now. But as I said to Vladimir Nesov, I will write to SIAI about this and see what happens.
•I think that the reason that governments are not researching existential risk and artificial intelligence is because (a) the actors involved in governments are shortsighted and (b) the public doesn’t demand that governments research these things. It seems quite possible to me that in the future governments will put large amounts of funding into these things.
I think that the reason that governments are not researching existential risk and artificial intelligence is because (a) the actors involved in governments are shortsighted and (b) the public doesn’t demand that governments research these things. It seems quite possible to me that in the future governments will put large amounts of funding into these things.
Maybe, but more likely rich individuals will see the benefits long before the public does, then the “establishment” will organize a secret AGI project. Though this doesn’t even seem remotely close to happening: the whole thing pattern matches for some kind of craziness/scam.
•I agree that there’s gap between when rich individuals see the benefits of existential risk research and when the general public sees the benefits of existential risk research.
•The gap may nevertheless be inconsequential relative to the time that it will take to build a general AI.
•I presently believe that it’s not desirable for general AI research to be done in secret. Secret research proceeds slower than open research, and we may be “on the clock” because of existential risks unrelated to general AI. In my mind this factor outweighs the arguments that Eliezer has advanced for general AI research being done in secret.
I presently believe that it’s not desirable for general AI research to be done in secret.
There are shades between complete secrecy and blurting it out on the radio. Right now, human-universal cognitive biases keep it effectively secret, but in the future we may find that the military closes in on it like knowledge of how to build nuclear weapons.
That, and secrets are damn hard to keep. In all of history, there has only been one military secret that has never been exposed, and that’s the composition of Greek fire. Someone is going to leak.
Moreover, I think that if a large community of people who value Friendly AI research emerges, there will be positive network effects that heighten the productivity of the researchers.
Note that if uFAI is >> easier than FAI, then the size of the research community must be kept small, otherwise FAI research may acquire a Klaus Fuchs who goes and builds a uFAI for fun and vengeance.
I think that at the margin a highly accountable existential risk charity would definitely be better than a third world charity. I could imagine that if a huge amount of money were being flooded into the study of existential risk, it would be more cost effective to send money to the developing world.
Do you buy the argument that we should take the ~10^50 future people the universe could support into account in our expected utility calculations?
If so, then it is hard to see how anything other than existential risks matters, i.e. all money devoted to the third world, animal welfare, poor people, diseases, etc, would ideally be redirected to the goal of ensuring a positive (rather then negative) singularity.
Of course this point is completely academic, because the vast majority of people won’t ever believe it, but I’d be interested to hear if you buy it.
Do you buy the argument that we should take the ~10^50 future people the universe could support into our expected utility calculations?
Yes, I buy this argument.
If so, then it is hard to see how anything other than existential risks matters.
The question is just whether donating to an existential risk charity is the best way to avert existential risk.
•I believe that political instability is conducive to certain groups desperately racing to produce and utilize powerful technologies. This points in the direction of promotion of political stability reducing existential risk.
•I believe that when people are leading lives that they find more fulfilling, they make better decisions, so that improving quality of life reduces existential risk
•I believe that (all else being equal), economic growth reduces “existential risk in the broad sense.” By this I mean that economic growth may prevent astronomical waste.
Of course, as a heuristic it’s more important that technologies develop safely than that they develop quickly, but one could still imagine that at some point, the marginal value of an extra dollar spent on existential risk research drops so low that speeding up economic growth is a better use of money.
•Of the above three points, the first two are more compelling than the third, but the third could still play a role, and I believe that there’s a correlation between each pair of political stability, quality of life, and economic growth, so that it’s possible to address the three simultaneously.
•As I said above, at the margin I think that a good charity devoted to studying existential risk should be getting more funding, but at present I do not believe that a good charity devoted to studying existential risk could cost effectively absorb arbitrarily many dollars.
Do you buy the argument that we should take the ~10^50 future people the universe could support into account in our expected utility calculations?
I do. In fact, I assign a person certain to be born a million years from now about the same intrinsic value as a person who exists today though there are a lot of ways in which my doing good for a person who exist today has significant insttrumental value which doing good for a person certain to be born a million years does not.
My impression is that existential risk charity is very much unlike third-world aid charity, in that how to deliver third world aid is not a philosophically challenging problem. Everyone has a good intuitive understanding of people, of food and the lack thereof, and at least some understanding of things like incentive problems.
I suspect helping dead states efficiently and sustainably is very difficult, possibly more so than developing FAI as a shortcut. Of course, it’s a completely different kind of challenge.
I disagree strongly. You can repeatedly get it it wrong with failed states, and learn from your mistakes. The utility cost for each failure is additive, whereas the first FAI failure is fatal. Also, third world development is a process that might spontaneously solve itself via economic development and cultural change. Much to the chagrin of many charities, that might even be the optimal way to solve the problem given our resource constraints. In fact the development of the west is a particular example of this; we started out as medieval third world nations.
I disagree strongly. You can repeatedly get it it wrong with failed states, and learn from your mistakes. The utility cost for each failure is additive, whereas the first FAI failure is fatal.
Distinguish the difficulty of developing an adequate theory, from the difficulty of verifying that a theory is adequate. It’s the failure with the latter that might lead to disaster, while not failing requires a lot of informed rational caution. On the other hand, not inventing an adequate theory doesn’t directly lead to a disaster, and failure to invent an adequate theory of FAI is something you can learn from (the story of my life for the last three years).
I read “not clear that X has positive expected value” as something like “I’m not sure an observer with perfect knowledge of all relevant information, but not of future outcomes would assign X a positive expected value.”
observer with perfect knowledge of all relevant information, but not of future outcomes
Nonsense!
In any case, trying to guess what variously omniscient yet handicapped ideal observers would say is a dumb way to do decision theory; just be a bayesian with a subjective probability.
To clarify: No knowledge of things like the state of individual electrons or photons, and therefore no knowledge of future “random” (chaos theory) outcomes. This was one of the possible objections I had considered, but decided against addressing in advance, turns out I should have.
Logical uncertainty is also something you must fight on your own. Like you can’t know what’s actually in the world, if you haven’t seen it, you can’t know what logically follows from what you know, if you didn’t perform the computation.
And that was the other possible objection I had thought of!
I had meant to include that sort of thing in “relevant knowledge”, but couldn’t think of any good way to phase it in the 5 seconds I thought about it. I wasn’t trying to make any important argument, it was just a throwaway comment.
I commented on the objection that being unsure whether the expected value of something is positive conflicts with the definition of expected value with:
I read “not clear that X has positive expected value” as something like “I’m not sure an observer with perfect knowledge of all relevant information, but not of future outcomes would assign X a positive expected value.”
When writing this I thought of two possible objections/comments/requests for clarification/whatever:
That perfect knowledge implies knowledge of future outcomes.
Your logical uncertainty point (though I had no good way to phrase this).
I briefly considered addressing them in advance, but decided against it. Both whatevers were made in fairly rapid succession (though yours apparently not with that comment in mind?), so I definitely should have.
There is no way that short throwaway comment deserved a seven post comment thread.
Perhaps I should ask for clarification of what you mean by “not clear that X has positive expected value.” Relative to what alternative?
Also, you shouldn’t say “not clear whether the expected value off X is positive, negative, or zero”—relative to a particular probability distribution, the expected value of something has a definite value.
You might express what you want to say better by saying “I currently think that X has positive expected value, but I expect my beliefs to change very rapidly with incoming evidence”.
This is an example of mistaking instability of a subjective probability with uncertainty about a subjective probability (there is no uncertainty about your subjective probabilities).
Okay, fine: I currently believe that funding SIAI and FHI has expected value near zero but my belief on this matter is unstable and subject to rapid change with incoming evidence.
As I see it, most of current worth of SIAI is in focusing attention on the problem of FAI, and it doesn’t need to produce any actual research on AI to make progress on that goal. The mere presence of this organization allows people like me to (1) recognize the problem of FAI, something you are unlikely to figure out or see as important on your own and (2) see the level of support for the cause, and as a result be more comfortable about seriously devoting time to studying the problem (in particular, extensive discussion by many smart people on Less Wrong and elsewhere gives more confidence that the idea is not a mirage).
Initially, most of the progress in this direction was produced personally by Eliezer, but now SIAI is strong enough to carry on. Publicity causes more people to seriously think about the problem, which will eventually lead to technical progress, if it’s possible at all, regardless of whether current SIAI is capable of making that progress.
This makes current SIAI clearly valuable, because whatever is the truth about possible paths towards FAI, it takes a significant effort to explore them, and SIAI calls attention to that task. If SIAI can make progress on the technical problem as well, more power to them. If other people begin to make technical progress, they now have the option of affiliating with SIAI, which might be a significant improvement over personally trying to fight for funding on FAI research.
Not all publicity is good publicity. The majority of people who I’ve met off of Less Wrong who have heard of SIAI think that the organization is full of crazy people. A lot of these people are smart. Some of these people have Ph.D.’s from top tier universities in sciences.
I think that SIAI should be putting way more emphasis on PR, networking within academic, etc. This is in consonance with a comment by Holden Karnofsky here
To the extent that your activities will require “beating” other organizations (in advocacy, in speed of innovation, etc.), what are the skills and backgrounds of your staffers that are relevant to their ability to do this?
I’m worried that SIAI’s poor ability to make a good public impression may poison the cause of existential risk in the mind of the public and dissuade good researchers from studying existential risk. There are some very smart people who it would be good to have working on Friendly AI who, despite their capabilities, care a lot about their status in broader society. I think that it’s very important that an organization that works toward Friendly AI at least be well regarded by a sizable minority people in the scientific community.
In my experience, academics often cannot distinguish between SIAI and Kurzweil-related activities such as the Singularity University. With its 25k tuition for two months, SU is viewed as some sort of scam, and Kurzweilian ideas of exponential change are seen as naive. People hear about Kurzweil, SU, the Singularity Summit, and the Singularity Institute, and assume that the latter is behind all those crazy singularity things.
We need to make it easier to distinguish the preference and decision theory research program as an attempt to solve a hard problem from the larger cluster of singularity ideas, which, even in the intelligence explosion variety, are not essential.
Agreed. I’m often somewhat embarrassed to mention SIAI’s full name, or the Singularity Summit, because of the term “singularity” which, in many people’s minds—to some extent including my own—is a red flag for “crazy”.
Honestly, even the “Artificial Intelligence” part of the name can misrepresent what SIAI is about. I would describe the organization as just “a philosophy institute researching hugely important fundamental questions.”
Agreed; I’ve had similar thoughts. Given recent popular coverage of the various things called “the Singularity”, I think we need to accept that it’s pretty much going to become a connotational dumping ground for every cool-sounding futuristic prediction that anyone can think of, centered primarily around Kurzweil’s predictions.
I disagree somewhat there. Its ultimate goal is still to create a Friendly AI, and all of its other activities (general existential risk reduction and forecasting, Less Wrong, the Singularity Summit, etc.) are, at least in principle, being carried out in service of that goal. Its day-to-day activities may not look like what people might imagine when they think of an AI research institute, but that’s because FAI is a very difficult problem with many prerequisites that have to be solved first, and I think it’s fair to describe SIAI as still being fundamentally about FAI (at least to anyone who’s adequately prepared to think about FAI).
Describing it as “a philosophy institute researching hugely important fundamental questions” may give people the wrong impressions, if it’s not quickly followed by more specific explanation. When people think of “philosophy” + “hugely important fundamental questions”, their minds will probably leap to questions which are 1) easily solved by rationalists, and/or 2) actually fairly silly and not hugely important at all. (“Philosophy” is another term I’m inclined toward avoiding these days.) When I’ve had to describe SIAI in one phrase to people who have never heard of it, I’ve been calling it an “artificial intelligence think-tank”. Meanwhile, Michael Vassar’s Twitter describes SIAI as a “decision theory think-tank”. That’s probably a good description if you want to address the current focus of their research; it may be especially good in academic contexts, where “decision theory” already refers to an interesting established field that’s relevant to AI but doesn’t share with “artificial intelligence” the connotations of missed goals, science fiction geekery, anthropomorphism, etc.
Ah, I think I can guess who you are. You work under a professor called Josh and have an umlaut in your surname. Shame that the others in that great research group don’t take you seriously.
I’m pretty sure usable suggestions for improvement are welcome. About ten years ago there was only the irrational version of Eliezer who just recently understood that the problem existed, while right now we have some non-crazy introductory and scholary papers, and a community that understands the problem. The progress seems to be in the right direction.
If you asked the same people about the idea of FAI fifteen years ago, say, they’d label it crazy just the same. SIAI gets labeled automatically, by association with the idea. Perceived craziness is the default we must push the public perception away from, not something initiated by actions of SIAI (you’d need to at least point out specific actions to attempt this argument).
Good point—I will write to SIAI about this matter.
I actually agree that up until this point progress has been in the right direction, I guess my thinking is that the SIAI has attracted a community consisting of a very particular kind of person, may have achieved near-saturation within this population, and that consequently SIAI as presently constituted may have outlived the function that you mention. This is the question of room for more funding
Agree with
There are things that I have in mind but I prefer to contact SIAI about them directly before discussing them in public.
I think there are many people who worry about AI in one form or another. They may not do very informed worrying and they may be anthropomorphising, but they still worry and that might be harnessable. See Stephen Hawkings on AI.
SIAIs emphasis on the singularity aspect of the possible dangers of AI is unfortunate as it requires people to get their heads around this. So it alienates the people who just worry about the robot uprising or their jobs being stolen and being outcompeted evolutionarily.
So lets say instead of SIAI you had IRDAI (Institute to research the Dangers of AI). It could look at each potential AI and assess the various risks each architecture posed. It could practice on things like feed forward neural networks and say what types of danger they might pose (job stealing, being rooted and used by a hacker, or going FOOM), based on their information theoretical ability to learn from different information sources, security model and the care being take to make sure human values are embedded in it. In the process of doing that it would have to develop theories of FAI in order to say whether a system was going to have human-like values stably.
The emphasis placed upon very hard take off just makes it less approachable and look more wacky to the casual observer.
Safe robots have nothing whatsoever to do with FAI. Saying otherwise would be incompetent, or a lie. I believe that there need not be an emphasis of hard takeoff, but likely for reasons not related to yours.
Agreed. My dissertation is on moral robots, and one of the early tasks was examining SIAI and FAI and determining that the work was pretty much unrelated (I presented a pretty bad conference paper on the topic).
Apart from they both need a fair amount of computer science to predict their capabilities and dangers?
Call your research institute something like the Institute for the prevention of Advanced Computational Threats, and have separate divisions for robotics and FAI. Gain the trust of the average scientist/technology aware person by doing a good job on robotics and they are more likely to trust you when it comes to FAI.
I recently shifted to believing that pure mathematics is more relevant for FAI than computer science.
A truly devious plan.
That’s interesting. What’s your line of thought?
In FAI, the central question is what a program wants (which is a certain kind of question about what the program means), and not what a program does.
Computer science will tell lots about what which programs can do how, and how to construct a program that does what you need, but less about what a program means (the sort of computer science that does is already a fair distance towards mathematics). This is also a problem with statistics/machine learning, and the reason they are not particularly useful for FAI: they teach certain tools, and how these tools work, but understanding they provide isn’t portable enough.
Mathematical logic, on the other hand, contains lots of wisdom in the right direction: what kinds of mathematical structures can be defined how, which structures a given definition defines, what concepts are definable, and so on. And to understands the concepts themselves one needs to go further.
Unfortunately, I can’t give a good positive argument for the importance of math; that would require a useful insight (arrived at through use of mathematical tools). At the least, I can attest finding a lot of confusion in my past thinking about FAI as a result of each “level up” in understanding of mathematics, and that counts for something.
I think that’s a clever idea that deserves more eyeballs.
Nothing whatsoever is a bit strong. About as much as preventing tiger attacks and fighting malaria, perhaps?
Saving tigers from killer robots.
This video addresses this question : Anna Salamon’s 2nd Talk at Singularity Summit 2009 -- How Much it Matters to Know What Matters: A Back of the Envelope Calculation
It is 15 minutes long, but you can take a look at 11m37s
Edit : added the name of the video, thanks for the remark Vladimir.
The link above is Anna Salamon’s 2nd Talk at Singularity Summit 2009 “How Much it Matters to Know What Matters: A Back of the Envelope Calculation.”
(You should give some hint of the content of a link you give, at least the title of the talk.)
Still not good enough. $10000000000 is near zero, for some definition of “near”. Why not just give a dollar value? I think people have a strong fear of being accused of spurious precision, but really, giving a precise number and then saying “unstable with a standard deviation of X per hour of debate” is the only mathematically consistent way of saying what you want to say.
Plus, I can’t disagree with your statement.
Okay, let’s try again. My current belief is that at present, donations to SIAI are a less cost effective way of accomplishing good than donating to a charity like VillageReach or StopTB which improves health in the developing world.
My internal reasoning is as follows:
Roughly speaking the potential upside of donating to SIAI (whatever research SIAI would get done) is outwieghed by the potential downside (the fact that SIAI could divert funding away from future existential risk organizations). By way of contrast, I’m reasonably confident that there’s some upside to improving health in the developing world (keep in mind that historically, development has been associated with political stability and getting more smart people in the pool of people thinking about worthwhile things) and giving to accountable effectiveness oriented organizations will raise the standard for accountability across the philanthropic world (including existential risk charities).
I wish that there were better donation opportunities than VillageReach and StopTB and I’m moderately optimistic that some will emerge in the near future (e.g. over the next ten years) but I don’t see any at the moment.
What about the comparison of a donor advised existential risks fund versus StopTB?
Good question. I haven’t considered this point—thanks for bringing it to my consideration!
So we both agree that a more-accountable set of existential risk organizations would (all else equal) be the best way to spend money, better than third world charity certainly.
The disagreement is about this idea of current existential risk organizations diverting money away from future organizations that are better.
My impression is that existential risk charity is very much unlike third-world aid charity, in that how to deliver third world aid is not a philosophically challenging problem. Everyone has a good intuitive understanding of people, of food and the lack thereof, and at least some understanding of things like incentive problems.
However, something like Friendly AI theory requires a virtually complete re-education of a person (that is if they are very smart to start with. If not, they’ll just never understand it). If it were easy to understand, it would be something for which charity was not required: governments would be doing it, not out of charity, but out of self-interest.
Given this difference, your idea of demanding high levels of accountability might itself need some scrutiny. My personal position is to require nothing in terms of accountability, competence or performance unless and until it is demonstrated that there are, in fact, other groups who want to start an existential risk charity, and to begin the process of competition by funding those other groups, should they in fact arise.
I am currently working for the Lifeboat foundation, by the way, which is such an “other group”, and is, in fact, funded to the tune of $200k. But three is still pretty darn small, and the number of people involved is tiny.
•I think that at the margin a highly accountable existential risk charity would definitely be better than a third world charity. I could imagine that if a huge amount of money were being flooded into the study of existential risk, it would be more cost effective to send money to the developing world.
•I’m very familiar with pure mathematics. My belief is that in pure mathematics the variability in productivity of researchers stretches over many orders of magnitude. By analogy, I would guess that the productivity of Friendly AI researchers will also differ by many orders of magnitude. I suspect that the current SIAI researchers are not at the high end of this range (out of virtue of the fact that the most talented researchers are very rare, very few people are currently thinking about these things, and my belief that the correlation between currently thinking about these things and having talent is weak).
Moreover, I think that if a large community of people who value Friendly AI research emerges, there will be positive network effects that heighten the productivity of the researchers.
For these reasons, I think that the expected value of the research that SIAI is doing is negligible in comparison with the expected value of the publicity that SIAI generates. At the margin, I’m not convinced that SIAI is generating good publicity for the cause of existential risk. I think that SIAI may be generating bad publicity for the cause of existential risk. See my exchange with Vladimir Nesov. Aside from the general issue of it being good to encourage accountability, this is why I don’t think that funding SIAI is a good idea right now. But as I said to Vladimir Nesov, I will write to SIAI about this and see what happens.
•I think that the reason that governments are not researching existential risk and artificial intelligence is because (a) the actors involved in governments are shortsighted and (b) the public doesn’t demand that governments research these things. It seems quite possible to me that in the future governments will put large amounts of funding into these things.
•Thanks for mentioning the Lifeboat foundation.
Maybe, but more likely rich individuals will see the benefits long before the public does, then the “establishment” will organize a secret AGI project. Though this doesn’t even seem remotely close to happening: the whole thing pattern matches for some kind of craziness/scam.
•I agree that there’s gap between when rich individuals see the benefits of existential risk research and when the general public sees the benefits of existential risk research.
•The gap may nevertheless be inconsequential relative to the time that it will take to build a general AI.
•I presently believe that it’s not desirable for general AI research to be done in secret. Secret research proceeds slower than open research, and we may be “on the clock” because of existential risks unrelated to general AI. In my mind this factor outweighs the arguments that Eliezer has advanced for general AI research being done in secret.
There are shades between complete secrecy and blurting it out on the radio. Right now, human-universal cognitive biases keep it effectively secret, but in the future we may find that the military closes in on it like knowledge of how to build nuclear weapons.
That, and secrets are damn hard to keep. In all of history, there has only been one military secret that has never been exposed, and that’s the composition of Greek fire. Someone is going to leak.
Note that if uFAI is >> easier than FAI, then the size of the research community must be kept small, otherwise FAI research may acquire a Klaus Fuchs who goes and builds a uFAI for fun and vengeance.
This makes it all a lot harder.
Do you buy the argument that we should take the ~10^50 future people the universe could support into account in our expected utility calculations?
If so, then it is hard to see how anything other than existential risks matters, i.e. all money devoted to the third world, animal welfare, poor people, diseases, etc, would ideally be redirected to the goal of ensuring a positive (rather then negative) singularity.
Of course this point is completely academic, because the vast majority of people won’t ever believe it, but I’d be interested to hear if you buy it.
Yes, I buy this argument.
The question is just whether donating to an existential risk charity is the best way to avert existential risk.
•I believe that political instability is conducive to certain groups desperately racing to produce and utilize powerful technologies. This points in the direction of promotion of political stability reducing existential risk.
•I believe that when people are leading lives that they find more fulfilling, they make better decisions, so that improving quality of life reduces existential risk
•I believe that (all else being equal), economic growth reduces “existential risk in the broad sense.” By this I mean that economic growth may prevent astronomical waste.
Of course, as a heuristic it’s more important that technologies develop safely than that they develop quickly, but one could still imagine that at some point, the marginal value of an extra dollar spent on existential risk research drops so low that speeding up economic growth is a better use of money.
•Of the above three points, the first two are more compelling than the third, but the third could still play a role, and I believe that there’s a correlation between each pair of political stability, quality of life, and economic growth, so that it’s possible to address the three simultaneously.
•As I said above, at the margin I think that a good charity devoted to studying existential risk should be getting more funding, but at present I do not believe that a good charity devoted to studying existential risk could cost effectively absorb arbitrarily many dollars.
I do. In fact, I assign a person certain to be born a million years from now about the same intrinsic value as a person who exists today though there are a lot of ways in which my doing good for a person who exist today has significant insttrumental value which doing good for a person certain to be born a million years does not.
I suspect helping dead states efficiently and sustainably is very difficult, possibly more so than developing FAI as a shortcut. Of course, it’s a completely different kind of challenge.
I disagree strongly. You can repeatedly get it it wrong with failed states, and learn from your mistakes. The utility cost for each failure is additive, whereas the first FAI failure is fatal. Also, third world development is a process that might spontaneously solve itself via economic development and cultural change. Much to the chagrin of many charities, that might even be the optimal way to solve the problem given our resource constraints. In fact the development of the west is a particular example of this; we started out as medieval third world nations.
Distinguish the difficulty of developing an adequate theory, from the difficulty of verifying that a theory is adequate. It’s the failure with the latter that might lead to disaster, while not failing requires a lot of informed rational caution. On the other hand, not inventing an adequate theory doesn’t directly lead to a disaster, and failure to invent an adequate theory of FAI is something you can learn from (the story of my life for the last three years).
I read “not clear that X has positive expected value” as something like “I’m not sure an observer with perfect knowledge of all relevant information, but not of future outcomes would assign X a positive expected value.”
Nonsense!
In any case, trying to guess what variously omniscient yet handicapped ideal observers would say is a dumb way to do decision theory; just be a bayesian with a subjective probability.
To clarify: No knowledge of things like the state of individual electrons or photons, and therefore no knowledge of future “random” (chaos theory) outcomes. This was one of the possible objections I had considered, but decided against addressing in advance, turns out I should have.
Logical uncertainty is also something you must fight on your own. Like you can’t know what’s actually in the world, if you haven’t seen it, you can’t know what logically follows from what you know, if you didn’t perform the computation.
And that was the other possible objection I had thought of!
I had meant to include that sort of thing in “relevant knowledge”, but couldn’t think of any good way to phase it in the 5 seconds I thought about it. I wasn’t trying to make any important argument, it was just a throwaway comment.
I don’t understand what this refers to. (Objection to what? What objection? In what context did you think of it?)
I commented on the objection that being unsure whether the expected value of something is positive conflicts with the definition of expected value with:
When writing this I thought of two possible objections/comments/requests for clarification/whatever:
That perfect knowledge implies knowledge of future outcomes.
Your logical uncertainty point (though I had no good way to phrase this).
I briefly considered addressing them in advance, but decided against it. Both whatevers were made in fairly rapid succession (though yours apparently not with that comment in mind?), so I definitely should have.
There is no way that short throwaway comment deserved a seven post comment thread.