Hello everyone,
Like many people, I come to this site via an interest in transhumanism, although it seems unlikely to me that FAI implementing CEV can actually be designed before the singularity (I can explain why, and possibly even what could be done instead, but it suddenly occurred to me that it seems presumptuous of me to criticize a theory put forward by very smart people when I only have 1 karma...).
Oddly enough, I am not interested in improving epistemic rationality right now, partially because I am already quite good at it. But more than that, I am trying to switch it off when talking to other people, for the simple reason (and I’m sure this has already been pointed out before) that if you compare three people, one who estimates the probability of an event at 110%, one who estimates it at 90%, and one who compensates for overconfidence bias and estimates it at 65%, the first two will win friends and influence people, while the third will seem indecisive (unless they are talking to other rationalists). I think I am borderline asperger’s (again, like many people here) and optimizing social skills probably takes precedence over most other things.
I am currently doing a PhD in “absurdly simplistic computational modeling of the blatantly obvious” which better damn well have some signaling value. In my spare time, to stop my brain turning to mush, among other things I am writing a story which is sort of rationalist, in that some of the characters keep using science effectively even when the world is going crazy and the laws of physics seem to change dependent upon whether you believe in them. On the other hand, some of the characters are (a) heroes/heroines (b) awesomely successful (c) hippies on acid who do not believe in objective reality (not that I am implying that all hippies/people who use lsd are irrational). Maybe the point of the story is that you need more than just rationality? Or that some people are powerful because of rationality, while others have imagination, and that friendship combines their powers in a my little pony like fashion? Or maybe its all just an excuse for pretentious philosophy and psychic battles?
But remember that it’s not just your own rationality that benefits you.
Are you saying that improving epistemic rationality is important because it benefits others as well as myself? This is true, but there are many other forms of self-improvement that would also have knock-on effects that benefit others.
I have actually read most of the relevant sequences, epistemic rationality really isn’t low-hanging fruit anymore for me, although I wish I had known about cognitive biases years ago.
Are you saying that improving epistemic rationality is important because it benefits others as well as myself?
No, I’m saying that improving the epistemic rationality of others benefits everyone, including yourself. It’s not just about improving our own rationality as individuals, it’s about trying to improve the rationality of people-in-general - ‘raising the sanity waterline’.
Ok, I see what you mean now. Yes, this is often true, but again, I am trying to be less preachy (at least IRL) about rationality—if someone believes in astrology, or faith healing, or reincarnation then:
(a) their beliefs probably bring them comfort
(b) Trying to persuade them is often like banging my head against a brick wall
(c) even the notion that there can be such a thing as a correct fact, independent of subjective mental states is very threatening to some people and I don’t want to start pointless arguments
So unless they are acting irrationally in a way which harms other people, or they seem capable of having a sensible discussion, or I am drunk, I tend to leave them be.
Hello everyone, Like many people, I come to this site via an interest in transhumanism, although it seems unlikely to me that FAI implementing CEV can actually be designed before the singularity
Many here would agree with you. (And, for instance, consider a ~10% chance of success better than near certain extinction.)
I agree that 10% chance of success is better than near zero, and furthermore I agree that expected utility maximization means that putting in a great deal of effort to achieve a positive outcome is wiser than saying “oh well, we’re doomed anyway, might as well party hard and make the most of the time we have left”. However, the question is whether, if FAI has a low probability of success, are other possibilities, e.g. tool AI a better option to pursue?
The last survey reported that Less Wrongers on average believe that humanity has about a 68% chance of surviving the century without a disaster killing >90% of the species. (Median 80%, though, which might be a better measure of the community feeling than the mean in this case.) That’s a lower bar than actual extinction, but also a shorter timescale, so I expect the answer to your question would be in the same ballpark.
I can explain why, and possibly even what could be done instead, but it suddenly occurred to me that it seems presumptuous of me to criticize a theory put forward by very smart people when I only have 1 karma...
IMO you should definitely do it. Even if LW karma is good an indicator of good ideas, more information rarely hurts, especially on a topic as important as this.
Ok—although maybe I should stick it in its own thread?
I realize much of this has been said before.
Part 1 : AGI will come before FAI, because:
Complexity of algorithm design:
Intuitively, FAI seems orders of magnitude more complex than AGI. If I decided to start trying to program an AGI tomorrow, I would have ideas on how to start, and maybe even make a minuscule amount of progress. Ben Goertzel even has a (somewhat optimistic) roadmap for AGI in a decade. Meanwhile, afaik FAI is still stuck at the stage of lob’s theorem. The fact that EY seems to be focusing on promoting rationality and writing (admittedly awesome) harry potter fanfiction seems to indicate that he doesn’t currently know how to write FAI (and nor does anyone else) otherwise he would be focusing on that now, and instead is planning for the long term.
Computational complexity
CEV requires modelling (and extrapolating) every human mind on the planet, while avoiding the creation of sentient entities. While modelling might be cheaper than ~10^17 flops per human due to short cuts, I doubt it’s going to come cheap. Randomly sampling a subset of humanity to extrapolate from, at least initially, could make this problem less severe. Furthermore, this can be partially circumvented by saying that the AI follows a specific utility function while bootstrapping to enough computing power to implement CEV, but then you have the problem of allowing it to bootstrap safely. Having to prove friendliness of each step in self-improvement strikes me as something that could also be costly.
Finally, I get the impression that people are considering using Solomonoff induction. It’s uncomputable, and while I realize that there exist approximations, I would imagine that these would be extremely expensive to calculate anything non-trivial. Is there any reason for using SI for FAI more than AGI, e.g. something todo with provability about the programs actions?
Infeasibility of relinquishment.
If you can’t convince Ben Goertzel that FAI is needed, even though he is familiar with the arguments and is an advisor to SIAI, you’re not going to get anywhere near a universal consensus on the matter. Furthermore, AI is increasingly being used in financial and possibly soon military applications, and so there are strong incentives to speed the development of AI. While these uses are unlikely to be full AGI, they could provide building blocks – I can imagine a plausible situation where an advanced AI that predict the stock exchange could easily be modified to be a universal predictor. The most powerful incentive to speed up AI development is the sheer number of people who die every day, and the amount of negentropy lost in the case that the 2nd law of thermodynamics cannot be circumvented. Even if there could be a worldwide ban on non-provably safe AGI, work would still probably continue in secret by people who thought the benefits of an earlier singularity outweighed the risks, and/or were worried about ideologically opposed groups getting their first.
Financial bootstrapping
If you are ok with running a non-provably friendly AGI, then even in the early stages when, for example, your AI can write simple code or make reasonably accurate predictions but not speak English or make plans, you can use these to earn money, and buy more hardware/programmers. This seems to be part of the approach Ben is taking.
Coming in Part II: is there any alternative (and doing nothing is not an alternative! even if FAI is unlikely to work its better than giving up!)
But more than that, I am trying to switch it off when talking to other people, for the simple reason (and I’m sure this has already been pointed out before) that if you compare three people, one who estimates the probability of an event at 110%, one who estimates it at 90%, and one who compensates for overconfidence bias and estimates it at 65%, the first two will win friends and influence people, while the third will seem indecisive.
Made me think of this article. Yes, you may be able, in the short run, to win friends and influence people by tricking yourself into being overconfident. But that belief is only in your head and doesn’t affect the universe–thus doesn’t affect the probability of Event X happening. Which means that if, realistically, X is 65% likely to happen, then you with your overconfidence, claiming that X is bound to happen, will eventually look like a fool 35% of the time, and will make it hard for yourself to leave a line of retreat.
Conclusion: in the long run, it’s very good to be honest with yourself about your predictions of the future, and probably preferable to be honest with others, too, if you want to recruit their support.
Excellent points, and of course it is situation dependent—if one makes erroneous predictions on archived forms of communication, e.g. these posts, then yes these predictions can come back to haunt you, but often, especially in non-archived communications, people will remember the correct predictions and forget the false ones. It should go without saying that I do not intend to be overconfident on LW—if I was going to be, then the last thing I would do is announce this intention!
In a strange way, I seem to want to hold three different beliefs:
1) An accurate assessment of what will happen, for planning my own actions.
2) A confidant, stopping just short of arrogant, belief in my predictions for impressing non-rationalists.
3) An unshakeable belief in my own invincibility, so that psychosomatic effects keep me healthy.
Unfortunately, this kinda sounds like “I want to have multiple personality disorder”.
So, call -C1 the social cost of reporting a .9 confidence of something that turns out false, and -C2 the social cost of reporting a .65 confidence of something that turns out false. Call C3 the benefit of reporting .9 confidence of something true, and C4 the benefit of .65 confidence.
How confident are you that that (.65C3 -.35C1) < (.65C4-.35C2)?
In certain situations, such as sporting events which do not involve betting, my confidence that (.65C3 -.35C1) < (.65C4-.35C2) is at most 10%. In these situations confidence is valued far more that epistemic rationality.
I would say I’m about 75% confident that (.65C3 -.35C1) < (.65C4-.35C2)… But one of the reasons I don’t even want to play that game is that I feel I am completely unqualified to estimate probabilities about that, and most other things. I would have no idea how to go about estimating the probability of, for example, the Singularity occurring before 2050...much less how to compensate for biases in my estimate.
I think I also have somewhat of an ick reaction towards the concept of “tricking” people to get what you want, even if in a very subtle form. I just...like...being honest, and it’s hard for me to tell if my arguments about honesty being better are rationalizations because I don’t want being dishonest to be justifiable.
The way to bridge that gap is to only volunteer predictions when you’re quite confident, and otherwise stay quiet, change the subject, or murmur a polite assent. You’re absolutely right that explicitly declaring a 65% confidence estimate will make you look indecisive—but people aren’t likely to notice that you make predictions less often than other people—they’ll be too focused on how when you do make predictions, you have an uncanny tendency to be correct...and also that you’re pleasantly modest and demure, too.
Hello everyone, Like many people, I come to this site via an interest in transhumanism, although it seems unlikely to me that FAI implementing CEV can actually be designed before the singularity (I can explain why, and possibly even what could be done instead, but it suddenly occurred to me that it seems presumptuous of me to criticize a theory put forward by very smart people when I only have 1 karma...).
Oddly enough, I am not interested in improving epistemic rationality right now, partially because I am already quite good at it. But more than that, I am trying to switch it off when talking to other people, for the simple reason (and I’m sure this has already been pointed out before) that if you compare three people, one who estimates the probability of an event at 110%, one who estimates it at 90%, and one who compensates for overconfidence bias and estimates it at 65%, the first two will win friends and influence people, while the third will seem indecisive (unless they are talking to other rationalists). I think I am borderline asperger’s (again, like many people here) and optimizing social skills probably takes precedence over most other things.
I am currently doing a PhD in “absurdly simplistic computational modeling of the blatantly obvious” which better damn well have some signaling value. In my spare time, to stop my brain turning to mush, among other things I am writing a story which is sort of rationalist, in that some of the characters keep using science effectively even when the world is going crazy and the laws of physics seem to change dependent upon whether you believe in them. On the other hand, some of the characters are (a) heroes/heroines (b) awesomely successful (c) hippies on acid who do not believe in objective reality (not that I am implying that all hippies/people who use lsd are irrational). Maybe the point of the story is that you need more than just rationality? Or that some people are powerful because of rationality, while others have imagination, and that friendship combines their powers in a my little pony like fashion? Or maybe its all just an excuse for pretentious philosophy and psychic battles?
But remember that it’s not just your own rationality that benefits you.
Presume away. Karma doesn’t win arguments, arguments win karma.
Are you saying that improving epistemic rationality is important because it benefits others as well as myself? This is true, but there are many other forms of self-improvement that would also have knock-on effects that benefit others.
I have actually read most of the relevant sequences, epistemic rationality really isn’t low-hanging fruit anymore for me, although I wish I had known about cognitive biases years ago.
No, I’m saying that improving the epistemic rationality of others benefits everyone, including yourself. It’s not just about improving our own rationality as individuals, it’s about trying to improve the rationality of people-in-general - ‘raising the sanity waterline’.
Ok, I see what you mean now. Yes, this is often true, but again, I am trying to be less preachy (at least IRL) about rationality—if someone believes in astrology, or faith healing, or reincarnation then: (a) their beliefs probably bring them comfort (b) Trying to persuade them is often like banging my head against a brick wall (c) even the notion that there can be such a thing as a correct fact, independent of subjective mental states is very threatening to some people and I don’t want to start pointless arguments
So unless they are acting irrationally in a way which harms other people, or they seem capable of having a sensible discussion, or I am drunk, I tend to leave them be.
Many here would agree with you. (And, for instance, consider a ~10% chance of success better than near certain extinction.)
I agree that 10% chance of success is better than near zero, and furthermore I agree that expected utility maximization means that putting in a great deal of effort to achieve a positive outcome is wiser than saying “oh well, we’re doomed anyway, might as well party hard and make the most of the time we have left”. However, the question is whether, if FAI has a low probability of success, are other possibilities, e.g. tool AI a better option to pursue?
Would you say that many people here (and yourself?) believe that the probable end of our species is within the next century or two?
The last survey reported that Less Wrongers on average believe that humanity has about a 68% chance of surviving the century without a disaster killing >90% of the species. (Median 80%, though, which might be a better measure of the community feeling than the mean in this case.) That’s a lower bar than actual extinction, but also a shorter timescale, so I expect the answer to your question would be in the same ballpark.
For myself: Yes! p(extinct within 200 years) > 0.5
Welcome!
IMO you should definitely do it. Even if LW karma is good an indicator of good ideas, more information rarely hurts, especially on a topic as important as this.
Ok—although maybe I should stick it in its own thread?
I realize much of this has been said before.
Part 1 : AGI will come before FAI, because:
Complexity of algorithm design:
Intuitively, FAI seems orders of magnitude more complex than AGI. If I decided to start trying to program an AGI tomorrow, I would have ideas on how to start, and maybe even make a minuscule amount of progress. Ben Goertzel even has a (somewhat optimistic) roadmap for AGI in a decade. Meanwhile, afaik FAI is still stuck at the stage of lob’s theorem.
The fact that EY seems to be focusing on promoting rationality and writing (admittedly awesome) harry potter fanfiction seems to indicate that he doesn’t currently know how to write FAI (and nor does anyone else) otherwise he would be focusing on that now, and instead is planning for the long term.
Computational complexity CEV requires modelling (and extrapolating) every human mind on the planet, while avoiding the creation of sentient entities. While modelling might be cheaper than ~10^17 flops per human due to short cuts, I doubt it’s going to come cheap. Randomly sampling a subset of humanity to extrapolate from, at least initially, could make this problem less severe. Furthermore, this can be partially circumvented by saying that the AI follows a specific utility function while bootstrapping to enough computing power to implement CEV, but then you have the problem of allowing it to bootstrap safely. Having to prove friendliness of each step in self-improvement strikes me as something that could also be costly. Finally, I get the impression that people are considering using Solomonoff induction. It’s uncomputable, and while I realize that there exist approximations, I would imagine that these would be extremely expensive to calculate anything non-trivial. Is there any reason for using SI for FAI more than AGI, e.g. something todo with provability about the programs actions?
Infeasibility of relinquishment. If you can’t convince Ben Goertzel that FAI is needed, even though he is familiar with the arguments and is an advisor to SIAI, you’re not going to get anywhere near a universal consensus on the matter. Furthermore, AI is increasingly being used in financial and possibly soon military applications, and so there are strong incentives to speed the development of AI. While these uses are unlikely to be full AGI, they could provide building blocks – I can imagine a plausible situation where an advanced AI that predict the stock exchange could easily be modified to be a universal predictor.
The most powerful incentive to speed up AI development is the sheer number of people who die every day, and the amount of negentropy lost in the case that the 2nd law of thermodynamics cannot be circumvented. Even if there could be a worldwide ban on non-provably safe AGI, work would still probably continue in secret by people who thought the benefits of an earlier singularity outweighed the risks, and/or were worried about ideologically opposed groups getting their first.
Financial bootstrapping If you are ok with running a non-provably friendly AGI, then even in the early stages when, for example, your AI can write simple code or make reasonably accurate predictions but not speak English or make plans, you can use these to earn money, and buy more hardware/programmers. This seems to be part of the approach Ben is taking.
Coming in Part II: is there any alternative (and doing nothing is not an alternative! even if FAI is unlikely to work its better than giving up!)
Definitely worth its own Discussion post, once you have min karma, which should not take long.
They already have it.
Welcome!
Made me think of this article. Yes, you may be able, in the short run, to win friends and influence people by tricking yourself into being overconfident. But that belief is only in your head and doesn’t affect the universe–thus doesn’t affect the probability of Event X happening. Which means that if, realistically, X is 65% likely to happen, then you with your overconfidence, claiming that X is bound to happen, will eventually look like a fool 35% of the time, and will make it hard for yourself to leave a line of retreat.
Conclusion: in the long run, it’s very good to be honest with yourself about your predictions of the future, and probably preferable to be honest with others, too, if you want to recruit their support.
Excellent points, and of course it is situation dependent—if one makes erroneous predictions on archived forms of communication, e.g. these posts, then yes these predictions can come back to haunt you, but often, especially in non-archived communications, people will remember the correct predictions and forget the false ones. It should go without saying that I do not intend to be overconfident on LW—if I was going to be, then the last thing I would do is announce this intention! In a strange way, I seem to want to hold three different beliefs: 1) An accurate assessment of what will happen, for planning my own actions. 2) A confidant, stopping just short of arrogant, belief in my predictions for impressing non-rationalists. 3) An unshakeable belief in my own invincibility, so that psychosomatic effects keep me healthy.
Unfortunately, this kinda sounds like “I want to have multiple personality disorder”.
If you’re going to go that route, at least research it first. For example:
http://healthymultiplicity.com/
Thanks for the advice, but I don’t actually want to have multiple personality disorder—I was just drawing an analogy.
Hm.
So, call -C1 the social cost of reporting a .9 confidence of something that turns out false, and -C2 the social cost of reporting a .65 confidence of something that turns out false. Call C3 the benefit of reporting .9 confidence of something true, and C4 the benefit of .65 confidence.
How confident are you that that (.65C3 -.35C1) < (.65C4-.35C2)?
In certain situations, such as sporting events which do not involve betting, my confidence that (.65C3 -.35C1) < (.65C4-.35C2) is at most 10%. In these situations confidence is valued far more that epistemic rationality.
I would say I’m about 75% confident that (.65C3 -.35C1) < (.65C4-.35C2)… But one of the reasons I don’t even want to play that game is that I feel I am completely unqualified to estimate probabilities about that, and most other things. I would have no idea how to go about estimating the probability of, for example, the Singularity occurring before 2050...much less how to compensate for biases in my estimate.
I think I also have somewhat of an ick reaction towards the concept of “tricking” people to get what you want, even if in a very subtle form. I just...like...being honest, and it’s hard for me to tell if my arguments about honesty being better are rationalizations because I don’t want being dishonest to be justifiable.
The way to bridge that gap is to only volunteer predictions when you’re quite confident, and otherwise stay quiet, change the subject, or murmur a polite assent. You’re absolutely right that explicitly declaring a 65% confidence estimate will make you look indecisive—but people aren’t likely to notice that you make predictions less often than other people—they’ll be too focused on how when you do make predictions, you have an uncanny tendency to be correct...and also that you’re pleasantly modest and demure, too.
(nods) That makes sense.