I hadn’t noticed that my worldview already implied intelligence explosion.
I’d like to see a post on that worldview. The possibility of an intelligence explosion seems to be an extraordinary belief. What evidence justified a prior strong enough as to be updated on a single paragraph, written in natural language, to the extent that you would afterwards devote your whole life to that possibility?
I’m not talking about the problem of free-floating beliefs that don’t control your anticipations. No, I’m talking about “proper” beliefs that require observation, can be updated by evidence, and pay rent in anticipated experiences.
How do you anticipate your beliefs to pay rent? What kind of evidence could possible convince you that an intelligence explosion is unlikely, how could your beliefs be surprised by data?
What evidence justified a prior strong enough as to be updated on a single paragraph
I can’t speak for lukeprog, but I believe that “update” is the wrong word to use here. If we acted like Bayesian updaters then compartmentalization wouldn’t be an issue in the first place. I.J. Good’s paragraph, rather than providing evidence, seems to have been more like a big sign saying “Look here! This is a place where you’re not being very Bayesian!”. Such a trigger doesn’t need to be written in any kind of formal language—it could have been an offhand comment someone made on a completely different subject. It’s simply that (to an honest mind), once attention is drawn to an inconsistency in your own logic, you can’t turn back.
That said, lukeprog hasn’t actually explained why his existing beliefs strongly implied an intelligence explosion. That wasn’t the point of this post, but like you it’s a post that I’d very much like to see. I’m interested in trying to build a Bayesian case for or against the intelligence explosion (and other singularity-ish outcomes).
You’re right that there’s a problem obtaining evidence for or against beliefs about the future. I can think of three approaches:
Expertology—seeing what kinds of predictions have been made by experts in the past, and what factors are correlated with them being right.
Models—build a number of parameterizable models of the world which are (necessarily) simplified but which are at least capable of modeling the outcome you’re interested in. Give a prior for each and then do a Bayesian update according to how well that model predicts the past.
There might be an intuitive heuristic along the lines of “if you can’t rule it out then it might happen”, but I don’t know how to formalize that or make it quantitative.
So I’m interested in whether these can be done without introducing horrible biases, whether anyone’s tried them before or whether there are any other approaches I’ve missed out.
Thanks, I have yet to learn the relevant math. I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong, as you indicated.
There are some things I am particularly confused about when it comes to probability. I am looking forward to the time when I start learning probability theory (currently doing Calculus).
Just a two examples:
I don’t understand why you can’t just ignore some possible outcomes. We are computationally limited agents after all. Thus if someone comes a long and tells me about a certain possibility that involves a huge amount of utility but doesn’t provide enough (how much would be enough anyway?) evidence, my intuition is to just ignore that possibility rather than to assign 50% probability to it. After all, if humanity was forced to throw a fair coin and heads would imply doom then I’d be in favor of doing everything to ensure that the coin doesn’t land heads (e.g. a research project that figures out how to throw it in a certain way that maximizes the probability of it landing tails). But that would be crazy because anyone could come along and make unjustified predictions that involve huge utility stakes and I would have to ignore all lower utility possibilities even if there is a lot of empirical evidence in favor of them.
Something else I am confused about is how probability is grounded. What rules are used to decide how much certain evidence can influence my probability estimates? I mean, if using probability theory only shifts my use of intuition towards the intuitive assignment of numerical probability estimates of evidence, in favor of a possibility, then it might as well distort my estimates more than trusting my intuition about the possibility alone. Because 1.) if using my intuition to decide how much each piece of evidence changes the overall probability, I can be wrong on each occasion rather than just one (accumulation of error) 2.) humans are really bad with numbers, and forcing them to put a number on something vague might in some cases cause them to over or underestimate their confidence by many orders of magnitude.
I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong
I think most people use “update” colloquially, i.e. something along the lines of “what you just said appears to constitute evidence that I wasn’t previously aware of, and I should change my beliefs accordingly”. I don’t know how often rationalists actually plug these things into a formula.
I don’t understand why you can’t just ignore some possible outcomes.
This is the problem of Pascal’s Mugging—I think it’s something that people are still confused about. In general, if someone tells you about a really weird possibility you should assign it a probability of a lot less than 50%, as it would essentially be a conjunction of a lot of unlikely events. The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
I’m still waiting for an answer to that one, but in the meantime it seems worth attacking the problems that lie in the grey area between “easily tractable expected utility calculations” and “classic Pascal’s mugging”. For me, AI risk mitigation is still in that grey area.
The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
Don’t worry about it; if you decline a Pascal’s mugging I’ll cause positive utility equal to twice the amount of negative utility you were threatened with, and if you accept one I’ll cause negative utility equal to twice what you were threatened with.
Why did I never think of this? I mean I have thought of very similar things in a thought experiment sense and I’ve even used it to explain to people that paying the mugging cannot be something correct but unpalatable, but it never occurred to me to use it on someone.
That’s not what I meant, I have been too vague. It is clear to me how to update on evidence given concrete data or goats behind doors in game shows. What I meant is how one could possible update on evidence like the victory of IBM Watson at Jeopardy regarding risks of AI. It seems to me that assigning numerical probability estimates to such evidence, that is then used to update on the overall probability of risks from AI, is a very shaky affair that might distort the end result as one just shifted the use of intuition towards the interpretation of evidence in favor of an outcome rather than the outcomes itself.
Causal analysis is probably closer to what you’re looking for. It displays stability under (small) perturbation of relative probabilities, and it’s probably closer to what humans do under the hood than Bayes’ theorem. Pearl often observes that humans work with cause and effect with more facility than numerical probabilities.
Numerical stability is definitely something we need in our epistemology. If small errors make the whole thing blow up, it’s not any good to us, because we know we make small errors all the time.
What kind of evidence could possible convince you that an intelligence explosion is unlikely, how could your beliefs be surprised by data?
There is no reason to believe intelligence stops being useful for problem solving as one gets more of it, but I can easily imagine evidence that would suggest that.
A non-AI intelligence above human level, such as a human with computer processors integrated into his or her brain, a biologically enhanced human, etc. might prove to be no more usefully intelligent than Newton, Von Neumann, etc. despite being an order of magnitude smarter by practical measures.
to the extent that you would afterwards devote your whole life to that possibility?
Leveraging makes little sense according to many reasonable utility functions. If one is guessing the color of random cards, and 70% of the cards are red, and 30% are blue, and red and blue pay out equally, one should always guess red each turn.
Where utility is related to the ln of money, it makes sense to diversify, but that is different from a life-impact sort of case where one seeks to maximize the utility of others.
The outside view is to avoid total commitment lest one be sucked into a happy death spiral and suffer from the sunk costs fallacy, but if those and similar fallacies can be avoided, total commitment makes sense.
The possibility of an intelligence explosion seems to be an extraordinary belief.
Extraordinary compared to what? We already now that most people are insane, so that belief beeing not shared by almost everybody doesn’t make it unlikely a priori.
In some ways the intellgence explosion is a straightforward extrapolation of what we know at the moment, so I don’t think your critisism is valid here.
What evidence justified a prior strong enough as to be updated on a single paragraph, written in natural language, to the extent that you would afterwards devote your whole life to that possibility?
I think one could tell a reasonably competent physicist 50 years prior to Schrödinger how to derive quantum mechanics in one paragraph of natural language. Human language can contain lots of information, especially if speaker and listener already share a lot of concepts.
I’m not sure why you’ve written your comment, are you just using the opportunity to bring up this old topic again? I find myself irritated by this, even though I probably agree with you :)
For me there are a lot of things that sound completely sane but might be just bunk. Antimatter weapons, grey goo, the creation of planet eating black holes by particle accelerators, aliens or string theory. I don’t have the necessary education to discern those ideas from an intelligence explosion. They all sound like possibilities to me that might or might not be true. All I can do is to recognize their extraordinary status and subsequently demand the peer-review and assessment of those ideas by a larger and educated audience. Otherwise I run the risk of being swayed by the huge utility associated with those ideas, I run the risk of falling prey to a Pascalian mugger.
I think one could tell a reasonably competent physicist 50 years prior to Schrödinger how to derive quantum mechanics in one paragraph of natural language.
Is Luke Muehlhauser that competent when it comes to all the fields associated with artificial intelligence research?
I’m not sure why you’ve written your comment, are you just using the opportunity to bring up this old topic again?
It’s not old, it becomes more relevant each day. Since I first voiced skepticism about the topic they expanded to the point of having world-wide meetings. At least a few people playing devil’s advocate is a healthy exercise in my opinion :-)
I’d like to see a post on that worldview. The possibility of an intelligence explosion seems to be an extraordinary belief. What evidence justified a prior strong enough as to be updated on a single paragraph, written in natural language, to the extent that you would afterwards devote your whole life to that possibility?
How do you anticipate your beliefs to pay rent? What kind of evidence could possible convince you that an intelligence explosion is unlikely, how could your beliefs be surprised by data?
I can’t speak for lukeprog, but I believe that “update” is the wrong word to use here. If we acted like Bayesian updaters then compartmentalization wouldn’t be an issue in the first place. I.J. Good’s paragraph, rather than providing evidence, seems to have been more like a big sign saying “Look here! This is a place where you’re not being very Bayesian!”. Such a trigger doesn’t need to be written in any kind of formal language—it could have been an offhand comment someone made on a completely different subject. It’s simply that (to an honest mind), once attention is drawn to an inconsistency in your own logic, you can’t turn back.
That said, lukeprog hasn’t actually explained why his existing beliefs strongly implied an intelligence explosion. That wasn’t the point of this post, but like you it’s a post that I’d very much like to see. I’m interested in trying to build a Bayesian case for or against the intelligence explosion (and other singularity-ish outcomes).
You’re right that there’s a problem obtaining evidence for or against beliefs about the future. I can think of three approaches:
Expertology—seeing what kinds of predictions have been made by experts in the past, and what factors are correlated with them being right.
Models—build a number of parameterizable models of the world which are (necessarily) simplified but which are at least capable of modeling the outcome you’re interested in. Give a prior for each and then do a Bayesian update according to how well that model predicts the past.
There might be an intuitive heuristic along the lines of “if you can’t rule it out then it might happen”, but I don’t know how to formalize that or make it quantitative.
So I’m interested in whether these can be done without introducing horrible biases, whether anyone’s tried them before or whether there are any other approaches I’ve missed out.
Thanks, I have yet to learn the relevant math. I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong, as you indicated.
There are some things I am particularly confused about when it comes to probability. I am looking forward to the time when I start learning probability theory (currently doing Calculus).
Just a two examples:
I don’t understand why you can’t just ignore some possible outcomes. We are computationally limited agents after all. Thus if someone comes a long and tells me about a certain possibility that involves a huge amount of utility but doesn’t provide enough (how much would be enough anyway?) evidence, my intuition is to just ignore that possibility rather than to assign 50% probability to it. After all, if humanity was forced to throw a fair coin and heads would imply doom then I’d be in favor of doing everything to ensure that the coin doesn’t land heads (e.g. a research project that figures out how to throw it in a certain way that maximizes the probability of it landing tails). But that would be crazy because anyone could come along and make unjustified predictions that involve huge utility stakes and I would have to ignore all lower utility possibilities even if there is a lot of empirical evidence in favor of them.
Something else I am confused about is how probability is grounded. What rules are used to decide how much certain evidence can influence my probability estimates? I mean, if using probability theory only shifts my use of intuition towards the intuitive assignment of numerical probability estimates of evidence, in favor of a possibility, then it might as well distort my estimates more than trusting my intuition about the possibility alone. Because 1.) if using my intuition to decide how much each piece of evidence changes the overall probability, I can be wrong on each occasion rather than just one (accumulation of error) 2.) humans are really bad with numbers, and forcing them to put a number on something vague might in some cases cause them to over or underestimate their confidence by many orders of magnitude.
I think most people use “update” colloquially, i.e. something along the lines of “what you just said appears to constitute evidence that I wasn’t previously aware of, and I should change my beliefs accordingly”. I don’t know how often rationalists actually plug these things into a formula.
This is the problem of Pascal’s Mugging—I think it’s something that people are still confused about. In general, if someone tells you about a really weird possibility you should assign it a probability of a lot less than 50%, as it would essentially be a conjunction of a lot of unlikely events. The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
I’m still waiting for an answer to that one, but in the meantime it seems worth attacking the problems that lie in the grey area between “easily tractable expected utility calculations” and “classic Pascal’s mugging”. For me, AI risk mitigation is still in that grey area.
Don’t worry about it; if you decline a Pascal’s mugging I’ll cause positive utility equal to twice the amount of negative utility you were threatened with, and if you accept one I’ll cause negative utility equal to twice what you were threatened with.
Trust me.
Excuse me, I’m going to go and use this on all my friends.
Why did I never think of this? I mean I have thought of very similar things in a thought experiment sense and I’ve even used it to explain to people that paying the mugging cannot be something correct but unpalatable, but it never occurred to me to use it on someone.
Bayes’ Theorem is precisely that rule.
That’s not what I meant, I have been too vague. It is clear to me how to update on evidence given concrete data or goats behind doors in game shows. What I meant is how one could possible update on evidence like the victory of IBM Watson at Jeopardy regarding risks of AI. It seems to me that assigning numerical probability estimates to such evidence, that is then used to update on the overall probability of risks from AI, is a very shaky affair that might distort the end result as one just shifted the use of intuition towards the interpretation of evidence in favor of an outcome rather than the outcomes itself.
Causal analysis is probably closer to what you’re looking for. It displays stability under (small) perturbation of relative probabilities, and it’s probably closer to what humans do under the hood than Bayes’ theorem. Pearl often observes that humans work with cause and effect with more facility than numerical probabilities.
Numerical stability is definitely something we need in our epistemology. If small errors make the whole thing blow up, it’s not any good to us, because we know we make small errors all the time.
There is no reason to believe intelligence stops being useful for problem solving as one gets more of it, but I can easily imagine evidence that would suggest that.
A non-AI intelligence above human level, such as a human with computer processors integrated into his or her brain, a biologically enhanced human, etc. might prove to be no more usefully intelligent than Newton, Von Neumann, etc. despite being an order of magnitude smarter by practical measures.
Leveraging makes little sense according to many reasonable utility functions. If one is guessing the color of random cards, and 70% of the cards are red, and 30% are blue, and red and blue pay out equally, one should always guess red each turn.
Where utility is related to the ln of money, it makes sense to diversify, but that is different from a life-impact sort of case where one seeks to maximize the utility of others.
The outside view is to avoid total commitment lest one be sucked into a happy death spiral and suffer from the sunk costs fallacy, but if those and similar fallacies can be avoided, total commitment makes sense.
Extraordinary compared to what? We already now that most people are insane, so that belief beeing not shared by almost everybody doesn’t make it unlikely a priori. In some ways the intellgence explosion is a straightforward extrapolation of what we know at the moment, so I don’t think your critisism is valid here.
I think one could tell a reasonably competent physicist 50 years prior to Schrödinger how to derive quantum mechanics in one paragraph of natural language. Human language can contain lots of information, especially if speaker and listener already share a lot of concepts.
I’m not sure why you’ve written your comment, are you just using the opportunity to bring up this old topic again? I find myself irritated by this, even though I probably agree with you :)
For me there are a lot of things that sound completely sane but might be just bunk. Antimatter weapons, grey goo, the creation of planet eating black holes by particle accelerators, aliens or string theory. I don’t have the necessary education to discern those ideas from an intelligence explosion. They all sound like possibilities to me that might or might not be true. All I can do is to recognize their extraordinary status and subsequently demand the peer-review and assessment of those ideas by a larger and educated audience. Otherwise I run the risk of being swayed by the huge utility associated with those ideas, I run the risk of falling prey to a Pascalian mugger.
Is Luke Muehlhauser that competent when it comes to all the fields associated with artificial intelligence research?
It’s not old, it becomes more relevant each day. Since I first voiced skepticism about the topic they expanded to the point of having world-wide meetings. At least a few people playing devil’s advocate is a healthy exercise in my opinion :-)
Are you playing devil’s advocate?
For what it’s worth, I consider these things to have very different levels and kinds of implausibility.