What evidence justified a prior strong enough as to be updated on a single paragraph
I can’t speak for lukeprog, but I believe that “update” is the wrong word to use here. If we acted like Bayesian updaters then compartmentalization wouldn’t be an issue in the first place. I.J. Good’s paragraph, rather than providing evidence, seems to have been more like a big sign saying “Look here! This is a place where you’re not being very Bayesian!”. Such a trigger doesn’t need to be written in any kind of formal language—it could have been an offhand comment someone made on a completely different subject. It’s simply that (to an honest mind), once attention is drawn to an inconsistency in your own logic, you can’t turn back.
That said, lukeprog hasn’t actually explained why his existing beliefs strongly implied an intelligence explosion. That wasn’t the point of this post, but like you it’s a post that I’d very much like to see. I’m interested in trying to build a Bayesian case for or against the intelligence explosion (and other singularity-ish outcomes).
You’re right that there’s a problem obtaining evidence for or against beliefs about the future. I can think of three approaches:
Expertology—seeing what kinds of predictions have been made by experts in the past, and what factors are correlated with them being right.
Models—build a number of parameterizable models of the world which are (necessarily) simplified but which are at least capable of modeling the outcome you’re interested in. Give a prior for each and then do a Bayesian update according to how well that model predicts the past.
There might be an intuitive heuristic along the lines of “if you can’t rule it out then it might happen”, but I don’t know how to formalize that or make it quantitative.
So I’m interested in whether these can be done without introducing horrible biases, whether anyone’s tried them before or whether there are any other approaches I’ve missed out.
Thanks, I have yet to learn the relevant math. I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong, as you indicated.
There are some things I am particularly confused about when it comes to probability. I am looking forward to the time when I start learning probability theory (currently doing Calculus).
Just a two examples:
I don’t understand why you can’t just ignore some possible outcomes. We are computationally limited agents after all. Thus if someone comes a long and tells me about a certain possibility that involves a huge amount of utility but doesn’t provide enough (how much would be enough anyway?) evidence, my intuition is to just ignore that possibility rather than to assign 50% probability to it. After all, if humanity was forced to throw a fair coin and heads would imply doom then I’d be in favor of doing everything to ensure that the coin doesn’t land heads (e.g. a research project that figures out how to throw it in a certain way that maximizes the probability of it landing tails). But that would be crazy because anyone could come along and make unjustified predictions that involve huge utility stakes and I would have to ignore all lower utility possibilities even if there is a lot of empirical evidence in favor of them.
Something else I am confused about is how probability is grounded. What rules are used to decide how much certain evidence can influence my probability estimates? I mean, if using probability theory only shifts my use of intuition towards the intuitive assignment of numerical probability estimates of evidence, in favor of a possibility, then it might as well distort my estimates more than trusting my intuition about the possibility alone. Because 1.) if using my intuition to decide how much each piece of evidence changes the overall probability, I can be wrong on each occasion rather than just one (accumulation of error) 2.) humans are really bad with numbers, and forcing them to put a number on something vague might in some cases cause them to over or underestimate their confidence by many orders of magnitude.
I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong
I think most people use “update” colloquially, i.e. something along the lines of “what you just said appears to constitute evidence that I wasn’t previously aware of, and I should change my beliefs accordingly”. I don’t know how often rationalists actually plug these things into a formula.
I don’t understand why you can’t just ignore some possible outcomes.
This is the problem of Pascal’s Mugging—I think it’s something that people are still confused about. In general, if someone tells you about a really weird possibility you should assign it a probability of a lot less than 50%, as it would essentially be a conjunction of a lot of unlikely events. The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
I’m still waiting for an answer to that one, but in the meantime it seems worth attacking the problems that lie in the grey area between “easily tractable expected utility calculations” and “classic Pascal’s mugging”. For me, AI risk mitigation is still in that grey area.
The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
Don’t worry about it; if you decline a Pascal’s mugging I’ll cause positive utility equal to twice the amount of negative utility you were threatened with, and if you accept one I’ll cause negative utility equal to twice what you were threatened with.
Why did I never think of this? I mean I have thought of very similar things in a thought experiment sense and I’ve even used it to explain to people that paying the mugging cannot be something correct but unpalatable, but it never occurred to me to use it on someone.
That’s not what I meant, I have been too vague. It is clear to me how to update on evidence given concrete data or goats behind doors in game shows. What I meant is how one could possible update on evidence like the victory of IBM Watson at Jeopardy regarding risks of AI. It seems to me that assigning numerical probability estimates to such evidence, that is then used to update on the overall probability of risks from AI, is a very shaky affair that might distort the end result as one just shifted the use of intuition towards the interpretation of evidence in favor of an outcome rather than the outcomes itself.
Causal analysis is probably closer to what you’re looking for. It displays stability under (small) perturbation of relative probabilities, and it’s probably closer to what humans do under the hood than Bayes’ theorem. Pearl often observes that humans work with cause and effect with more facility than numerical probabilities.
Numerical stability is definitely something we need in our epistemology. If small errors make the whole thing blow up, it’s not any good to us, because we know we make small errors all the time.
I can’t speak for lukeprog, but I believe that “update” is the wrong word to use here. If we acted like Bayesian updaters then compartmentalization wouldn’t be an issue in the first place. I.J. Good’s paragraph, rather than providing evidence, seems to have been more like a big sign saying “Look here! This is a place where you’re not being very Bayesian!”. Such a trigger doesn’t need to be written in any kind of formal language—it could have been an offhand comment someone made on a completely different subject. It’s simply that (to an honest mind), once attention is drawn to an inconsistency in your own logic, you can’t turn back.
That said, lukeprog hasn’t actually explained why his existing beliefs strongly implied an intelligence explosion. That wasn’t the point of this post, but like you it’s a post that I’d very much like to see. I’m interested in trying to build a Bayesian case for or against the intelligence explosion (and other singularity-ish outcomes).
You’re right that there’s a problem obtaining evidence for or against beliefs about the future. I can think of three approaches:
Expertology—seeing what kinds of predictions have been made by experts in the past, and what factors are correlated with them being right.
Models—build a number of parameterizable models of the world which are (necessarily) simplified but which are at least capable of modeling the outcome you’re interested in. Give a prior for each and then do a Bayesian update according to how well that model predicts the past.
There might be an intuitive heuristic along the lines of “if you can’t rule it out then it might happen”, but I don’t know how to formalize that or make it quantitative.
So I’m interested in whether these can be done without introducing horrible biases, whether anyone’s tried them before or whether there are any other approaches I’ve missed out.
Thanks, I have yet to learn the relevant math. I only have a very vague idea about Bayesian probability at this point so my use of “update” might very well be wrong, as you indicated.
There are some things I am particularly confused about when it comes to probability. I am looking forward to the time when I start learning probability theory (currently doing Calculus).
Just a two examples:
I don’t understand why you can’t just ignore some possible outcomes. We are computationally limited agents after all. Thus if someone comes a long and tells me about a certain possibility that involves a huge amount of utility but doesn’t provide enough (how much would be enough anyway?) evidence, my intuition is to just ignore that possibility rather than to assign 50% probability to it. After all, if humanity was forced to throw a fair coin and heads would imply doom then I’d be in favor of doing everything to ensure that the coin doesn’t land heads (e.g. a research project that figures out how to throw it in a certain way that maximizes the probability of it landing tails). But that would be crazy because anyone could come along and make unjustified predictions that involve huge utility stakes and I would have to ignore all lower utility possibilities even if there is a lot of empirical evidence in favor of them.
Something else I am confused about is how probability is grounded. What rules are used to decide how much certain evidence can influence my probability estimates? I mean, if using probability theory only shifts my use of intuition towards the intuitive assignment of numerical probability estimates of evidence, in favor of a possibility, then it might as well distort my estimates more than trusting my intuition about the possibility alone. Because 1.) if using my intuition to decide how much each piece of evidence changes the overall probability, I can be wrong on each occasion rather than just one (accumulation of error) 2.) humans are really bad with numbers, and forcing them to put a number on something vague might in some cases cause them to over or underestimate their confidence by many orders of magnitude.
I think most people use “update” colloquially, i.e. something along the lines of “what you just said appears to constitute evidence that I wasn’t previously aware of, and I should change my beliefs accordingly”. I don’t know how often rationalists actually plug these things into a formula.
This is the problem of Pascal’s Mugging—I think it’s something that people are still confused about. In general, if someone tells you about a really weird possibility you should assign it a probability of a lot less than 50%, as it would essentially be a conjunction of a lot of unlikely events. The problem is that the utility might be so high (or low) that when you multiply it by this tiny probability you still get something huge.
I’m still waiting for an answer to that one, but in the meantime it seems worth attacking the problems that lie in the grey area between “easily tractable expected utility calculations” and “classic Pascal’s mugging”. For me, AI risk mitigation is still in that grey area.
Don’t worry about it; if you decline a Pascal’s mugging I’ll cause positive utility equal to twice the amount of negative utility you were threatened with, and if you accept one I’ll cause negative utility equal to twice what you were threatened with.
Trust me.
Excuse me, I’m going to go and use this on all my friends.
Why did I never think of this? I mean I have thought of very similar things in a thought experiment sense and I’ve even used it to explain to people that paying the mugging cannot be something correct but unpalatable, but it never occurred to me to use it on someone.
Bayes’ Theorem is precisely that rule.
That’s not what I meant, I have been too vague. It is clear to me how to update on evidence given concrete data or goats behind doors in game shows. What I meant is how one could possible update on evidence like the victory of IBM Watson at Jeopardy regarding risks of AI. It seems to me that assigning numerical probability estimates to such evidence, that is then used to update on the overall probability of risks from AI, is a very shaky affair that might distort the end result as one just shifted the use of intuition towards the interpretation of evidence in favor of an outcome rather than the outcomes itself.
Causal analysis is probably closer to what you’re looking for. It displays stability under (small) perturbation of relative probabilities, and it’s probably closer to what humans do under the hood than Bayes’ theorem. Pearl often observes that humans work with cause and effect with more facility than numerical probabilities.
Numerical stability is definitely something we need in our epistemology. If small errors make the whole thing blow up, it’s not any good to us, because we know we make small errors all the time.