I have a reason to believe it has less than a 50% chance of being possible. Does that count?
I figure after the low-hanging fruit is taken care of, it simply becomes a question of if a unit of additional intelligence is enough to add another additional unit of intelligence. If the feedback constant is less than one, intelligence growth stops. If it is greater, the intelligence grows until it falls below one. It will vary somewhat with intelligence, and it would have to fall below one eventually. We have no way of knowing what the feedback constant is, so we’d have to guess that there’s a 50% chance of it being above 1, and a 50% chance of being below. Furthermore, it’s unlikely to be right next to 1, so if it is possible, it will most likely get pretty darn intelligent.
Also, I figure that if the constant is below 1, or even close, humanity will be incapable of creating an AI from scratch, though if they get uploaded they could improve themselves.
We have no way of knowing what the feedback constant is, so we’d have to guess that there’s a 50% chance of it being above 1, and a 50% chance of being below.
Not being able to determine what the constant is doesn’t mean that there is a 50-50 chance that it is larger than 1. In particular, what in your logic prevents one from also concluding that there is also a 50-50 chance of it being larger than 2?
It can’t be less than zero. From what I understand about priors, the maximum entropy prior would be a logarithmic prior. A more reasonable prior would be a log-normal prior with the mean on 1 and a high standard deviation
By logarithmic do you mean p(x) = exp(-x)? That would only have an entropy of 1, I believe, whereas one can easily obtain unboundedly large amounts of entropy, or even infinite entropy (for instance, p(x) = a exp(-a x) has entropy 1-log(a), so letting a go to zero yields arbitrarily large entropy).
Also, as I’ve noted before, entropy doesn’t make that much sense for continuous distributions.
I think it’s Jeffreys prior or something. Anyway, it seems like a good prior. It doesn’t have any arbitrary constants in it like you’d need with p(x) = exp(-x). If you change the scale, the prior stays the same.
p(x) = 1/x isn’t an integrable function (diverges at both 0 and infinity).
(My real objection is more that it’s pretty unlikely that we really have so little information that we have to quibble about which prior to use. It’s also good to be aware of the mathematical difficulties inherent in trying to be an “objective Bayesian”, but the real problem is that it’s not very helpful for making more accurate empirical predictions.)
Which is why I said a log-normal prior would be more reasonable.
Why a log-normal prior with mu = 0? Why not some other value for the location parameter? Log-normal makes pretty strong assumptions, which aren’t justified if we for all practical purposes we have no information about the feedback constant.
How much information do we have? We know that we haven’t managed to build an AI in 40 years, and that’s about it.
We may have little specific information about AIs, but we have tons of information about feedback laws, and some information about self-improving systems in general*. I agree that it can be tricky to convert this information to a probability, but that just seems to be an argument against using probabilities in general. Whatever makes it hard to arrive at a good posterior should also make it hard to arrive at a good prior.
(I’m being slightly vague here for the purpose of exposition. I can make these statements more precise if you prefer.)
(* See for instance the Yudkowsky-Hanson AI Foom Debate.)
If the feedback constant is less than one, intelligence growth stops.
You should distinguish between exponential and linear growth.
First, the feedback constant is different for every level of intelligence.
Whenever the constant is greater than one, the machine is limited by making the transformations involved, and the intelligence is not well characterized as being limited by its intelligence and instead should be thought of as limited by its resources.
Whenever the constant is less than one and greater than zero, intelligence growth is only linear, but it is not zero. If the constant remains low enough for long enough, whole periods of time, series of iterations, that have some places where the constant is above one also have sub-exponential growth.
The relationship between the AI’s growth rate and our assisted intelligence growth rate (including FAIs, paper and pen, computers, etc.) is most of what is important, with the tie-breaker being our starting resources.
An AI with fast linear growth between patches of exponential growth, or even one with only fast linear growth, would quickly outclass humans’ thinking.
First, the feedback constant is different for every level of intelligence.
I meant to mention that, but I didn’t. It looks like I didn’t so much forget as write an answer so garbled you can’t really tell what I’m trying to say. I’ll fix that.
Anyway, it will move around as the intelligence changes, but I figure it would be far enough from one that it won’t cross it for a while. Either the intelligence is sufficiently advanced before the constant goes below one, or there’s no way you’d ever be able to get something intelligent enough to recursively self-improve.
Whenever the constant is less than one and greater than zero, intelligence growth is only linear, but it is not zero.
No, it’s zero, or at least asymptotic. If each additional IQ point allows you to work out how to grant yourself half an IQ point, you’ll only ever get twice as many extra IQ points as you started with.
Having extra time will be somewhat helpful, but this is limited. If you get extra time, you’d be able to accomplish harder problems, but you won’t be able to accomplish all problems. This will mean that the long-term feedback constant is somewhat higher, but if it’s nowhere near one to begin with, that won’t matter much.
Were you using “feedback constant” to mean the second derivative of intelligence, and assuming each increase in intelligence will be more difficult than the previous one (accounting for size difference)? I took “feedback constant” to mean the first derivative. I shouldn’t have used an existing term and should have said what i meant directly.
I used “feedback constant” to mean the amount of intelligence an additional unit of intelligence would allow you to bring (before using the additional unit of intelligence). For example, if at an IQ of 1000, you can design a brain with an IQ of 1010, but with an IQ of 1001, you can design a brain with an IQ of 10012, the feedback constant is two.
It’s the first derivative of the most intelligent brain you can design in terms of your own intelligence.
Looking at it again, it seems that the feedback constant and whether or not we are capable of designing better brains aren’t completely tied together. It may be that someone with an IQ of 100 can design a brain with an IQ of 10, and someone with an IQ of 101 can design a brain with an IQ of 12, so the feedback constant is two, but you can’t get enough intelligence in the first place. Similarly, the feedback constant could be less than one, but we could nonetheless be able to make brains more intelligent than us, just without an intelligence explosion. I’m not sure how much the two correlate.
I have a reason to believe it has less than a 50% chance of being possible. Does that count?
I figure after the low-hanging fruit is taken care of, it simply becomes a question of if a unit of additional intelligence is enough to add another additional unit of intelligence. If the feedback constant is less than one, intelligence growth stops. If it is greater, the intelligence grows until it falls below one. It will vary somewhat with intelligence, and it would have to fall below one eventually. We have no way of knowing what the feedback constant is, so we’d have to guess that there’s a 50% chance of it being above 1, and a 50% chance of being below. Furthermore, it’s unlikely to be right next to 1, so if it is possible, it will most likely get pretty darn intelligent.
Also, I figure that if the constant is below 1, or even close, humanity will be incapable of creating an AI from scratch, though if they get uploaded they could improve themselves.
Not being able to determine what the constant is doesn’t mean that there is a 50-50 chance that it is larger than 1. In particular, what in your logic prevents one from also concluding that there is also a 50-50 chance of it being larger than 2?
It can’t be less than zero. From what I understand about priors, the maximum entropy prior would be a logarithmic prior. A more reasonable prior would be a log-normal prior with the mean on 1 and a high standard deviation
By logarithmic do you mean p(x) = exp(-x)? That would only have an entropy of 1, I believe, whereas one can easily obtain unboundedly large amounts of entropy, or even infinite entropy (for instance, p(x) = a exp(-a x) has entropy 1-log(a), so letting a go to zero yields arbitrarily large entropy).
Also, as I’ve noted before, entropy doesn’t make that much sense for continuous distributions.
I mean p(x) = 1/x
I think it’s Jeffreys prior or something. Anyway, it seems like a good prior. It doesn’t have any arbitrary constants in it like you’d need with p(x) = exp(-x). If you change the scale, the prior stays the same.
p(x) = 1/x isn’t an integrable function (diverges at both 0 and infinity).
(My real objection is more that it’s pretty unlikely that we really have so little information that we have to quibble about which prior to use. It’s also good to be aware of the mathematical difficulties inherent in trying to be an “objective Bayesian”, but the real problem is that it’s not very helpful for making more accurate empirical predictions.)
Which is why I said a log-normal prior would be more reasonable.
How much information do we have? We know that we haven’t managed to build an AI in 40 years, and that’s about it.
We probably have enough information if we can process it right, but because we don’t know how, we’re best off sticking close to the prior.
Why a log-normal prior with mu = 0? Why not some other value for the location parameter? Log-normal makes pretty strong assumptions, which aren’t justified if we for all practical purposes we have no information about the feedback constant.
We may have little specific information about AIs, but we have tons of information about feedback laws, and some information about self-improving systems in general*. I agree that it can be tricky to convert this information to a probability, but that just seems to be an argument against using probabilities in general. Whatever makes it hard to arrive at a good posterior should also make it hard to arrive at a good prior.
(I’m being slightly vague here for the purpose of exposition. I can make these statements more precise if you prefer.)
(* See for instance the Yudkowsky-Hanson AI Foom Debate.)
You should distinguish between exponential and linear growth.
First, the feedback constant is different for every level of intelligence.
Whenever the constant is greater than one, the machine is limited by making the transformations involved, and the intelligence is not well characterized as being limited by its intelligence and instead should be thought of as limited by its resources.
Whenever the constant is less than one and greater than zero, intelligence growth is only linear, but it is not zero. If the constant remains low enough for long enough, whole periods of time, series of iterations, that have some places where the constant is above one also have sub-exponential growth.
The relationship between the AI’s growth rate and our assisted intelligence growth rate (including FAIs, paper and pen, computers, etc.) is most of what is important, with the tie-breaker being our starting resources.
An AI with fast linear growth between patches of exponential growth, or even one with only fast linear growth, would quickly outclass humans’ thinking.
I meant to mention that, but I didn’t. It looks like I didn’t so much forget as write an answer so garbled you can’t really tell what I’m trying to say. I’ll fix that.
Anyway, it will move around as the intelligence changes, but I figure it would be far enough from one that it won’t cross it for a while. Either the intelligence is sufficiently advanced before the constant goes below one, or there’s no way you’d ever be able to get something intelligent enough to recursively self-improve.
No, it’s zero, or at least asymptotic. If each additional IQ point allows you to work out how to grant yourself half an IQ point, you’ll only ever get twice as many extra IQ points as you started with.
Having extra time will be somewhat helpful, but this is limited. If you get extra time, you’d be able to accomplish harder problems, but you won’t be able to accomplish all problems. This will mean that the long-term feedback constant is somewhat higher, but if it’s nowhere near one to begin with, that won’t matter much.
Were you using “feedback constant” to mean the second derivative of intelligence, and assuming each increase in intelligence will be more difficult than the previous one (accounting for size difference)? I took “feedback constant” to mean the first derivative. I shouldn’t have used an existing term and should have said what i meant directly.
I used “feedback constant” to mean the amount of intelligence an additional unit of intelligence would allow you to bring (before using the additional unit of intelligence). For example, if at an IQ of 1000, you can design a brain with an IQ of 1010, but with an IQ of 1001, you can design a brain with an IQ of 10012, the feedback constant is two.
It’s the first derivative of the most intelligent brain you can design in terms of your own intelligence.
Looking at it again, it seems that the feedback constant and whether or not we are capable of designing better brains aren’t completely tied together. It may be that someone with an IQ of 100 can design a brain with an IQ of 10, and someone with an IQ of 101 can design a brain with an IQ of 12, so the feedback constant is two, but you can’t get enough intelligence in the first place. Similarly, the feedback constant could be less than one, but we could nonetheless be able to make brains more intelligent than us, just without an intelligence explosion. I’m not sure how much the two correlate.