I think you sufficiently addressed my confusion, so you don’t need to reply to this comment, but I still had a few responses to what you said.
What does this mean? On my understanding, singularities don’t proceed at fixed rates?
No, I agree. But growth is generally measured over an interval. In the original comment I proposed the interval of one year during the peak rate of economic growth. To allay your concern that a 25% growth rate indicates we didn’t experience a singularity, I meant that we were halving the growth rate during the peak economic growth year in our future, regardless of whether that rate was very fast.
I agree that in practice there will be some maximum rate of GDP growth, because there are fundamental physical limits (and more tight in-practice limits that we don’t know), but it seems like they’ll be way higher than 25% per year.
The 25% figure was totally arbitrary. I didn’t mean it as any sort of prediction. I agree that an extrapolation from biological growth implies that we can and should see >1000% growth rates eventually, though it seems plausible that we would coordinate to avoid that.
If you actually mean halving the peak rate of GDP growth during the singularity, and a singularity actually happens, then I think it doesn’t affect my actions at all; all of the relevant stuff happened well before we get to the peak rate.
That’s reasonable. A separate question might be about whether the rate of growth during the entire duration from now until the peak rate will cut in half.
Let’s say for purposes of argument I think 10% chance of extinction, and 90% chance of “moderate costs but nothing terrible”. Which of the following am I supposed to have updated to?
I think the way you’re bucketing this into “costs if we go extinct” and “costs if we don’t go extinct” is reasonable. But one could also think that the disvalue of extinction is more continuous with disvalue in non-extinction scenarios, which makes things a bit more tricky. I hope that makes sense.
But one could also think that the disvalue of extinction is more continuous with disvalue in non-extinction scenarios, which makes things a bit more tricky.
I’m happy to use continuous notions (and that’s what I was doing in my original comment) as long as “half the cost” means “you update such that the expected costs of misalignment according to your probability distribution over the future are halved”. One simple way to imagine this update is to take all the worlds where there was any misalignment, halve their probability, and distribute the extra probability mass to worlds with zero costs of misalignment. At which point I reason “well, 10% extinction changes to 5% extinction, I don’t need to know anything else to know that I’m still going to work on alignment, and given that, none of my actions are going to change (since the relative probabilities of different misalignment failure scenarios remain the same, which is what determines my actions within alignment)”.
I got the sense from your previous comment that you wanted me to imagine some different form of update and I was trying to figure out what.
I think you sufficiently addressed my confusion, so you don’t need to reply to this comment, but I still had a few responses to what you said.
No, I agree. But growth is generally measured over an interval. In the original comment I proposed the interval of one year during the peak rate of economic growth. To allay your concern that a 25% growth rate indicates we didn’t experience a singularity, I meant that we were halving the growth rate during the peak economic growth year in our future, regardless of whether that rate was very fast.
The 25% figure was totally arbitrary. I didn’t mean it as any sort of prediction. I agree that an extrapolation from biological growth implies that we can and should see >1000% growth rates eventually, though it seems plausible that we would coordinate to avoid that.
That’s reasonable. A separate question might be about whether the rate of growth during the entire duration from now until the peak rate will cut in half.
I think the way you’re bucketing this into “costs if we go extinct” and “costs if we don’t go extinct” is reasonable. But one could also think that the disvalue of extinction is more continuous with disvalue in non-extinction scenarios, which makes things a bit more tricky. I hope that makes sense.
Cool, that all makes sense.
I’m happy to use continuous notions (and that’s what I was doing in my original comment) as long as “half the cost” means “you update such that the expected costs of misalignment according to your probability distribution over the future are halved”. One simple way to imagine this update is to take all the worlds where there was any misalignment, halve their probability, and distribute the extra probability mass to worlds with zero costs of misalignment. At which point I reason “well, 10% extinction changes to 5% extinction, I don’t need to know anything else to know that I’m still going to work on alignment, and given that, none of my actions are going to change (since the relative probabilities of different misalignment failure scenarios remain the same, which is what determines my actions within alignment)”.
I got the sense from your previous comment that you wanted me to imagine some different form of update and I was trying to figure out what.