I agree that all exponentials eventually end, but I don’t think this matters that much, because trying to predict the future using sigmoids/functions that don’t grow without bound is also very hard, and you need to have very good priors or very low noise data to not be very off, so an exponential is usually a better fit unless you have reason to believe you’ve identified a specific upper bound and the dampening term:
A key crux is I basically disagree with the assumption that you can reliably detect either the dampening term or the upper bound in early data, because I expect real-life data to be far too noisy to figure both of these out, and this is especially the case in AI.
There are special cases like disease modeling (though even there predictions can be wildly off), but in general I think the exponential fit is almost always better than a sigmoid/function that doesn’t grow without bound unless you have strong reason to believe you’ve identified both the dampening term and the upper bound.
There are special cases like disease modeling (though even there predictions can be wildly off), but in general I think the exponential fit is almost always better than a sigmoid/function that doesn’t grow without bound unless you have strong reason to believe you’ve identified both the dampening term and the upper bound.
I find these arguments about fit strange, like they are failing to remember what we’re trying to do with predictions.
We want to have an accurate model of the future. That exponentials fit data better is nice, and maybe they are easier to predict the next point under many circumstances, but they are leaving something out.
Like suppose I’m trying to predict if I’ll be alive tomorrow. I could have a naive model that predicts I will be because I was alive every previous day. But this model is wrong in a very important way: one day I will die! That the model fails to account for this fact makes the model less useful, because even if it’s right for a long time, eventually it won’t be, and it’s a failure of the model that I’d get surprised to be dead.
I agree that all exponentials eventually end, but I don’t think this matters that much, because trying to predict the future using sigmoids/functions that don’t grow without bound is also very hard, and you need to have very good priors or very low noise data to not be very off, so an exponential is usually a better fit unless you have reason to believe you’ve identified a specific upper bound and the dampening term:
Why sigmoids are so hard to predict
Sigmoids behaving badly: why they usually cannot predict the future as well as they seem to promise (paper)
A key crux is I basically disagree with the assumption that you can reliably detect either the dampening term or the upper bound in early data, because I expect real-life data to be far too noisy to figure both of these out, and this is especially the case in AI.
I see this comment as making the assumption that you can actually just guess an upper bound without being OOMs wrong in the general case.
There are special cases like disease modeling (though even there predictions can be wildly off), but in general I think the exponential fit is almost always better than a sigmoid/function that doesn’t grow without bound unless you have strong reason to believe you’ve identified both the dampening term and the upper bound.
I find these arguments about fit strange, like they are failing to remember what we’re trying to do with predictions.
We want to have an accurate model of the future. That exponentials fit data better is nice, and maybe they are easier to predict the next point under many circumstances, but they are leaving something out.
Like suppose I’m trying to predict if I’ll be alive tomorrow. I could have a naive model that predicts I will be because I was alive every previous day. But this model is wrong in a very important way: one day I will die! That the model fails to account for this fact makes the model less useful, because even if it’s right for a long time, eventually it won’t be, and it’s a failure of the model that I’d get surprised to be dead.