AIFM is extremely sensitive to time horizon in a way I wouldn’t endorse.
In particular, the “doubling difficulty growth factor”, which measures whether time horizon increases superexponentially, could change the date of automated coder from 2028 to 2049!
You say that like it’s a bad thing. This sounds like the common implicit assumption that precise models are better. Good models capture reality. Reality in this case seems highly uncertain.
It’s possible for there to be irreducible uncertainty when modeling things, but AIFM is using time horizon to predict uplift / automation, and my claim is that we should collect data on uplift directly instead, which would bypass the poorly understood relationship between time horizon and uplift.
I actually do think parallel uplift is something like 1.4x-2.5x. This is just an educated guess but so are AIFM’s credible intervals for time horizon trends. There’s not really an empirical basis for AIF’s guess that something like a 125 human-year time horizon AI will be just sufficient to automate all coding, nor that each time horizon doubling will be say 10% faster than the last. The uplift assumption naturally leads to narrower timelines uncertainty than the time horizon ones.
If and when we can prove something about uplift we’ll hopefully publish more than a research note. At this point I would revisit the modeling assumptions too, and the end result may or may not be wide uncertainty. But you shouldn’t just artificially make your confidence intervals wider because your all-things-considered view has wide uncertainty. More likely, a mismatch between them reflects factors you’re not modeling, and you have to choose whether it’s worth the cost to include them or not.
My point was that you mentioned removing a source of uncertainty in the model as though that’s by default a good thing. The exclamation point on your mention of the wide uncertainty seems to imply that pretty strongly. I still don’t know if you endorse that general attitude. I was not arguing that you shouldn’t have removed that factor; I was arguing that you didn’t argue for why removing it was good, but implied it was good anyway.
Perhaps this is a nitpick. But in my mind this point is about taking into account how your models are used and interpreted. It seems to me that precise models tend to cause overconfidence, so it would be wiser to err in the direction of including uncertainty in the models, rather than letting that uncertainty sit in complex grounding assumptions.
You say that like it’s a bad thing.
This sounds like the common implicit assumption that precise models are better.
Good models capture reality. Reality in this case seems highly uncertain.
It’s possible for there to be irreducible uncertainty when modeling things, but AIFM is using time horizon to predict uplift / automation, and my claim is that we should collect data on uplift directly instead, which would bypass the poorly understood relationship between time horizon and uplift.
That does seem like it would be better. If you haven’t done that yet, you need to keep the estimate in there with its wide uncertainty, right?
I actually do think parallel uplift is something like 1.4x-2.5x. This is just an educated guess but so are AIFM’s credible intervals for time horizon trends. There’s not really an empirical basis for AIF’s guess that something like a 125 human-year time horizon AI will be just sufficient to automate all coding, nor that each time horizon doubling will be say 10% faster than the last. The uplift assumption naturally leads to narrower timelines uncertainty than the time horizon ones.
If and when we can prove something about uplift we’ll hopefully publish more than a research note. At this point I would revisit the modeling assumptions too, and the end result may or may not be wide uncertainty. But you shouldn’t just artificially make your confidence intervals wider because your all-things-considered view has wide uncertainty. More likely, a mismatch between them reflects factors you’re not modeling, and you have to choose whether it’s worth the cost to include them or not.
My point was that you mentioned removing a source of uncertainty in the model as though that’s by default a good thing. The exclamation point on your mention of the wide uncertainty seems to imply that pretty strongly. I still don’t know if you endorse that general attitude. I was not arguing that you shouldn’t have removed that factor; I was arguing that you didn’t argue for why removing it was good, but implied it was good anyway.
Perhaps this is a nitpick. But in my mind this point is about taking into account how your models are used and interpreted. It seems to me that precise models tend to cause overconfidence, so it would be wiser to err in the direction of including uncertainty in the models, rather than letting that uncertainty sit in complex grounding assumptions.
Naturally every model will have some of both.