50%, 75%, 87.5%, 93.75%, … are linear jumps in predictive accuracy (one bit each), but returns seem to diminish at an exponential rate
On the other hand 6.25%, 12.5%, 25%, 50% represent the same linear jumps, but this time with returns growing at an exponential returns
This suggests that the nature of returns to cognitive investment might exhibit differing behaviour depending on where in the cognitive capabilities curve you are
Though I’ve not yet thought about how this behaviour generalises to other aspects of cognition separate from predictive accuracy
But if you look at the length of a chain of reasoning you can do while staying under 10% error or something, then returns don’t diminish at all in terms of predictive accuracy. Going from 99% to 99.9% accuracy lets you plan 10 times further ahead in the future with the same final accuracy.
Take for instance the accuracy of humans in classifying daily objects (chairs, pencils, doors, that sort of stuff), for which I’d think we have a greater than 99.9% accuracy, and I’m being conservative here, I don’t misclassify every 1 in 1000 objects I see in daily life. If that accuracy dropped to 99%, you’d make noticeable mistakes pretty often and long tasks would get tricky. If it dropped to 90%, you’d be very severely impaired and I don’t think you could function in society, seems to me like the returns scale with predictive accuracy, not the linear probabilities.
If I go to pick up an item on my desk, and I accidentally grab the wrong item, it’s no big deal, I just put it down and grab the right item. I have no idea how often this happens for a typical human, but I bet it’s way more than 0.1%. When I was younger, I was very clumsy and I often spilled my drink. It never impaired my ability to function in society.
I don’t misclassify every 1 in 1000 objects I see in daily life
Perhaps this is too nitpicky about semantics, but when I tried to evaluate the plausibility of this claim I came into a bit of an impasse.
You’re supposing that there is a single natural category for each object you see (and buries even the definition of an object, which isn’t obvious to me). I’d agree that you probably classify a given thing the same way >99.9% of the time. However, what would the inter-rater reliability be for these classifications? Are those classifications actually “correct” in a natural sense?
But if you look at the length of a chain of reasoning you can do while staying under 10% error or something, then returns don’t diminish at all in terms of predictive accuracy. Going from 99% to 99.9% accuracy lets you plan 10 times further ahead in the future with the same final accuracy.
Take for instance the accuracy of humans in classifying daily objects (chairs, pencils, doors, that sort of stuff), for which I’d think we have a greater than 99.9% accuracy, and I’m being conservative here, I don’t misclassify every 1 in 1000 objects I see in daily life. If that accuracy dropped to 99%, you’d make noticeable mistakes pretty often and long tasks would get tricky. If it dropped to 90%, you’d be very severely impaired and I don’t think you could function in society, seems to me like the returns scale with predictive accuracy, not the linear probabilities.
I don’t think humans are anywhere near that accurate. For example, ImageNet was found to have 6% label errors.
If I go to pick up an item on my desk, and I accidentally grab the wrong item, it’s no big deal, I just put it down and grab the right item. I have no idea how often this happens for a typical human, but I bet it’s way more than 0.1%. When I was younger, I was very clumsy and I often spilled my drink. It never impaired my ability to function in society.
Perhaps this is too nitpicky about semantics, but when I tried to evaluate the plausibility of this claim I came into a bit of an impasse.
You’re supposing that there is a single natural category for each object you see (and buries even the definition of an object, which isn’t obvious to me). I’d agree that you probably classify a given thing the same way >99.9% of the time. However, what would the inter-rater reliability be for these classifications? Are those classifications actually “correct” in a natural sense?