As a result, the model learned heuristic H, that works in all the circumstances you did consider, but fails in circumstance C.
That’s basically where I start, but then I want to try to tell some story about why it kills you, i.e. what is it about the heuristic H and circumstance C that causes it to kill you?
I agree this involves discretion, and indeed moving beyond the trivial story “The algorithm fails and then it turns out you die” requires discretion, since those stories are certainly plausible. The other extreme would be to require us to keep making the story more and more concrete until we had fully specified the model, which also seems intractable. So instead I’m doing some in between thing, which is roughly like: I’m allowed to push on the story to make it more concrete along any axis, but I recognize that I won’t have time to pin down every axis so I’m basically only going to do this a bounded number of times before I have to admit that it seems plausible enough (so I can’t fill in a billion parameters of my model one by one this way; what’s worse, filling in those parameters would take even more than a billion time and so this may become intractable even before you get to a billion).
I agree this involves discretion [...] So instead I’m doing some in between thing
Yeah, I think I feel like that’s the part where I don’t think I could replicate your intuitions (yet).
I don’t think we disagree; I’m just noting that this methodology requires a fair amount of intuition / discretion, and I don’t feel like I could do this myself. This is much more a statement about what I can do, rather than a statement about how good the methodology is on some absolute scale.
(Probably I could have been clearer about this in the original opinion.)
That’s basically where I start, but then I want to try to tell some story about why it kills you, i.e. what is it about the heuristic H and circumstance C that causes it to kill you?
I agree this involves discretion, and indeed moving beyond the trivial story “The algorithm fails and then it turns out you die” requires discretion, since those stories are certainly plausible. The other extreme would be to require us to keep making the story more and more concrete until we had fully specified the model, which also seems intractable. So instead I’m doing some in between thing, which is roughly like: I’m allowed to push on the story to make it more concrete along any axis, but I recognize that I won’t have time to pin down every axis so I’m basically only going to do this a bounded number of times before I have to admit that it seems plausible enough (so I can’t fill in a billion parameters of my model one by one this way; what’s worse, filling in those parameters would take even more than a billion time and so this may become intractable even before you get to a billion).
Yeah, I think I feel like that’s the part where I don’t think I could replicate your intuitions (yet).
I don’t think we disagree; I’m just noting that this methodology requires a fair amount of intuition / discretion, and I don’t feel like I could do this myself. This is much more a statement about what I can do, rather than a statement about how good the methodology is on some absolute scale.
(Probably I could have been clearer about this in the original opinion.)