This seems like a promising direction that I tentatively agree with. It sounds similar to the “iterative natural kind” strategy that Megan Peters mentions here, though her approach is a bit more formal. (Would be curious if that sounds right to you, though no need to answer.)
I would think the problem is that your judgments about when to revise your top-down theory end up being ad hoc. The GWT/MoE example is making a judgment call about how similar the two structures are, but it’s fundamentally just a judgment call. Probably we all do and will in fact make judgment calls about which theories we endorse in part based on what they imply about the world, and so this is just admitting that and doing it more honestly and deliberately, but I guess it’s just unfortunate that this is what we have to go with.
Thanks! I haven’t listened yet but I think the iterative natural kind approach is quite similar indeed.
Agreed that you’re going to need to make judgement calls one way or another. But these don’t have to be final! You can also run follow-up investigations to gather new information. This should make judgement calls easier (but maybe bring up some new ones). One nice thing with models (compared to humans) is that it’s much easier to run experiments on them. E.g. we can resample and we have access to activations. So i would argue iterative approaches are less bottlenecked by judgement calls.
This seems like a promising direction that I tentatively agree with. It sounds similar to the “iterative natural kind” strategy that Megan Peters mentions here, though her approach is a bit more formal. (Would be curious if that sounds right to you, though no need to answer.)
I would think the problem is that your judgments about when to revise your top-down theory end up being ad hoc. The GWT/MoE example is making a judgment call about how similar the two structures are, but it’s fundamentally just a judgment call. Probably we all do and will in fact make judgment calls about which theories we endorse in part based on what they imply about the world, and so this is just admitting that and doing it more honestly and deliberately, but I guess it’s just unfortunate that this is what we have to go with.
Thanks! I haven’t listened yet but I think the iterative natural kind approach is quite similar indeed.
Agreed that you’re going to need to make judgement calls one way or another. But these don’t have to be final! You can also run follow-up investigations to gather new information. This should make judgement calls easier (but maybe bring up some new ones). One nice thing with models (compared to humans) is that it’s much easier to run experiments on them. E.g. we can resample and we have access to activations. So i would argue iterative approaches are less bottlenecked by judgement calls.