It also makes me think of getting from maps (in your sense of abstraction) to the original variable. Because the simple model has a basic abstraction, a map built from throwing a lot of information; and you want to build a better map. So you’re asking how to decide what to add to the map to improve it, without having access to the initial variable. Or maybe differently, you have a bunch of simple abstractions (from the simple models), a bunch of more complex abstractions (from the complex model), and given a simple abstraction, you want to find the best complex abstraction that “improve” the simple abstraction. Is that correct?
For a solution, here’s a possible idea if the complex model is also a cluster model, and both models are trained on the same data: look for the learned cluster in the complex model with the bigger intersection with the detective story cluster in the simple model. Obvious difficulties are “how do you compute that intersection?”, and for more general complex models (like GPT-N), “how to even find the clusters?” Still, I would say that this idea satisfies your requirement of improving the concept if the initial one is good enough (most of the “true” cluster lies in the simple model’s cluster/most of the simple model’s cluster lies in the “true” cluster).
Your thoughts on abstraction here are exactly on the mark.
For the cluster-intersection thing, I think the biggest barrier is that “biggest cluster intersection” implicitly assumes some space in which the cluster intersection size is measured—some features, or some distance metric, or something along those lines. A more complex model will likely be using a different feature space, a different implicit metric, etc.
That looks like a fun problem.
It also makes me think of getting from maps (in your sense of abstraction) to the original variable. Because the simple model has a basic abstraction, a map built from throwing a lot of information; and you want to build a better map. So you’re asking how to decide what to add to the map to improve it, without having access to the initial variable. Or maybe differently, you have a bunch of simple abstractions (from the simple models), a bunch of more complex abstractions (from the complex model), and given a simple abstraction, you want to find the best complex abstraction that “improve” the simple abstraction. Is that correct?
For a solution, here’s a possible idea if the complex model is also a cluster model, and both models are trained on the same data: look for the learned cluster in the complex model with the bigger intersection with the detective story cluster in the simple model. Obvious difficulties are “how do you compute that intersection?”, and for more general complex models (like GPT-N), “how to even find the clusters?” Still, I would say that this idea satisfies your requirement of improving the concept if the initial one is good enough (most of the “true” cluster lies in the simple model’s cluster/most of the simple model’s cluster lies in the “true” cluster).
Your thoughts on abstraction here are exactly on the mark.
For the cluster-intersection thing, I think the biggest barrier is that “biggest cluster intersection” implicitly assumes some space in which the cluster intersection size is measured—some features, or some distance metric, or something along those lines. A more complex model will likely be using a different feature space, a different implicit metric, etc.