I am interested in semantic distance, but before that I am interested in semantic continuity. I think the idealized topology wouldn’t have a metric, but the geometric spaces in which that topology is embedded gives it semantic distance, implicitly giving it a metric.
For example, in image space slight changes in lighting would give small distances, but translating or rotating an image would move it a very great distance away. So the visual space is great for humans to look at, but the semantic metric describes things about pixel similarity that we usually don’t care about outside of computer algorithms.
The labelling space would have a much more useful metric. Assuming a 1d logit, distance would correspond to how much something does or does not seem like a cat. With 2d or more logits the situation would become more complicated, but again, distance represents motion towards or away from confidence of whether we’re looking at a cat, a dog, or something else.
But in both cases, the metric is a choice that tells you something about certain kinds of semantics. I’m not confident there would exist a universal metric for semantic distance.
You could define the topology on the output space so that by definition the network is continuous (quotient topology) but then topology really is getting you nothing.
I’d actually be more inclined to do this. I agree it immediately gets you nothing, but it becomes more interesting when you start asking questions like “what are the open sets” and “what do the open sets look like in the latent spaces”.
Bringing back the cat identifier net, if I look at the set of high cat confidence, will the preimage be the set of all images that are definitely cats? I think that’s a common intuition, but could we prove it? Would there be a way to systematically explore diverse sections of that preimage to verify that they are indeed all definitely cats?
The fact that it’s starting from a trivial assertion doesn’t make it a bad place to start exploring imo.
I think that kinda direction might be what you’re getting at mentioning “informal ideas I discuss in between the topology slop”. So it’s true, I might stop thinking in terms of topology eventually, but for now I think it’s helping guide my thinking. I want to try to move towards thinking in terms of manifolds, and I think noticing the idea of semantic connectivity, ie, a semantic topological space, without requiring the idea of semantic distance is worthwhile.
I think that might be one of the ideas I’m trying to zero in on: The distributions in the data are always the same and what networks do is change from embedding that distribution in one geometry to embedding it in a different geometry which has different (more useful?) semantic properties.
I am interested in semantic distance, but before that I am interested in semantic continuity. I think the idealized topology wouldn’t have a metric, but the geometric spaces in which that topology is embedded gives it semantic distance, implicitly giving it a metric.
For example, in image space slight changes in lighting would give small distances, but translating or rotating an image would move it a very great distance away. So the visual space is great for humans to look at, but the semantic metric describes things about pixel similarity that we usually don’t care about outside of computer algorithms.
The labelling space would have a much more useful metric. Assuming a 1d logit, distance would correspond to how much something does or does not seem like a cat. With 2d or more logits the situation would become more complicated, but again, distance represents motion towards or away from confidence of whether we’re looking at a cat, a dog, or something else.
But in both cases, the metric is a choice that tells you something about certain kinds of semantics. I’m not confident there would exist a universal metric for semantic distance.
I’d actually be more inclined to do this. I agree it immediately gets you nothing, but it becomes more interesting when you start asking questions like “what are the open sets” and “what do the open sets look like in the latent spaces”.
Bringing back the cat identifier net, if I look at the set of high cat confidence, will the preimage be the set of all images that are definitely cats? I think that’s a common intuition, but could we prove it? Would there be a way to systematically explore diverse sections of that preimage to verify that they are indeed all definitely cats?
The fact that it’s starting from a trivial assertion doesn’t make it a bad place to start exploring imo.
I think that kinda direction might be what you’re getting at mentioning “informal ideas I discuss in between the topology slop”. So it’s true, I might stop thinking in terms of topology eventually, but for now I think it’s helping guide my thinking. I want to try to move towards thinking in terms of manifolds, and I think noticing the idea of semantic connectivity, ie, a semantic topological space, without requiring the idea of semantic distance is worthwhile.
I think that might be one of the ideas I’m trying to zero in on: The distributions in the data are always the same and what networks do is change from embedding that distribution in one geometry to embedding it in a different geometry which has different (more useful?) semantic properties.
Good luck with it. I do think the broad direction is pretty promising.
Thanks : )