I wrote “etc.”, but really the main ones I can think of are probably the ones I listed there. Let’s start with correlations since this is the really old-school one.
The basic causal principle behind a correlation/covariance-based method is that if you see a correlation where the same thing appears in different places, then that correlation is due to there being a shared cause. This is in particular useful for representation learning, because the shared cause is likely not just an artifact of your perception (“this pixel is darker or lighter”) but instead a feature of the world itself (“the scene depicted in this image has properties XYZ”). This then leads to the insight of Factor Analysis[1]: it’s easy to set up a linear generative model with a fixed number of independent latent variables to model your data.
Factor Analysis still gets used a lot in various fields like psychology, but for machine perception it’s too bad because perception requires nonlinearity. (Eigenfaces are a relic of the past.) However, the core concept that correlations imply latent variables, and that these latent variables are likely more meaningful features of reality contains to be relevant in many models:
Variational autoencoders, generative adversarial networks, etc., learn to encode the distribution of images, and tend to contain a meaningful “latent space” that you can use to generate counterfactual images at a high level of abstraction. They rely on covariances because they must fit the latent variables so that they capture the covariances between different parts of the images.
Triplet loss for e.g. facial recognition tries to filter out the features that correlate for different images of a single person, vs the features that do not correlate for people on different images and are thus presumably artifacts of the image.
Covariance-based techniques aren’t the only game in town, though; a major alternate one is symmetries. Often, we know that some symmetries hold; for instance reality is symmetric under translations, but images have to pick some particular translation to capture only a specific part of reality. However, small perturbations in the translation of an image do not matter much for its contents, as the contents tends to be extended geometrically over a large part of the image. Therefore, if you do random cropping of your images, you still get basically the same images. Similarly, various other data augmentation methods can be viewed as being about symmetries. In particular there are a number of symmetry-oriented feature learners:
Facebook’s DINO model (as well as various other models in this general class) trains an embedding to be invariant to various symmetries. As a result, it learns to pay attention to the objects in the scene.
I really like the concept in pi-GAN. Rather than training it with symmetries given by image augmentation over the input data, they train it using rotation symmetries over the latent variables. This allows it to learn 3D representations of faces and objects in an unsupervised fashion.
Finally, there is the notion of conserved properties. John Wentworth has persuasively argued that variables that act at a distance must have deterministically conserved mediators. I am only aware of one architecture that uses this insight, though, namely Noether Networks. Basically, Noether Networks are trained to identify features where your predictions can be improved by assuming the features to stay constant. They do not directly use these features for much, but they seem potentially promising for the future.
This set is probably not exhaustive. There’s probably methods that are hyper-specialized to specific domains, and there’s probably also other general methods. But I think it’s a good general taste of what’s available?
Factor Analysis might not be a familiar algorithm to you, but it is computationally basically equivalent to Principal Component Analysis, which you are almost certainly familiar with. There’s only a few caveats for this, and they don’t really matter for the purposes of this post.
You state in the first comment that they can be given causal justification. As far as I understand you argue with covariances above. Can you elaborate on what this causal justification is?
In a causal universe, if you observe things in different places that correlate with each other, they must have a common cause. That’s the principle VAEs/triplet losses/etc. can be understood as exploiting.
Right, but Reichenbach’s principle of common cause doesn’t tell you anything about how they are causally related? They could just be some nodes in a really large complicated causal graph. So I agree that we can assume causality somehow but we are much more interested in how the graph looks like, right?
So I agree that we can assume causality somehow but we are much more interested in how the graph looks like, right?
Not necessarily? Reality is really really big. It would be computationally infeasible to work with raw reality. Rather, you want abstractions that cover aggregate causality in a computationally practical way, throwing away most of the causal details. See also this:
Sure!
I wrote “etc.”, but really the main ones I can think of are probably the ones I listed there. Let’s start with correlations since this is the really old-school one.
The basic causal principle behind a correlation/covariance-based method is that if you see a correlation where the same thing appears in different places, then that correlation is due to there being a shared cause. This is in particular useful for representation learning, because the shared cause is likely not just an artifact of your perception (“this pixel is darker or lighter”) but instead a feature of the world itself (“the scene depicted in this image has properties XYZ”). This then leads to the insight of Factor Analysis[1]: it’s easy to set up a linear generative model with a fixed number of independent latent variables to model your data.
Factor Analysis still gets used a lot in various fields like psychology, but for machine perception it’s too bad because perception requires nonlinearity. (Eigenfaces are a relic of the past.) However, the core concept that correlations imply latent variables, and that these latent variables are likely more meaningful features of reality contains to be relevant in many models:
Variational autoencoders, generative adversarial networks, etc., learn to encode the distribution of images, and tend to contain a meaningful “latent space” that you can use to generate counterfactual images at a high level of abstraction. They rely on covariances because they must fit the latent variables so that they capture the covariances between different parts of the images.
Triplet loss for e.g. facial recognition tries to filter out the features that correlate for different images of a single person, vs the features that do not correlate for people on different images and are thus presumably artifacts of the image.
Covariance-based techniques aren’t the only game in town, though; a major alternate one is symmetries. Often, we know that some symmetries hold; for instance reality is symmetric under translations, but images have to pick some particular translation to capture only a specific part of reality. However, small perturbations in the translation of an image do not matter much for its contents, as the contents tends to be extended geometrically over a large part of the image. Therefore, if you do random cropping of your images, you still get basically the same images. Similarly, various other data augmentation methods can be viewed as being about symmetries. In particular there are a number of symmetry-oriented feature learners:
Facebook’s DINO model (as well as various other models in this general class) trains an embedding to be invariant to various symmetries. As a result, it learns to pay attention to the objects in the scene.
I really like the concept in pi-GAN. Rather than training it with symmetries given by image augmentation over the input data, they train it using rotation symmetries over the latent variables. This allows it to learn 3D representations of faces and objects in an unsupervised fashion.
Finally, there is the notion of conserved properties. John Wentworth has persuasively argued that variables that act at a distance must have deterministically conserved mediators. I am only aware of one architecture that uses this insight, though, namely Noether Networks. Basically, Noether Networks are trained to identify features where your predictions can be improved by assuming the features to stay constant. They do not directly use these features for much, but they seem potentially promising for the future.
This set is probably not exhaustive. There’s probably methods that are hyper-specialized to specific domains, and there’s probably also other general methods. But I think it’s a good general taste of what’s available?
Factor Analysis might not be a familiar algorithm to you, but it is computationally basically equivalent to Principal Component Analysis, which you are almost certainly familiar with. There’s only a few caveats for this, and they don’t really matter for the purposes of this post.
Nice summary, thanks for sharing!
You state in the first comment that they can be given causal justification. As far as I understand you argue with covariances above. Can you elaborate on what this causal justification is?
In a causal universe, if you observe things in different places that correlate with each other, they must have a common cause. That’s the principle VAEs/triplet losses/etc. can be understood as exploiting.
Right, but Reichenbach’s principle of common cause doesn’t tell you anything about how they are causally related? They could just be some nodes in a really large complicated causal graph. So I agree that we can assume causality somehow but we are much more interested in how the graph looks like, right?
Not necessarily? Reality is really really big. It would be computationally infeasible to work with raw reality. Rather, you want abstractions that cover aggregate causality in a computationally practical way, throwing away most of the causal details. See also this:
https://www.lesswrong.com/posts/Gkv2TCbeE9jjMHXKR/reductionism-is-not-the-ultimate-tool-for-causal-inference