I’m slightly confused here by the results here. Is there idea that pictures with texta drawn over them are out of distribution?
If so, should I be more impressed by the difference between pic 1 and pic 2? It doesn’t seem to be that much, although it’s hard to tell from just seeing the metric from 4 images.
I’m also somewhat confused about why this works. It seems to me that within a distribution it’s perfectly fine to have some areas where it changes quickly and some where it changes slowly. And it seems like this would break this metric. I assume there’s something obvious that I’m missing.
Abstraction is about what information you throw away. For a ReLU activation function, all negative inputs are mapped to zero—you lose information there, in a way that you don’t when applying a linear transformation.
Imagine your model (or a submodule thereof) as a mapping from one vector space to another. In order to focus on the features relevant to the questions you care about (is the image a truck, is it a lizard, …) you throw away information that is not relevant to these questions—you give it less real-estate in your representation-space. We can expect more out-of-distribution regions of input-space to be “pinched” by the model—they’re not represented as expressively as are the more in-distribution regions of input-space.
So if your cosine-similarity decreases less when you nudge your input, you’re in a more “pinched” region of input-space, and if it decreases more, you’re in a more “expanded” region of input space—which means the model was tuned to focus on that region, which means it’s more in-distribution.
Is there idea that pictures with texta drawn over them are out of distribution?
Yes, the idea is that images that have been taken with a camera were present in the training set, whereas images that were taken with a camera and then scribbled on in GIMP were not.
If you refer to section 4.2 in the paper that leogao linked, those authors also use “corrupted input detection” to benchmark their method. You’re also welcome to try it on your own images—to run the code you just have to install the pip dependencies and then use paths to your own files. (If you uncomment the block at the bottom, you can run it off your webcam in real-time!)
I’m slightly confused here by the results here. Is there idea that pictures with texta drawn over them are out of distribution?
If so, should I be more impressed by the difference between pic 1 and pic 2? It doesn’t seem to be that much, although it’s hard to tell from just seeing the metric from 4 images.
I’m also somewhat confused about why this works. It seems to me that within a distribution it’s perfectly fine to have some areas where it changes quickly and some where it changes slowly. And it seems like this would break this metric. I assume there’s something obvious that I’m missing.
Abstraction is about what information you throw away. For a ReLU activation function, all negative inputs are mapped to zero—you lose information there, in a way that you don’t when applying a linear transformation.
Imagine your model (or a submodule thereof) as a mapping from one vector space to another. In order to focus on the features relevant to the questions you care about (is the image a truck, is it a lizard, …) you throw away information that is not relevant to these questions—you give it less real-estate in your representation-space. We can expect more out-of-distribution regions of input-space to be “pinched” by the model—they’re not represented as expressively as are the more in-distribution regions of input-space.
So if your cosine-similarity decreases less when you nudge your input, you’re in a more “pinched” region of input-space, and if it decreases more, you’re in a more “expanded” region of input space—which means the model was tuned to focus on that region, which means it’s more in-distribution.
Ok, that’s fascinating! Thanks for the explanation.
Yes, the idea is that images that have been taken with a camera were present in the training set, whereas images that were taken with a camera and then scribbled on in GIMP were not.
If you refer to section 4.2 in the paper that leogao linked, those authors also use “corrupted input detection” to benchmark their method. You’re also welcome to try it on your own images—to run the code you just have to install the pip dependencies and then use paths to your own files. (If you uncomment the block at the bottom, you can run it off your webcam in real-time!)