Estimated MSE loss for three diffrent ways of embedding features into neuons, when there are more possible features than neurons.
I’ve typed up some math notes for how much MSE loss we should expect for random embedings, and some other alternative embedings, for when you have more features than neurons. I don’t have a good sense for how ledgeble this is to anyone but me.
Note that neither of these embedings are optimal. I belive that the optimal embeding for minimising MSE loss is to store the features in almost orthogonal directions, which is similar to random embedings but can be optimised more. But I also belive that MSE loss don’t prefeer this solution very much, which means that when there are other tradeofs, MSE loss might not be enough to insentivise superposstion.
This does not mean we should not expect superpossition in real network.
Many networks uses other loss functions, e.g. cross-entropy.
Even if the loss is MSE on the final output, this does not mean MSE is the right loss for modeling the dynamics in the middle of the network.
Setup and notation
T features
D neurons
z active featrues
Assuming:
z≪D<T
True feature values:
y = 1 for active featrus
y = 0 for inactive features
Using random embedding directions (superpossition)
Estimated values:
^y=a + ϵ where E[ϵ2]=(z−1)a2/D for active features
We emebd a single feature in each neuron, and the rest of the features, are just not represented.
Estimated values:
^y=y for represented features
^y=0 for non represented features
Total Mean Squared Error (MSE)
MSEsingle=zT−DD
One neuron per feature
We embed each feature in a single neuron.
^y=a∑y where the sum is over all feature that shares the same neuron
We assume that the probability of co-activated features on the same neuron is small enough to ignore. We also assume that every neuron is used at least once. Then for any active neuron, the expected number of inactive neurons that will be wrongfully activated, are T−DD, giving us the MSE loss for this case as
MSEmulti=z((1−a)2+(TD−1)a2)
We can already see that this is smaller than MSErand, but let’s also calculate what the minimum value is.MSErand is minimised by
Estimated MSE loss for three diffrent ways of embedding features into neuons, when there are more possible features than neurons.
I’ve typed up some math notes for how much MSE loss we should expect for random embedings, and some other alternative embedings, for when you have more features than neurons. I don’t have a good sense for how ledgeble this is to anyone but me.
Note that neither of these embedings are optimal. I belive that the optimal embeding for minimising MSE loss is to store the features in almost orthogonal directions, which is similar to random embedings but can be optimised more. But I also belive that MSE loss don’t prefeer this solution very much, which means that when there are other tradeofs, MSE loss might not be enough to insentivise superposstion.
This does not mean we should not expect superpossition in real network.
Many networks uses other loss functions, e.g. cross-entropy.
Even if the loss is MSE on the final output, this does not mean MSE is the right loss for modeling the dynamics in the middle of the network.
Setup and notation
T features
D neurons
z active featrues
Assuming:
z≪D<T
True feature values:
y = 1 for active featrus
y = 0 for inactive features
Using random embedding directions (superpossition)
Estimated values:
^y=a + ϵ where E[ϵ2]=(z−1)a2/D for active features
^y=ϵ where E[ϵ2]=za2/D for active features
Total Mean Squared Error (MSE)
MSErand=z((1−a)2+(z−1)a2D)+(T−z)za2D≈z(1−a)2+zTDa2This is minimised by
a=DT+DMaking MSE
MSErand=zTT+D=z(1−DT+D)One feature per neuron
We emebd a single feature in each neuron, and the rest of the features, are just not represented.
Estimated values:
^y=y for represented features
^y=0 for non represented features
Total Mean Squared Error (MSE)
MSEsingle=zT−DDOne neuron per feature
We embed each feature in a single neuron.
^y=a∑y where the sum is over all feature that shares the same neuron
We assume that the probability of co-activated features on the same neuron is small enough to ignore. We also assume that every neuron is used at least once. Then for any active neuron, the expected number of inactive neurons that will be wrongfully activated, are T−DD, giving us the MSE loss for this case as
MSEmulti=z((1−a)2+(TD−1)a2)We can already see that this is smaller than MSErand, but let’s also calculate what the minimum value is.MSErand is minimised by
a=DTMaking MSE
MSErand=z(1−DT)