I just discovered about the ‘ping back’ on LessWrong...
I gave a first read of your description. Most of it is correct. I will check in more details.
I used the terms ‘total’ and ‘groups’ to make things simpler, but yours are better.
four corrections:
1.
The potential of a neuron can be negative. It is the pure sum of all weights, positive and negative. There is no ‘negative spiking’ (It is one of the huge number of things I tried that did not bring any benefit). It think I remember trying to set a bottom limit at 0 (no negative potential) and that, as always, it did not make any real difference...
2.
‘Our system thus has four receptors per pixel. Exactly one receptor activates per pixel per image’ is incorrect.
The MNIST pixels are grey shades 0-255.
It is reduced down to 4: 0, 1-63, 64-127, 128-191, >=192. (only keep the 2 top bits). That is enough for MNIST. Many papers have noted that the depth can be reduced and it is true.
Images are presented over 4 ‘cycles’, filtered by those 4 limits. In the first cycle, only pixels with a value over 191 are presented, in the second one, those over 127, the third 63, the last, 0 (not nul).
In the code the array ‘wd’ contains the 4 limits, and at each cycle, the pixel values are tested to be superior or equal to those limits.
Connexions are established with matrix pixels. Over the 4 cycles, they are presented with 4 successive 0 or 1.
If a connexion is on a pixel that shows value 112, on the first cycle it will not be active (<192), on the second it will not be active (<128), but on the third and forth, it will be (>63 and >0).
That is what allows the ‘model averaging across cycles’.
From there, you can understand a first, fatal, reason why F-MNIST cannot be processed as-is: the grey shades are reversed. In MNIST, the background is white, in F-MNIST, it is black. So the cycle limits would have to be reversed.
I will have a look at it.
3.
The Δ⊥i computation includes the ⊤ column.
Note that the ‘highest’ ⊥ selection can be easily implemented using population coding with inhibition of the ⊤ column.
I do not know if the options I used are only valid for this dataset or if there have larger validity across dataset as I only have used that one. Maybe they are and you won’t have to figure out each time.
4.
When a new connection is established, the initial weight is always the same. It is given as a fraction of the threshold in the variable ‘divNew’, that is the divider. You can do random, it does not make a difference. You can change it to another value. But it has to be small enough (the divider) that the number of connections of a neuron multiplied by the number of cycles be superior to the threshold, or the system will never ‘boot’ as no neuron would ever spike. So I use 1⁄10 of the threshold with 10 connections and 4 cycle, and it is fine.
I just discovered about the ‘ping back’ on LessWrong...
I gave a first read of your description. Most of it is correct. I will check in more details.
I used the terms ‘total’ and ‘groups’ to make things simpler, but yours are better.
four corrections:
1.
The potential of a neuron can be negative. It is the pure sum of all weights, positive and negative. There is no ‘negative spiking’ (It is one of the huge number of things I tried that did not bring any benefit). It think I remember trying to set a bottom limit at 0 (no negative potential) and that, as always, it did not make any real difference...
2.
‘Our system thus has four receptors per pixel. Exactly one receptor activates per pixel per image’ is incorrect.
The MNIST pixels are grey shades 0-255.
It is reduced down to 4: 0, 1-63, 64-127, 128-191, >=192. (only keep the 2 top bits). That is enough for MNIST. Many papers have noted that the depth can be reduced and it is true.
Images are presented over 4 ‘cycles’, filtered by those 4 limits. In the first cycle, only pixels with a value over 191 are presented, in the second one, those over 127, the third 63, the last, 0 (not nul).
In the code the array ‘wd’ contains the 4 limits, and at each cycle, the pixel values are tested to be superior or equal to those limits.
Connexions are established with matrix pixels. Over the 4 cycles, they are presented with 4 successive 0 or 1.
If a connexion is on a pixel that shows value 112, on the first cycle it will not be active (<192), on the second it will not be active (<128), but on the third and forth, it will be (>63 and >0).
That is what allows the ‘model averaging across cycles’.
From there, you can understand a first, fatal, reason why F-MNIST cannot be processed as-is: the grey shades are reversed. In MNIST, the background is white, in F-MNIST, it is black. So the cycle limits would have to be reversed.
I will have a look at it.
3.
The Δ⊥i computation includes the ⊤ column.
Note that the ‘highest’ ⊥ selection can be easily implemented using population coding with inhibition of the ⊤ column.
I do not know if the options I used are only valid for this dataset or if there have larger validity across dataset as I only have used that one. Maybe they are and you won’t have to figure out each time.
4.
When a new connection is established, the initial weight is always the same. It is given as a fraction of the threshold in the variable ‘divNew’, that is the divider. You can do random, it does not make a difference. You can change it to another value. But it has to be small enough (the divider) that the number of connections of a neuron multiplied by the number of cycles be superior to the threshold, or the system will never ‘boot’ as no neuron would ever spike. So I use 1⁄10 of the threshold with 10 connections and 4 cycle, and it is fine.