D𝜋

Karma: 178

D𝜋 18 Jan 2022 20:23 UTC
3 points
in reply to: Kenoubi’s comment on: D𝜋′s Spiking Network
Welcome onboard this IT ship to baldly go where no one as gone before !
Indeed, I just wrote ‘when it spikes’ and further as the ‘low threshold’ and no more. I work in complete isolation and some things are so obvious inside my brain that I do not consider them as non obvious to others.
It is part of the ‘when’ aspect of learning, but uses an internal state of the neuron instead of an external information from the quantilisers.
If there is little reaction to a sample in a neuron (spiking does happen slowly, or not), it is meaningless and you should ignore it. If it comes too fast, it is already ‘in’ the system and there is no point in adding to it. You are right to say the first rule is more important than the second.
Originally, there was only one threshold instead of 3. When learning, the update would only take place if the threshold was reached after a minimum of two cycles (or 3, but then it converges unbearably slowly), and only for the connections that had been active at least twice. I ‘compacted’ it for use within one cycle (to make it look simpler), so it was 50% of the threshold minimum, and then adjusted (might as well) that value by scanning around and, then, added the upper threshold, but more to limit the number of updates than to improve the accuracy (although it contributes a small bit). The best result is with 30% and 120%, whatever the size or the other parameters.
Before I write this, I quickly checked on PI-F-MNIST. It is still ongoing, but it seems to hold true even on that dataset (BTW: use quantUpP = 3.4 and quantUpN = 40.8 to get to 90.2% with 792 neurons and 90.5% with 7920).
As it seems you are interested, feel free to contact me through private message. There is plenty more in my bag than can fit in a post or comment. I can provide you some more complete code (this one is triple distilled).
Thank you very much for your interest.

D𝜋 16 Jan 2022 11:19 UTC
6 points
in reply to: NicholasKross’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
I am going to answer this comment because it is the first to address the analysis section. Thank you.
I close the paragraph saying that there is no functions anywhere and it will aggrieve some. The shift I am trying to suggest is for those who want to analyse the system using mathematics, and could be dismayed by the absence of functions to work with.
Distributions can be a place to start. The quantilisers are a place to restart mathematical analysis. I gave some links to an existing field of mathematical research that is working along those lines.
Check this out: they are looking for a multi-dimensional extension to the concept. Here it is, I suggest.

D𝜋 9 Jan 2022 15:11 UTC
−1 points
in reply to: philip_b’s comment on: SONN : What’s Next ?
This introduces a new paradigm. Read T.Kuhn. You cannot compare different paradigms.
Everything that matters is in the post. Read it; really.
What is needed next is engineering, ingenuity and suitable ICs, not maths. The IT revolution came from IT (coders) and ICs, not CS.
As for your recommendation, I have tried so many things over the past four years… I posted here first to get to the source of one of the evidences; to no avail.
Good bye everyone
I am available through private messages

SONN : What’s Next ?

D𝜋9 Jan 2022 8:15 UTC

−17 points

3 comments1 min readLW link

D𝜋 7 Jan 2022 15:43 UTC
3 points
in reply to: philip_b’s comment on: D𝜋′s Spiking Network
BP is Back-Propagation.
We are completely missing the plot here.
I had to use a dataset for my explorations and MNIST was simple; and I used PI-MNIST to show an ‘impressive’ result so that people have to look at it. I expected the ‘PI’ to be understood, and it is not. Note that I could readily answer the ‘F-MNIST challenge’.
If I had just expressed an opinion on how to go about AI, the way I did in the roadmap, it would have been just, rightly, ignored. The point was to show it is not ‘ridiculous’ and the system fits with that roadmap.
I see that your last post is about complexity science. This is an example of it. The domain of application is nature. Nature is complex, and maths have difficulties with complexity. The field of chaos theory puttered in the 80s for that reason. If you want to know more about it, start with Turing morphogenesis (read the conclusion), then Prigogine. In NN, there is Kohonen.
Some things are theoretical correct, but practically useless. You know how to win the lotto, but nobody does it. Better something simple that works and can be reasoned about, even without a mathematical theory. AI is not quantum physics.
Maybe it could be said that intelligence is to cut through all the details to, then, reason using what is left, but the devil is in those details.

D𝜋 7 Jan 2022 7:28 UTC
3 points
in reply to: D𝜋’s comment on: D𝜋′s Spiking Network
Also,
No regularisation. I wrote about that in the analysis.
Without max-norm (or maxout, ladder, VAT: all forms of regularisation), BP/SGD only achieves 98.75% (from the dropout −2014- paper).
Regularisation must come from outside the system. - SO can be seen that way—or through local interactions (neighbors). Many papers clearly suggest that should improve the result.
That is yet to do.

D𝜋 6 Jan 2022 13:58 UTC
2 points
in reply to: D𝜋’s comment on: D𝜋′s Spiking Network
… and it is in this description:
“The spiking network can adjust the weights of the active connections”

D𝜋 6 Jan 2022 13:54 UTC
3 points
in reply to: NicholasKross’s comment on: D𝜋′s Spiking Network
It is not a toolbox you will be using tomorrow.
I applied it to F-MNIST, in a couple of hours after being challenged, to show that is not just only MNIST. I will not do it again, that is not the point.
It is a completely different approach to AGI, that sounds so ridiculous that I had to demonstrate that it is not, by getting near SOTA on one widely used dataset (so PI-MNIST) and finding relevant mathematical evidence.

D𝜋 6 Jan 2022 13:42 UTC
3 points
in reply to: ZankerH’s comment on: D𝜋′s Spiking Network
I am going after pure BP/SGD, so neural networks (no SVM), no convolution,...
No pre-processing either. That is changing the dataset.
It is just a POC, to make a point: you do not need mathematics for AGI. Our brain does not.
I will publish a follow-up post soon.

D𝜋 6 Jan 2022 8:09 UTC
3 points
in reply to: philip_b’s comment on: D𝜋′s Spiking Network
I doubt that this would be the best a MLP can achieve on F-MNIST.
I will put it this way: SONNs and MLPs do the same thing, in a different way. Therefore they should achieve the same accuracy. If this SONN can get near 90%, so should MLPs.
It is likely that nobody has bothered to try ‘without convolutions’ because it is so old-fashioned.
Convolutions are for repeated locally aggregated correlations.

D𝜋 5 Jan 2022 12:20 UTC
4 points
in reply to: Bucky’s comment on: D𝜋′s Spiking Network
Spot on.
I hope your explanation will be better understood than mine. Thank you.
It ‘so happens’ that MNIST (but not PI) can also be used for basic geometry. That is why I selected it for my exploration (easy switch between the two modes).

D𝜋 5 Jan 2022 10:50 UTC
9 points
in reply to: Maxime Riché’s comment on: D𝜋′s Spiking Network
no convolution.
You are comparing pears and apples.
I have shared the base because it has real scientific (and philosophical) value.
Geometry and other are separate, and of lesser scientific value. they are more technology.

D𝜋 5 Jan 2022 9:51 UTC
3 points
in reply to: jessicata’s comment on: D𝜋′s Spiking Network
That is correct.
I am referring to that paper as a vindication of the concept, but I do not use the quantiliser algorithm provided.
The one I use I devised on my own, a long time ago, with the thought experiment described, but it has since been mathematically studied. Actually, when I searched for it and found it the first time, it was in a much simpler version, but I cannot find that one again now…
I have not been down to every detail of lsusr’s rewrite yet, just the main corrections to the description of the mechanism. I had to do F-MNIST first.
Side note: The thought experiment describes a mechanical system. Why should it be called algorithm when implemented in code ? because that makes it un-patentable ? I am not sure a super-intelligent AI could understand human politics.

D𝜋 5 Jan 2022 9:32 UTC
3 points
in reply to: MadHatter’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
Please do, and thank you for trying.
That is exactly what I am trying to elicit.
If you have any question, I am available to help (through private messages).
I do not know Python (I am extremely comfortable with C and I get full speed and I do not have the time or need), but it seems the ML community is.

D𝜋 5 Jan 2022 9:19 UTC
23 points
in reply to: lsusr’s comment on: D𝜋′s Spiking Network
Update: 3 runs (2 random) , 10 million steps. All three over 88.33 (average 9.5-10.5 million on the 3: 88.43). New SOTA ? Please check and update.
Update 2: 89.85 at step 50 Million with QuantUpP = 3.2 and quantUpN = 39. It does perform very well. I will leave it at that. As said in my post, those are the two important parameters (no, it is not a universal super-intelligence in 600 lines of code). Be rational, and think about what the fact that this mechanism works so well means (I am talking to everybody, there).
I looked at it, the informed way.
It gets over 88% with very limited effort.
As I pointed, the two dataset are similar in technical description, but they are ‘reversed’ in the data.
MNIST is black dots on white background. F-MNIST is white dots on black background. The histograms are very different.
I tried to make it work despite that, just with parameter changes, and it does.
Here are the changes to the code:
on line 555: quantUpP = 1.9 ;
on line 556: quantUpN = 24.7 ;
with rand(1000), as it is in the code, you already clear 86% at step 300,000 and 87% at step 600,000 and 88% at 3 Million.
I had made another, small and irrelevant, change, in my full tests, so I am running the full tests again without it (the value/steps above are from that new series). It seems to be better again without it… maybe a new SOTA (update: touched 88.33% at step 4,800,000 ! … and 88.5 at 6.8 Millions !. MLPs perform poorly when applied to data even slightly more complicated than MNIST)
I do not understand what is all the hype around MNIST. Once again, this is PI-MNIST and that makes it very different (to put it simply: no geometry, so no convolution).
I would like anybody to give me a reference to some ‘other method that worked on MNIST but did not make it further’, that uses PI-MNIST and gets more than 98.4% on it.
And if anybody tries it on yet another dataset, could they please notify me so I look at it, before they make potentially damaging statements.
What links here?
- D𝜋′s Spiking Network by lsusr (4 Jan 2022 4:08 UTC; 50 points)

D𝜋 4 Jan 2022 17:06 UTC
2 points
in reply to: p.b.’s comment on: D𝜋′s Spiking Network
See my comment on reversing the shades on F-MNIST. I will check it later but I see it gets up to 48% in the ‘wrong’ order and that is surprisingly good. I worked on CIFAR, but that is another story. As-is it gives bad results and you have to add other ‘things’.
As you guessed, I belong to neuroinspired branch and most of my ‘giants’ belong there. I strongly expected, when I started my investigations, to use some of the works that I knew and appreciated along the lines your are mentioning, and I investigated some of them early on.
To my surprise, I did not need them to get to this result, so they are absent.
The two neuronal layers form of the neocortex is where they will be useful. This is only one layer.
Another (bad) reason, is that they add to the GPU hell… that has limited my investigations. It is identified source of potential improvements.

D𝜋 4 Jan 2022 16:09 UTC
3 points
in reply to: lsusr’s comment on: D𝜋′s Spiking Network
That actually brings us to the core of it.
The way I phrased that was, deliberately, ambiguous.
Since 1958, the question the field has been trying to answer is how to transfer the information we get when a sample is presented, to the weights, so next time it will perform better.
BP computes the difference between what would be expected and what is measured, and the propagates it to all intermediary weights according to a set of mathematically derived rules (the generalised delta rule). A lot of work as gone into figuring out the best way to do that. This is what I called ‘how’ to learn.
In this system, the method used is just the simplest possible, and the most intuitive, one: INC and DEC of the weights depending on wether it is or not the correct answer.
The quantiliser, then, tell the system to only apply that simple rule under certain conditions (the Δ⊤and $Δ ⊥_{i}$ limits). That is ‘when’.
You can use the delta rule instead of our basic update rule if you want (I tried). The result is not better and it is less stable, so you have to use small gradients. The problem, as I see it, is that the conditions under witch Jessica Taylor’s theorem is valid are not met any more and you have to ‘fix’ that. I did not investigate that extensively.

D𝜋 4 Jan 2022 15:40 UTC
2 points
in reply to: Measure’s comment on: D𝜋′s Spiking Network
‘those seen before’ are.values of Δ⊥i across all samples seen before, not within a sample.

D𝜋 4 Jan 2022 15:31 UTC
3 points
in reply to: tailcalled’s comment on: D𝜋′s Spiking Network
It is not, per se, Hebb’s rule. Hebb’s rule is very general. I personally see this as belonging to it, that’s all. I give attributions where is think it is deserved.

D𝜋 4 Jan 2022 15:26 UTC
13 points
on: D𝜋′s Spiking Network
I just discovered about the ‘ping back’ on LessWrong...
I gave a first read of your description. Most of it is correct. I will check in more details.
I used the terms ‘total’ and ‘groups’ to make things simpler, but yours are better.
four corrections:
1.
The potential of a neuron can be negative. It is the pure sum of all weights, positive and negative. There is no ‘negative spiking’ (It is one of the huge number of things I tried that did not bring any benefit). It think I remember trying to set a bottom limit at 0 (no negative potential) and that, as always, it did not make any real difference...
2.
‘Our system thus has four receptors per pixel. Exactly one receptor activates per pixel per image’ is incorrect.
The MNIST pixels are grey shades 0-255.
It is reduced down to 4: 0, 1-63, 64-127, 128-191, >=192. (only keep the 2 top bits). That is enough for MNIST. Many papers have noted that the depth can be reduced and it is true.
Images are presented over 4 ‘cycles’, filtered by those 4 limits. In the first cycle, only pixels with a value over 191 are presented, in the second one, those over 127, the third 63, the last, 0 (not nul).
In the code the array ‘wd’ contains the 4 limits, and at each cycle, the pixel values are tested to be superior or equal to those limits.
Connexions are established with matrix pixels. Over the 4 cycles, they are presented with 4 successive 0 or 1.
If a connexion is on a pixel that shows value 112, on the first cycle it will not be active (<192), on the second it will not be active (<128), but on the third and forth, it will be (>63 and >0).
That is what allows the ‘model averaging across cycles’.
From there, you can understand a first, fatal, reason why F-MNIST cannot be processed as-is: the grey shades are reversed. In MNIST, the background is white, in F-MNIST, it is black. So the cycle limits would have to be reversed.
I will have a look at it.
3.
The $Δ ⊥_{i}$ computation includes the $⊤$ column.
Note that the ‘highest’ ⊥ selection can be easily implemented using population coding with inhibition of the ⊤ column.
I do not know if the options I used are only valid for this dataset or if there have larger validity across dataset as I only have used that one. Maybe they are and you won’t have to figure out each time.
4.
When a new connection is established, the initial weight is always the same. It is given as a fraction of the threshold in the variable ‘divNew’, that is the divider. You can do random, it does not make a difference. You can change it to another value. But it has to be small enough (the divider) that the number of connections of a neuron multiplied by the number of cycles be superior to the threshold, or the system will never ‘boot’ as no neuron would ever spike. So I use ¹⁄₁₀ of the threshold with 10 connections and 4 cycle, and it is fine.