Happy new year.
I have just posted on LessWrong the result of my work on AI.
Your work on quantilisers is the core mathematical evidence of what I propose (the code is another).
I would really appreciate your opinion on it.
Kind regards
Dđ
Karma: 178
Self-OrÂganised NeuÂral NetÂworks: A simÂple, natÂuÂral and effiÂcient way to intelligence
It is PI-MNIST.
Permutation Invariant. To keep it simple, you cannot use convolutions. It is all explained in the text.
Real SOTA on that version is 99.04% (Maxout), but that is with 65 Millions+ parameters. I do not have the hardware (or time).
I stopped at 98.9% with 750,000 connections (integers and additions) and this is close to what BP/âSGD (table 2) gets with 3 hidden layer of 1024 units each, for a total of >3,000,000 parameters (floating-points and multiplication) with max-norm and Relu.
For a similar accuracy, the number of âparametersâ is almost an order of magnitude lower with this system and efficiency even more.
Remember, it is not supposed to work at all, and it is not optimised.
We never stop learning.
To kill the program, shoot âCtrl+Câ.
Seriously, this system works âonlineâ. I gave the exemple of the kids and the Dutch to illustrate that in nature, things change around us and we adjust what we know to the new conditions. A learning process should not have a stopping criteria.
The system converges, on PI-MNIST, at 3-5 million steps. To compare, recent research papers stop at 1 million, but keep in mind that we only update about 2 out of 10 groups each time, so it is equivalent.
So you can use âfor( ; b<5000000 ; b++ )â instead of âwhile( 1 == 1 )â in the batch() function.
After convergence, it stays within a 0.1% margin forever after. You can design a stop test around that if you want, or around the fact that weights stabilise, or anything of that kind.
If you were to use a specific selection of the dataset, wait until it stabilises and, then, use the whole set, the system would âstart learning againâ and adjust to that change. Forever.
It is a feature, not a bug.
I am not sure I understand your question (sorry, I do not know what is YudkowskyâDMs)
I basically disclosed, to all, that the way we all think we think, does work.
What kind of responsibility could that bear ?
I think I have expressed my views on the matter of responsibility quiet clearly in the conclusion.
I just checked Yudkowsky on Google. He founded this website, so good.
Here is not the place to argue my views on super-intelligence, but I clearly side with Russell and Norvig. Life is just too complex; luckily.
As for safety, the title of Jessica Taylorâs article is:
âQuantilizers: A Safer Alternative to Maximizers for Limited Optimizationâ.
I will just be glad to have proved that alternative to be effective.
That is a question of low philosophical value, but of the highest practical importance.
At line 3,000,000 with the 98.9% setup, in the full log, there is these two informations:
âsp: 1640 734 548â, and ânbupd: 7940.54 6894.20â (and the test accuracy is exactly 98.9%)
It means that the average spike count for the IS-group is 1640 per sample, 734 for the highest ISNOT-group and 548 for the other ones. The number of weight updates per IS-learn is 7940.54 and 6894.20 per ISNOT-learn. With the coding scheme used, the average number of inputs per sample over the four cycles is 2162 (total number of activated pixels in the input matrix) for 784 pixels. There is 7920 neurons per group with 10 connections each (so 10â784 th of the pixel matrix), for a total of 79200 neurons.
From those numbers:
The average number of integer additions done for all neurons in a group when a sample is presented is: 79200 * 2162 * 10â784 = 2,184,061 integer additions in total.
And for the spike counts: 1640 + 734 + 8*548 = 6758 INCrements (counting the spikes).
When learning, for each IS-learn, there is 7940, and for each ISNOT-learn, 6894, weight updates . Those are additions of integers. So an average of (7940 + 9*6894) /â 10 = 7000 additions of integer.
That is to be compared with a 3 fully connected, say 800 units (to make up for the difference between 98.90% and 98.94%) layers.
That would be at least 800*800*2 + 800*784 = 1,907,200 floating point multiplications, plus what would be used for Max-norm, ReLu,⊠that I am not qualified to evaluate, but might roughly double that ?
And the same for each update (low estimate).
Even with recent works on sparse updates that do reduce that by 93%, it is still more than 133,000 floating-point multiplications (against 7000 additions of integers).
I have managed to get over 98.5% with 2000 neurons (20,000 connections). I would like to know if BP/âSGD can perform to that level with such a small number of parameters (that would be one fully connected layer of 25 units) ? And, as I said in the roadmap, that is what will matter for full real systems.
That is the basic building bloc. the 1x1x1 Lego brick. 1.5/â1.1 = 36% improvement with 40 times the ressources is useless in practice.
And that is missing the real point laid out in the Roadmap: this system CAN and MUST be implemented in analog (hybrid until we get practical memristors), whereas BP/âSGD CAN NOT.
There is, at least, another order of magnitude in efficiency to be gained there.
There is a lot of effort invested, at this time, in the industry to implement AI at IC-level. Now is the time.
I have, deliberately, taken away everything relating to geometry from this presentation.
It took 12 years (1986-1998) and (how much research effort ?) ,to go from BP/âSGD to convolutions.
This is a one man effort, on my own personal time (20,000 hours over the past 6 years), that I am giving away for the community to freely take over. I am really sorry if it is not enough. Their choice.
It is not an add-on to something that exist but a complete restart. One thing at a time.
As for CUDA, if you have a lot of threads, it is bearable, and you can use old, cheap, GPUs with very little loss (they have been optimised recently for FP multiply/âadd, at the expense of ADD of INT).
FYI, I got >99.3% with only adding a fix layer of simple preset filters (no augmentation) and the idea behind can be readily extended. And you can also train, unsupervised, convolutions.
The only function that is implemented in CUDA is the test one (test_gpu).
It is also implement for CPU (test_mt_one), identically.
What matters is all clearly (I hope) explained in the text. It is simple enough that its reach is not limited to ML researchers and clearly within that of a lot of coders. The IT revolution started when amateurs got PCs.
In this version of the code, I had to make a tradeoff between completeness, usability and practicality. Write your own code, it does not matter. It is the concept that does.
The (upcoming) website will give separate, readable, versions. I am waiting to get a proper idea of what is demanded before I do that, so thank you for that input.
No, there isnât, but it is interesting.
I gave it a quick look. It seems to be closer to this (this is closer to the point)
I was heavily influenced, back in the 70s, by the works of Mandelbrot and the chaos theory that developed at the time, and has gone nowhere.
The concept of self-organisation has been around for a long time but it is hard to study from the mathematical point of view, and, probably for that reason, it has never âpicked upâ.
So, of course, there are similarities, and, please, go back to all of those old papers and re-think it all.
You will benefit from an hands-on approach rather then a theoretical one. First you experiment, then you find, then you analyse and finally, you theorise. This is not quantum physics and we have the tools (computers) to easily conduct experiments.
This is just another exemple, one that could prove very useful. Thatâs it.
They belong to the same, forgotten, family.
T.Kuhn said it all.
Update:
I changed the adjustment values for the 98.65% version to 500/â-500 (following lsusr comments).
1000/â-1000 is good for the larger network (98.9%), but too much for the smaller ones. It makes the accuracy reduce after it has reached the peak value.
I was too fast publishing and did not go through all the required verifications. My fault.
I am running a series of tests to confirm. The first two are in spec and stable at 10 million steps.
Larger values speed up the convergence, and I was trying to make it as fast as possible, to not waste the time of those who would spend it verifying. Sorry about that.
I wrote a comment on that but this is a better to place for it.
I changed the update value from 1000 to 500 for that network size (in the code).
1000 is for the large network (98.9%). At size 792 (for 98.6%) it is too much, and the accuracy goes down after reaching the top. I did not take the time to check properly before publishing. My fault.
If you check it out now, it will get to >98.6% and stay there (tested up to 10 millions, three times, random).
You are comparing step and ladder (I had to seize on it !).
If you look at Table 2 in your last reference, you will see that they, carefully, show results improving has steps are added. Ladder is just another step (an optimisation one). There is a reason why researchers use PI-MNIST: it is to reduce the size of the ladder to make comparisons clearer.
What I am trying to bring here is a new first step.
I could have tried a 784-25-10 BP/âSGD network (784*25 = 19600 parameters) to compare with this system with 196 neurons and 10 connections. I have managed to get 98% with that. How much for the same with BP/âSGD ?
The current paradigm has been building up since 1986, and was itself based on the perceptron from 1958.
Here, I take the simplest form of the perceptron (single layer), only adjoin a, very basic, quantiliser to drive it, and already get near SOTA. I also point out that this quantiliser is just another form of neuron.
I am trying to show it might be an interesting step to take.
See my answer to mlem_mlem_mlem for the second part of your comment.
You are bringing another interesting point: scaling up and tuning.
As I indicated in the roadmap, nature has chosen the way of width to that of depth.
The cortical sheet is described as a 6 layers structure, but only 3 are neurons and 2 pyramidal neurons. That is not deep. Then we see columns, functional âzonesâ, âregionsâ⊠There is an organisation, but it is not very deep. The number of columns in each âzoneâ is very large. Also note that the neuron is deemed âstochasticâ, so precision is not possible. Lastly, note (sad but true) that those who got the prize worked on technical simplification for practical use.
There is two options at this stage:
We consider, as the symbolic school has since 1969, that the underlying substrate is unimportant and, if we can find mathematical ways to describe it, we will be able to reproduce it, or...
We consider that nature has done the work (we are here to attest to that), properly, and we should look at how it did it.
1986 was an acceptable compromise, for a time. 2026, will mark one century of the 5th Solvay conference.
Thank you for the congrats, it helps.
Note, that I only claim to reach SOTA, not to beat it.
It would be preposterous to convince anybody with this limited evidence. The goal is to raise interest so some will spend some time to look deeper into it. Most will not, of course, for many reasons, and yours is a valid one.
The advantage of this one is its simplicity. At this point any coder can take it up and build on it. This has to be turned into a new type of construction set. I would like this to provide the 15 years old of today the pleasure my first computer (machine language) gave me, and Legos before that.
You got the last bit correctly. That is what self-organisation provides: ad-hoc selection.
I just discovered about the âping backâ on LessWrong...
I gave a first read of your description. Most of it is correct. I will check in more details.
I used the terms âtotalâ and âgroupsâ to make things simpler, but yours are better.
four corrections:
1.
The potential of a neuron can be negative. It is the pure sum of all weights, positive and negative. There is no ânegative spikingâ (It is one of the huge number of things I tried that did not bring any benefit). It think I remember trying to set a bottom limit at 0 (no negative potential) and that, as always, it did not make any real difference...
2.
âOur system thus has four receptors per pixel. Exactly one receptor activates per pixel per imageâ is incorrect.
The MNIST pixels are grey shades 0-255.
It is reduced down to 4: 0, 1-63, 64-127, 128-191, >=192. (only keep the 2 top bits). That is enough for MNIST. Many papers have noted that the depth can be reduced and it is true.
Images are presented over 4 âcyclesâ, filtered by those 4 limits. In the first cycle, only pixels with a value over 191 are presented, in the second one, those over 127, the third 63, the last, 0 (not nul).
In the code the array âwdâ contains the 4 limits, and at each cycle, the pixel values are tested to be superior or equal to those limits.
Connexions are established with matrix pixels. Over the 4 cycles, they are presented with 4 successive 0 or 1.
If a connexion is on a pixel that shows value 112, on the first cycle it will not be active (<192), on the second it will not be active (<128), but on the third and forth, it will be (>63 and >0).
That is what allows the âmodel averaging across cyclesâ.
From there, you can understand a first, fatal, reason why F-MNIST cannot be processed as-is: the grey shades are reversed. In MNIST, the background is white, in F-MNIST, it is black. So the cycle limits would have to be reversed.
I will have a look at it.
3.
The computation includes the column.
Note that the âhighestâ â„ selection can be easily implemented using population coding with inhibition of the †column.
I do not know if the options I used are only valid for this dataset or if there have larger validity across dataset as I only have used that one. Maybe they are and you wonât have to figure out each time.
4.
When a new connection is established, the initial weight is always the same. It is given as a fraction of the threshold in the variable âdivNewâ, that is the divider. You can do random, it does not make a difference. You can change it to another value. But it has to be small enough (the divider) that the number of connections of a neuron multiplied by the number of cycles be superior to the threshold, or the system will never âbootâ as no neuron would ever spike. So I use 1â10 of the threshold with 10 connections and 4 cycle, and it is fine.
It is not, per se, Hebbâs rule. Hebbâs rule is very general. I personally see this as belonging to it, thatâs all. I give attributions where is think it is deserved.
âthose seen beforeâ are.values of Îâ„i across all samples seen before, not within a sample.
The link I typed in and appears when hovering over is, indeed, âhttp://ââyann.lecun.com/ââexdb/ââmnist/âââ, and it works on my machine⊠Thanks for the additional link.