No free lunch theorem is irrelevant

I often see people cite the “no free lunch theorem” as a counterargument to the existence of an AGI. I think this is a very bad argument. This theorem is formulated as a problem of predicting random and uniformly distributed data. Just open the Wikipedia!. In my opinion, this is another example of how people use results from mathematics simply because they sound relevant without understanding their original meaning and limitations. What makes them think it’s relevant to AGI?

It sounds something like “well, since there is such a theorem, it turns out that we will not be able to create an algorithm that can do many tasks.” This is complete nonsense. In order to somehow bind it to real systems, you need to make a reservation and add a restriction on some kind of physical size of the model: the number of parameters, disk space, total equipment weight, training time, total budget whatever. In this case, well, yes. If we have a general algorithm for solving many problems, then for any problem we can probably use the same resources to make an algorithm that performs better. The problem is that we don’t say “how much better”. Any improvement will do.

It will not be the original theorem, but I think this is close to “what people think when they say no free lunch theorem”.

If our algorithm for solving many problems shows itself “kind of meh” compared to specialized algorithms, then what prevents us from increasing the resource budget and collecting all our specialized algorithms into one big one? Yes, the new one can also be reassembled into specialized algorithms, but sooner or later we will hit diminishing returns. It is impossible to calculate arithmetic more accurately than in 100% of cases, it is impossible to predict the structure of a protein more accurately than it will turn out in reality, and so on. By increasing the available resources, distilling and amplifying, we can make a general-purpose algorithm as good as we like in any task. As close to maximum performance for the metrics as we need.