Presumably, reality can be fully described with a very simple model—the Standard Model of Physics. The number of transistors to implement it is probably a few K (the field equations a smaller to write but depend on math to encode too; turning machine size would also be a measure, but transistors are more concrete). But if you want to simulate reality at that level you need a lot of them for all the RAM and it would be very slow.
So we build models that abstract large parts of physics away—atoms, molecules, macroscopic mechanics. I would include even social process models in this.
But details are lost and you have to know when your models stop giving precise results.
It would be interesting to get some tight bounds on the amount of compute needed to extract computable models from samples of measured phenomena or from more precise lower models. Such bounds would allow to give some complexity limitations for AGI.
Funny! I’ve now been doing ML-adjacent work for long enough that I have internalized the idea that data is part of the model, not just calculations. The separation of reality as “simple physics” plus “lots storage for starting/current quantum configurations” just doesn’t click for me. The data is huge, and that’s all that matters in terms of model size/complexity.
Presumably, reality can be fully described with a very simple model—the Standard Model of Physics. The number of transistors to implement it is probably a few K (the field equations a smaller to write but depend on math to encode too; turning machine size would also be a measure, but transistors are more concrete). But if you want to simulate reality at that level you need a lot of them for all the RAM and it would be very slow.
So we build models that abstract large parts of physics away—atoms, molecules, macroscopic mechanics. I would include even social process models in this.
But details are lost and you have to know when your models stop giving precise results.
It would be interesting to get some tight bounds on the amount of compute needed to extract computable models from samples of measured phenomena or from more precise lower models. Such bounds would allow to give some complexity limitations for AGI.
Funny! I’ve now been doing ML-adjacent work for long enough that I have internalized the idea that data is part of the model, not just calculations. The separation of reality as “simple physics” plus “lots storage for starting/current quantum configurations” just doesn’t click for me. The data is huge, and that’s all that matters in terms of model size/complexity.
This goes into the same direction and may be more to your liking: How Many Bits Of Optimization Can One Bit Of Observation Unlock?
Maybe you can see it as a factoring of a model into sub-models?