With “thin model / fat service” I’m attempting to contrast with the typical end-to-end model architecture, where there is no separate “physics simulator”, and instead the model just learns its own physics model, embedded with all the other relations that it has learned. So under that dichotomy, I think there is no “thin layer in front of the physics simulation” in an end-to-end model, as any part of the physics simulator can connect to or be connected from any other part of the model.
In such an end-to-end model, it’s really hard to figure out where the “physics simulator” (such as it is) even lives, and what kind of ontology it uses. I’m explicitly thinking of to the ELK paper’s Ontology Identification problem here. How do we know the AI is being honest when it says “I predict that object X will be in position Y at time T”? If you can interpret its physics model, you can see what simulations it is running and know if it’s being honest.
It’s possible, if we completely solve interpretability, to extract the end-to-end model’s physics simulator, I suppose. But if we have completely solved interpretability, I think we’re most of the way towards solving alignment. On the other hand, if we use Services (e.g. MuJoCo for Physics) then we already have interpretability on that service, by construction, without having to interpret whatever structures the model has learned to solve physics problems.
You need some way to correct the physics sims predictions to correspond with actual results
Sure, in some sense an end-to-end model needs to normalize inputs, extract features, and so on, before invoking its physics knowledge. But the point I’m getting at is that with Services we know for sure what the model’s representation has been normalized to (since we can just look at the data passed to MuJoCo), whereas this is (currently) opaque in an end-to-end model.
Note that you would probably never use such a “model learns it’s own physics” as the solution in any production AI system. For either the reason you gave, or more pedantically, so long as rival architectures using a human written physics system at the core perform more consistently in testing, you should never pick the less reliable solution.
With “thin model / fat service” I’m attempting to contrast with the typical end-to-end model architecture, where there is no separate “physics simulator”, and instead the model just learns its own physics model, embedded with all the other relations that it has learned. So under that dichotomy, I think there is no “thin layer in front of the physics simulation” in an end-to-end model, as any part of the physics simulator can connect to or be connected from any other part of the model.
In such an end-to-end model, it’s really hard to figure out where the “physics simulator” (such as it is) even lives, and what kind of ontology it uses. I’m explicitly thinking of to the ELK paper’s Ontology Identification problem here. How do we know the AI is being honest when it says “I predict that object X will be in position Y at time T”? If you can interpret its physics model, you can see what simulations it is running and know if it’s being honest.
It’s possible, if we completely solve interpretability, to extract the end-to-end model’s physics simulator, I suppose. But if we have completely solved interpretability, I think we’re most of the way towards solving alignment. On the other hand, if we use Services (e.g. MuJoCo for Physics) then we already have interpretability on that service, by construction, without having to interpret whatever structures the model has learned to solve physics problems.
Sure, in some sense an end-to-end model needs to normalize inputs, extract features, and so on, before invoking its physics knowledge. But the point I’m getting at is that with Services we know for sure what the model’s representation has been normalized to (since we can just look at the data passed to MuJoCo), whereas this is (currently) opaque in an end-to-end model.
Note that you would probably never use such a “model learns it’s own physics” as the solution in any production AI system. For either the reason you gave, or more pedantically, so long as rival architectures using a human written physics system at the core perform more consistently in testing, you should never pick the less reliable solution.