Beliefs at different timescales

Why is a chess game the op­po­site of an ideal gas? On short timescales an ideal gas is de­scribed by elas­tic col­li­sions. And a sin­gle move in chess can be mod­eled by a policy net­work.

The differ­ence is in long timescales: If we simu­lated elas­tic col­li­sions for a long time, we’d end up with a com­pli­cated dis­tri­bu­tion over the microstates of the gas. But we can’t run simu­la­tions for a long time, so we have to make do with the Boltz­mann dis­tri­bu­tion, which is a lot less ac­cu­rate.

Similarly, if we rol­led out our policy net­work to get a dis­tri­bu­tion over chess game out­comes (win/​loss/​draw), we’d get the dis­tri­bu­tion of out­comes un­der self-play. But if we’re ob­serv­ing a game be­tween two play­ers who are bet­ter play­ers than us, we have ac­cess to a more ac­cu­rate model based on their Elo rat­ings.

Can we for­mal­ize this? Sup­pose we’re ob­serv­ing a chess game. Our be­liefs about the next move are con­di­tional prob­a­bil­ities of the form , and our be­liefs about the next moves are con­di­tional prob­a­bil­ities of the form . We can trans­form be­liefs of one type into the other us­ing the operators

If we’re log­i­cally om­ni­scient, we’ll have and . But in gen­eral we will not. A chess game is short enough that is easy to com­pute, but is too hard be­cause it has ex­po­nen­tially many terms. So we can have a long-term model that is more ac­cu­rate than the rol­lout , and a short-term model that is less ac­cu­rate than . This is a sign that we’re deal­ing with an in­tel­li­gence: We can pre­dict out­comes bet­ter than ac­tions.

If in­stead of a chess game we’re pre­dict­ing an ideal gas, the rele­vant timescales are so long that we can’t com­pute or . Our long-term ther­mo­dy­namic model is less ac­cu­rate than a simu­la­tion . This is of­ten a fea­ture of re­duc­tion­ism: Com­pli­cated things can be re­duced to sim­ple things that can be mod­eled more ac­cu­rately, al­though more slowly.

In gen­eral, we can have sev­eral mod­els at differ­ent timescales, and and op­er­a­tors con­nect­ing all the lev­els. For ex­am­ple, we might have a short-term model de­scribing the physics of fund­men­tal par­ti­cles; a medium-term model de­scribing a per­son’s mo­tor ac­tions; and a long-term model de­scribing what that per­son ac­com­plishes over the course of a year. The medium-term model will be less ac­cu­rate than a rol­lout of the short-term model, and the long-term model may be more ac­cu­rate than a rol­lout of the medium-term model if the per­son is smarter than us.