because it uses backwards-mode automatic differentiation based on pytorch, which (unlike other major python AD packages) creates its execution graphs on-the-fly.
Because pyro is primarily designed for variational inference, not MCMC. The problem with dynamic structure in MCMC is that it’s inherently discrete: either the model has this structure, or it has that one. That makes it tough to do MCMC with, especially in high dimensions, because these days it’s clear that making high-dimensional MCMC practical requires at least some use of the tricks of HMC, which means differentiating the likelihood with respect to the parameters, which means you need continuous parameters.
However: variational inference doesn’t actually work well with complex dynamic structure, because the variational approximations themselves are usually “smooth” in ways that a correct posterior of a dynamic model would not be.
In other words: although it would not be hard to create a syntax / notation for expressing dynamic models, the reason there is not yet a standard syntax/notation for that is that the methods for fitting such models, or, to the same purpose, sampling from their posterior… are still mostly lacking.
It seems to me this is a fundamentally hard problem overall, so I don’t expect it will be fully solved even in the next 5-10 years, but I do expect to see significant progress. PPLs are getting better by leaps and bounds now, and I’d expect something like Gen.jl to usefully cover at least some special cases of dynamic models, and to have a clearer syntax for at least talking about those models it still struggles to cover, in even just the 1-2 year time frame.
Pyro can do this for certain technical reasons:
because it uses backwards-mode automatic differentiation based on pytorch, which (unlike other major python AD packages) creates its execution graphs on-the-fly.
Because pyro is primarily designed for variational inference, not MCMC. The problem with dynamic structure in MCMC is that it’s inherently discrete: either the model has this structure, or it has that one. That makes it tough to do MCMC with, especially in high dimensions, because these days it’s clear that making high-dimensional MCMC practical requires at least some use of the tricks of HMC, which means differentiating the likelihood with respect to the parameters, which means you need continuous parameters.
However: variational inference doesn’t actually work well with complex dynamic structure, because the variational approximations themselves are usually “smooth” in ways that a correct posterior of a dynamic model would not be.
In other words: although it would not be hard to create a syntax / notation for expressing dynamic models, the reason there is not yet a standard syntax/notation for that is that the methods for fitting such models, or, to the same purpose, sampling from their posterior… are still mostly lacking.
It seems to me this is a fundamentally hard problem overall, so I don’t expect it will be fully solved even in the next 5-10 years, but I do expect to see significant progress. PPLs are getting better by leaps and bounds now, and I’d expect something like Gen.jl to usefully cover at least some special cases of dynamic models, and to have a clearer syntax for at least talking about those models it still struggles to cover, in even just the 1-2 year time frame.