Shannon’s definition of entropy corresponds very closely to the definition of entropy used in statistical mechanics. It’s slightly more general and devoid of “physics baggage” (macro states and so on).
Analogy: Ising model of spin glasses vs undirected graphical models (Markov random fields). The former has a lot of baggage like “magnetization, external field, energy.” The latter is just a statistical model of conditional independence on a graph. The Ising model is a special case (in fact the first developed case, back in 1910) of a Markov random field.
Shannon’s definition of entropy corresponds very closely to the definition of entropy used in statistical mechanics. It’s slightly more general and devoid of “physics baggage” (macro states and so on).
Analogy: Ising model of spin glasses vs undirected graphical models (Markov random fields). The former has a lot of baggage like “magnetization, external field, energy.” The latter is just a statistical model of conditional independence on a graph. The Ising model is a special case (in fact the first developed case, back in 1910) of a Markov random field.
Physicists have a really good nose for models.