My intuitions about the mathematics of entropy didn’t come from mathematics, statistics, computer science or even cryptography. They came from the statistical mechanics I learned while studying thermodynamics for my physics degree. Only many years later did I apply the ideas to information theory. I see information theory through the lens of thermodynamics.
Do you have suggestions for where to dive into that? That same gap between “Here’s a fuzzy overview” and “Here’s a textbook optimized for demonstrating your ability to regurgitate formalisms” appears in my skimming of that too. I have strong but fuzzy intuitions for how thermodynamics works, and I have a lot of formal skill, but I have basically zero connection between those two.
I wish I could answer your question. When I studied physics, I bucketed textbooks into “good textbooks I like” (such as Griffiths) and boring forgettable textbooks. Alas, thermodynamics belongs to the latter category. I literally don’t remember what textbooks I read.
I’ve been contemplating whether I should just write my own sequence on statistical mechanics. Your post is good evidence that such a sequence might be valuable.
Oh, I would certainly love that. Statistical mechanics looks like it’s magic, and it strikes me as absolutely worth grokking, and yeah I haven’t found any entry point into it other than the Great Formal Slog.
I remember learning about “inner product spaces” as a graduate student, and memorizing structures and theorems about it, but it wasn’t until I had already finished something like a year of grad school that I found out that the intuition behind inner products was “What kind of thing is a dot product in a vector space? What would ‘dot product’ mean in vector spaces other than the Euclidean ones?” Without that guiding intuition, the whole thing becomes a series of steps of “Yep, I agree, that’s true and you’ve proven it. I don’t know why we’re proving that or where we’re going, but okay. One more theorem to memorize.”
I wonder if most “teachers” of formal topics either assume the guiding intuitions are obvious or implicitly think they don’t matter. And maybe for truly gifted researchers they don’t? But at least for people like me, they’re damn close to all that matters.
My intuitions about the mathematics of entropy didn’t come from mathematics, statistics, computer science or even cryptography. They came from the statistical mechanics I learned while studying thermodynamics for my physics degree. Only many years later did I apply the ideas to information theory. I see information theory through the lens of thermodynamics.
Do you have suggestions for where to dive into that? That same gap between “Here’s a fuzzy overview” and “Here’s a textbook optimized for demonstrating your ability to regurgitate formalisms” appears in my skimming of that too. I have strong but fuzzy intuitions for how thermodynamics works, and I have a lot of formal skill, but I have basically zero connection between those two.
I wish I could answer your question. When I studied physics, I bucketed textbooks into “good textbooks I like” (such as Griffiths) and boring forgettable textbooks. Alas, thermodynamics belongs to the latter category. I literally don’t remember what textbooks I read.
I’ve been contemplating whether I should just write my own sequence on statistical mechanics. Your post is good evidence that such a sequence might be valuable.
Oh, I would certainly love that. Statistical mechanics looks like it’s magic, and it strikes me as absolutely worth grokking, and yeah I haven’t found any entry point into it other than the Great Formal Slog.
I remember learning about “inner product spaces” as a graduate student, and memorizing structures and theorems about it, but it wasn’t until I had already finished something like a year of grad school that I found out that the intuition behind inner products was “What kind of thing is a dot product in a vector space? What would ‘dot product’ mean in vector spaces other than the Euclidean ones?” Without that guiding intuition, the whole thing becomes a series of steps of “Yep, I agree, that’s true and you’ve proven it. I don’t know why we’re proving that or where we’re going, but okay. One more theorem to memorize.”
I wonder if most “teachers” of formal topics either assume the guiding intuitions are obvious or implicitly think they don’t matter. And maybe for truly gifted researchers they don’t? But at least for people like me, they’re damn close to all that matters.
I believe this is the best one for learning about entropy and thermodynamics (and a bit of stat mech) from this perspective: http://www.av8n.com/physics/thermo/