Thanks for the yearly update! I have some thoughts on why we care about string diagrams and commutative diagrams so much. (It’s not even just “category theory”.) I’ll poke you later to talk about them in greater depth but for quick commentary:
For string diagrams it’s something like “string diagrams are a minimal way to represent both timelike propagation of information and spacelike separation of causal influence”. If you want to sketch out some causal graph, string diagrams are the natural best way to do that. From there you start caring about monoidal structure and you’re off to the races.
For commutative diagrams the story is different but related, though admittedly I understand what’s going on with [commutative diagrams]+[sparse activations] way way less. I’d say it’s something like “the existence of a satisfied commutative diagram puts strong constraints on other aspects of the neural net, like what form the latent space(s?) and maps to and from them have to look like and what they have to do and what information has to get preserved or discarded”.
For one last observation, a friend’s been poking me about the sense that constraints and equipartition/environment are dual to each other, and that there’s a correspondence (for bounded systems at least) between phase volume size-change and the sign of something like an informational analogue to thermodynamic temperature. (And also that your approach is importantly incomplete in currently only dealing in theory and not engineering, but for my part I think that that’s priced in to how you talk about your plan.)
Thanks for the yearly update! I have some thoughts on why we care about string diagrams and commutative diagrams so much. (It’s not even just “category theory”.) I’ll poke you later to talk about them in greater depth but for quick commentary:
For string diagrams it’s something like “string diagrams are a minimal way to represent both timelike propagation of information and spacelike separation of causal influence”. If you want to sketch out some causal graph, string diagrams are the natural best way to do that. From there you start caring about monoidal structure and you’re off to the races.
For commutative diagrams the story is different but related, though admittedly I understand what’s going on with [commutative diagrams]+[sparse activations] way way less. I’d say it’s something like “the existence of a satisfied commutative diagram puts strong constraints on other aspects of the neural net, like what form the latent space(s?) and maps to and from them have to look like and what they have to do and what information has to get preserved or discarded”.
For one last observation, a friend’s been poking me about the sense that constraints and equipartition/environment are dual to each other, and that there’s a correspondence (for bounded systems at least) between phase volume size-change and the sign of something like an informational analogue to thermodynamic temperature. (And also that your approach is importantly incomplete in currently only dealing in theory and not engineering, but for my part I think that that’s priced in to how you talk about your plan.)
Bother me on Discord?