I’ve re-read your proposal and thought about it more. First, to answer your question, I think SRAM architectures fit into my proposal by accounting for the operations that are performed (just as in a von-Neumann machine). The only difference is the location, amount, and speed of memory. Ideal channel capacity and operational mutual information apply to any hardware that moves and processes data. An application most suited for SRAM architectures are slightly different than those for traditional architectures, but the performance accounting stays the same. This and the other deployment parameters just affect how close the runtime mutual information is to the upper limit (channel capacity). I have follow-on, unpublished work that adapts the roofline model to the information-based measure of computing that might clarify some of these things. Overall, my goal with this measure was what this article explicitly suggests NOT to do: rely on a central, general measure of computing (which is what caught my attention).
Thinking more on your work, I do really like the wholistic approach from manufacturing through operational outcomes. However, what additional value does this broader evaluation bring? For whom? If you get more funding and more people to work on this, does the outcome look something like a more academic SemiAnalysis? I think the computing ecosystem is certainly very complex, so going for depth and breadth will quickly exhaust your given resources. Then what is the more narrowed focus of this project? I personally would like to see a continually-updated characterization of the ecosystem, but I’m not sure that’s what you all are aiming for. Regardless, I look forward to your work. Please keep me up-to-date.
“I do really like the holistic approach from manufacturing through operational outcomes” Happy to hear that :)
“what additional value does this broader evaluation bring? For whom?” I have quite a few ideas, but I am working on polishing them up a bit more. Expect to hear more from us :).
“I personally would like to see a continually-updated characterization of the ecosystem”, :) Happy to hear that too. How might such a “continually-updated characterization” look like? A series of blogs? videos? A consistent structured framework that gets updated continually? Any thoughts on this?
I look forward to seeing it. I’m not sure what that characterization would look like, but I personally would like to see a diagram/roadmap of computing from raw resource extraction through manufacturing as well as an organized list of computing architectures. It seems there are enough novel devices out there to warrant some categorization and listing.
Thanks for the detailed response Max. This is certainly a very valid take. I am still brushing up my information theory, and frankly I do not have a very deep understanding of the field.
My initial understanding from a very shallow reading of your paper was that it is essentially calculating “bits passing through the logic unit/second”. However, having given it a little more attention, I realize that it goes deeper. Essentially, it’s calculating the change of bit values as they pass through the compute(Is this framing correct? suppose we have 1 input and the compute just reproduces the same value, the estimation of this compute would be 0? So the amount of work done is dependent on the actual input values as much as it is on the underlying hardware? What if input1 is 0 or 1 and the compute op is a multiply, this will output one of the inputs (either 0 or input2)? do we estimate that this op did half as much work as when both inputs are >1 ). If this is the case, how does it measure a compute’s performance “rating” independently of the workload running on it? (If I have entirely miss-understood the concept, please link something I can read to understand the underlying concepts) Also is there an “information per time” component in the performance metric?
Good questions. First, but answering your last question, my thesis is really just about proposing another way to quantify computational WORK. Currently, we use application-generic measures like Flops or application-specific measures like images/tokens processed. These definitions of work can then be normalized by unit time, energy, power, GHG emissions, water, etc, but my focus is on getting the numerator right (work) regardless of the denominator. I’m also only interested in application-generic measures of work.
You’ve got it exactly right that the amount of work done is dependent on the runtime data seen. This is a critical aspect. Mutual information is this runtime quantity that measure the actual work performed whereas channel capacity is an upper-bound on the mutual information possible and is only dependent on the hardware itself. The channel capacity is what would show up on a spec sheet, and MI is what would be plotted for a kernel on a roofline plot or similar runtime evaluation.
You’ve also got it right that if a compute operation always outputs the same value, its mutual information (and computational work) is zero. Could this operation not just be optimized away with a fixed constant? If so, what was the use of performing the operation? Either the inputs are fixed, or the inputs vary but the output is always the same. In both cases, replacing the output with a constant changes nothing. Shouldn’t this mean there was no value to performing the operation in the first place? This is where discussions about structural sparsity come into play. If we know a certain input is 0 and we’re multiplying it with another number, we already know what the outcome is, so there is no reduction of uncertainty by multiplying by a known 0.
The general intuition behind this work is that, just like communication, computation is not just about the outputs you see, its about what outputs you DIDN’T see. We use math to formalize relationships and to talk about the exactness of real numbers, but no hardware in finite space/time can operate on the infinite set of real numbers. Instead, when we get an output value, it’s simply one of 2^64 (or 2^32, 2^16, 2^8, 2^4, …) possible output states. An FP64 operation manages a larger state space than an NVFP4 operation does. Bit width approximations of Flops (where a Flops value is scaled by the bit-width of the operands) is sometimes used to capture this difference in complexity—like with the TPP export control definitions. However, this is exactly where the runtime distribution of data matters. If I know my inputs are restricted to NVFP4 range, and computing in FP64 doesn’t use any more number of states than computing in NVFP4, should such an inefficient program’s work really scale by a factor of 16? I think not, and that’s why the runtime data is critical to measuring computational work IMHO.
Essentially, how much uncertainty about the output does an operation resolve? That’s what I think we should measure. By doing so, we unify computation and communication performance measurement and are able to generalize computing measures across digital, noisy, analog, neuromorphic, and maybe even quantum regimes. Communication has had a theoretically-grounded and universal measure of throughput for ~80 years. Why doesn’t computing?
There’s more, but I’ll leave it here. Please feel free to email me anytime: mhawkins60@gatech.edu
I’ve re-read your proposal and thought about it more. First, to answer your question, I think SRAM architectures fit into my proposal by accounting for the operations that are performed (just as in a von-Neumann machine). The only difference is the location, amount, and speed of memory. Ideal channel capacity and operational mutual information apply to any hardware that moves and processes data. An application most suited for SRAM architectures are slightly different than those for traditional architectures, but the performance accounting stays the same. This and the other deployment parameters just affect how close the runtime mutual information is to the upper limit (channel capacity). I have follow-on, unpublished work that adapts the roofline model to the information-based measure of computing that might clarify some of these things. Overall, my goal with this measure was what this article explicitly suggests NOT to do: rely on a central, general measure of computing (which is what caught my attention).
Thinking more on your work, I do really like the wholistic approach from manufacturing through operational outcomes. However, what additional value does this broader evaluation bring? For whom? If you get more funding and more people to work on this, does the outcome look something like a more academic SemiAnalysis? I think the computing ecosystem is certainly very complex, so going for depth and breadth will quickly exhaust your given resources. Then what is the more narrowed focus of this project? I personally would like to see a continually-updated characterization of the ecosystem, but I’m not sure that’s what you all are aiming for. Regardless, I look forward to your work. Please keep me up-to-date.
“I do really like the holistic approach from manufacturing through operational outcomes” Happy to hear that :)
“what additional value does this broader evaluation bring? For whom?” I have quite a few ideas, but I am working on polishing them up a bit more. Expect to hear more from us :).
“I personally would like to see a continually-updated characterization of the ecosystem”, :) Happy to hear that too. How might such a “continually-updated characterization” look like? A series of blogs? videos? A consistent structured framework that gets updated continually? Any thoughts on this?
I look forward to seeing it. I’m not sure what that characterization would look like, but I personally would like to see a diagram/roadmap of computing from raw resource extraction through manufacturing as well as an organized list of computing architectures. It seems there are enough novel devices out there to warrant some categorization and listing.
Thanks for the detailed response Max. This is certainly a very valid take. I am still brushing up my information theory, and frankly I do not have a very deep understanding of the field.
My initial understanding from a very shallow reading of your paper was that it is essentially calculating “bits passing through the logic unit/second”. However, having given it a little more attention, I realize that it goes deeper. Essentially, it’s calculating the change of bit values as they pass through the compute(Is this framing correct? suppose we have 1 input and the compute just reproduces the same value, the estimation of this compute would be 0? So the amount of work done is dependent on the actual input values as much as it is on the underlying hardware? What if input1 is 0 or 1 and the compute op is a multiply, this will output one of the inputs (either 0 or input2)? do we estimate that this op did half as much work as when both inputs are >1 ). If this is the case, how does it measure a compute’s performance “rating” independently of the workload running on it?
(If I have entirely miss-understood the concept, please link something I can read to understand the underlying concepts)
Also is there an “information per time” component in the performance metric?
Good questions. First, but answering your last question, my thesis is really just about proposing another way to quantify computational WORK. Currently, we use application-generic measures like Flops or application-specific measures like images/tokens processed. These definitions of work can then be normalized by unit time, energy, power, GHG emissions, water, etc, but my focus is on getting the numerator right (work) regardless of the denominator. I’m also only interested in application-generic measures of work.
You’ve got it exactly right that the amount of work done is dependent on the runtime data seen. This is a critical aspect. Mutual information is this runtime quantity that measure the actual work performed whereas channel capacity is an upper-bound on the mutual information possible and is only dependent on the hardware itself. The channel capacity is what would show up on a spec sheet, and MI is what would be plotted for a kernel on a roofline plot or similar runtime evaluation.
You’ve also got it right that if a compute operation always outputs the same value, its mutual information (and computational work) is zero. Could this operation not just be optimized away with a fixed constant? If so, what was the use of performing the operation? Either the inputs are fixed, or the inputs vary but the output is always the same. In both cases, replacing the output with a constant changes nothing. Shouldn’t this mean there was no value to performing the operation in the first place? This is where discussions about structural sparsity come into play. If we know a certain input is 0 and we’re multiplying it with another number, we already know what the outcome is, so there is no reduction of uncertainty by multiplying by a known 0.
The general intuition behind this work is that, just like communication, computation is not just about the outputs you see, its about what outputs you DIDN’T see. We use math to formalize relationships and to talk about the exactness of real numbers, but no hardware in finite space/time can operate on the infinite set of real numbers. Instead, when we get an output value, it’s simply one of 2^64 (or 2^32, 2^16, 2^8, 2^4, …) possible output states. An FP64 operation manages a larger state space than an NVFP4 operation does. Bit width approximations of Flops (where a Flops value is scaled by the bit-width of the operands) is sometimes used to capture this difference in complexity—like with the TPP export control definitions. However, this is exactly where the runtime distribution of data matters. If I know my inputs are restricted to NVFP4 range, and computing in FP64 doesn’t use any more number of states than computing in NVFP4, should such an inefficient program’s work really scale by a factor of 16? I think not, and that’s why the runtime data is critical to measuring computational work IMHO.
Essentially, how much uncertainty about the output does an operation resolve? That’s what I think we should measure. By doing so, we unify computation and communication performance measurement and are able to generalize computing measures across digital, noisy, analog, neuromorphic, and maybe even quantum regimes. Communication has had a theoretically-grounded and universal measure of throughput for ~80 years. Why doesn’t computing?
There’s more, but I’ll leave it here. Please feel free to email me anytime: mhawkins60@gatech.edu
Thank you. That clarifies a lot of my questions. I will re-read your paper. Let’s talk in detail, in a couple of weeks. :)