Hi, I’m a lurker. I work on CPUs. This also motivated me to post!
This is a rather niche topic, but I want to express it, because I greatly enjoy seeing other ramble about their deep-work domain expertise, so maybe someone will find this interesting too? This is relatively similar to the concept behind the podcast [What’s your problem?], in which engineers talk about ridiculously niche problems that are integral to their field.
Anyways—here’s my problem.
Fuzzing (maybe known as mutation based testing, or coverage directed verification, or 10 other different names) has, in my opinion, been revolutionary for the software security industry. [AFL] is probably the best and most successful example, and I think most people would agree with me that this tool has saved millions of manhours in finding security vulnerabilities.
Why don’t we have such tools in hardware? Well, my personal opinion is that EDA tools are rather monopolistic and cumbersome relative to e.g. GCC (e.g. imagine paying millions of dollars for a GCC release!), and a partial side-effect of that is that the language hardware codes in (verilog, systemverilog) is so ingrained we can’t get out of it.
This is just barely starting to change.
[Here] is my personal top favorite of a contender. What makes this cool is not entirely revolutionary new ideas, but rather the sheer amount of effort to make things just work is truly truly commendable.
The main perk of fuzzing is, frankly, finding low-ish hanging fruit. Just like how buffer overflows are, in some sense, a “known problem”, there’s a plethora of hardware vulnerabilities I’ve found that you wouldn’t believe are insanely easy to find. And I firmly think this can be done by simple fuzzing.
My project plan? Convert the detected vulnerabilities into generated exploitable software vulnerabilities. And I think the above project can fit into the “detection” aspect—honestly still a WIP for me to evaluate how good it is, or how complicated the fuzzer is (most of the time it’s just a wrapper around SMT solvers), but it’s something I’m excited about.
(On the “exploitable vulnerabilities” end, there is similar work [here], but I’ve done some experimentation with this and still find it rather cumbersome for a variety of details I won’t get in to.)
Oh I guess, while I’m on the topic of “bringing software paradigms into the hardware world”, let me also talk about CirctIR briefly.
I also believe LLVM was a bit of a boon for the software security world, enabling some really cool symbolic execution and/or reverse engineering tools. CirctIR is an attempt to basically bring this “intermediate representation” idea to hardware.
This “generator for intermediate language representation”, by the way, is similar to what Chisel currently does w.r.t generating verilog. But CirctIR is a little more generic, and frankly Chisel’s generator (called FIRRTL) is annoying in many ways.
Chris Lattner worked at SiFive for a bit, and made these same observations, so he spearheaded the CirctIR movement. Partially as a result, there are many similarities with FIRRTL and CirctIR (Chisel’s goal is to make hardware design easier, and CirctIR’s goal is to make designs portable and/or decouple these toolchain flows. Related goals, but still differentiable)
.
I’ve wanted for some time to play with this as well, but the fuzzing work has gotten me more interested currently and something I’m trying to make an MVP for at work.
I’m not sure whether you mean fuzzing of the synthesis tools (quick google here) or fuzzing of the synthesized devices (e.g., here; corrected). I worked with FPGAs a while back before even unit testing was established practice in software. I’m surprised that fuzzing isn’t used much, esp. as it seems much faster so close to the HW.
Those two links are the same. But yeah I’m referring to the latter, w.r.t fuzzing of the synthesized devices.
“Fuzzing” as a concept is used, but not very “block-level” (some some exceptions, e.g. you likely know about UVM’s support for random data streams, coming from an FPGA background). The fuzzing analogue in hardware might be called “constrained random verification”.
Fuzzing as I’ve heard it referenced is more of a jargon used in the software security world, the aforementioned AFL fuzzer being one example.
I do agree that traditional fuzzing isn’t used in hardware is rather surprising to me.
Maybe one reason fuzzing isn’t used more is that it is harder to detect failure? You don’t get page faults or exceptions or some such with hardware. What is your idea there?
Hi, I’m a lurker. I work on CPUs. This also motivated me to post!
This is a rather niche topic, but I want to express it, because I greatly enjoy seeing other ramble about their deep-work domain expertise, so maybe someone will find this interesting too? This is relatively similar to the concept behind the podcast [What’s your problem?], in which engineers talk about ridiculously niche problems that are integral to their field.
Anyways—here’s my problem.
Fuzzing (maybe known as mutation based testing, or coverage directed verification, or 10 other different names) has, in my opinion, been revolutionary for the software security industry. [AFL] is probably the best and most successful example, and I think most people would agree with me that this tool has saved millions of manhours in finding security vulnerabilities.
Why don’t we have such tools in hardware? Well, my personal opinion is that EDA tools are rather monopolistic and cumbersome relative to e.g. GCC (e.g. imagine paying millions of dollars for a GCC release!), and a partial side-effect of that is that the language hardware codes in (verilog, systemverilog) is so ingrained we can’t get out of it.
This is just barely starting to change.
[Here] is my personal top favorite of a contender. What makes this cool is not entirely revolutionary new ideas, but rather the sheer amount of effort to make things just work is truly truly commendable.
The main perk of fuzzing is, frankly, finding low-ish hanging fruit. Just like how buffer overflows are, in some sense, a “known problem”, there’s a plethora of hardware vulnerabilities I’ve found that you wouldn’t believe are insanely easy to find. And I firmly think this can be done by simple fuzzing.
My project plan? Convert the detected vulnerabilities into generated exploitable software vulnerabilities. And I think the above project can fit into the “detection” aspect—honestly still a WIP for me to evaluate how good it is, or how complicated the fuzzer is (most of the time it’s just a wrapper around SMT solvers), but it’s something I’m excited about.
(On the “exploitable vulnerabilities” end, there is similar work [here], but I’ve done some experimentation with this and still find it rather cumbersome for a variety of details I won’t get in to.)
Oh I guess, while I’m on the topic of “bringing software paradigms into the hardware world”, let me also talk about CirctIR briefly.
I also believe LLVM was a bit of a boon for the software security world, enabling some really cool symbolic execution and/or reverse engineering tools. CirctIR is an attempt to basically bring this “intermediate representation” idea to hardware.
This “generator for intermediate language representation”, by the way, is similar to what Chisel currently does w.r.t generating verilog. But CirctIR is a little more generic, and frankly Chisel’s generator (called FIRRTL) is annoying in many ways.
Chris Lattner worked at SiFive for a bit, and made these same observations, so he spearheaded the CirctIR movement. Partially as a result, there are many similarities with FIRRTL and CirctIR (Chisel’s goal is to make hardware design easier, and CirctIR’s goal is to make designs portable and/or decouple these toolchain flows. Related goals, but still differentiable)
.
I’ve wanted for some time to play with this as well, but the fuzzing work has gotten me more interested currently and something I’m trying to make an MVP for at work.
I’m not sure whether you mean fuzzing of the synthesis tools (quick google here) or fuzzing of the synthesized devices (e.g., here; corrected). I worked with FPGAs a while back before even unit testing was established practice in software. I’m surprised that fuzzing isn’t used much, esp. as it seems much faster so close to the HW.
Those two links are the same. But yeah I’m referring to the latter, w.r.t fuzzing of the synthesized devices.
“Fuzzing” as a concept is used, but not very “block-level” (some some exceptions, e.g. you likely know about UVM’s support for random data streams, coming from an FPGA background). The fuzzing analogue in hardware might be called “constrained random verification”.
Fuzzing as I’ve heard it referenced is more of a jargon used in the software security world, the aforementioned AFL fuzzer being one example.
I do agree that traditional fuzzing isn’t used in hardware is rather surprising to me.
ups. corrected.
Didn’t know UVMs.
Maybe one reason fuzzing isn’t used more is that it is harder to detect failure? You don’t get page faults or exceptions or some such with hardware. What is your idea there?