Thanks for writing this post. I have a handful of quick questions: (a) What was the reference MIPS (or the corresponding CPU) you used for the c. 2019-2020 data point? (b) What was the constant amount of RAM you used to run Stockfish? (c) Do I correctly understand that the Stockfish-to-MIPS comparison is based on the equation [edit: not sure how to best format this LaTeX...]:
slowed down SF8 running timereference SF8 running time=historical MIPSreference MIPS
So, your post piqued my interest to investigate the Intel 80486 a bit more with the question in mind: how comparable are old vs. new CPUs according not just to MIPS but also to other metrics?
Instructions per second (MIPS): Using the Wikipedia MIPS source, the 80486 has 70 MIPS at 100 MHz, whereas a recent Skylake 8-core CPU (i9-9900K) has around 50,000 MIPS/core at a clock of 4.7 GHz, and closer to 40,000 MIPS/core at a sustained clock of 3.6 GHz.
Memory bandwidth: From Wikipedia, the 80486 has a 33 MHz bus and a data rate of up to 32 bits, so roughly 130 MB/s, whereas the 9900K is closer to 40 GB/s (from Intel).
Memory latency: The 80486 takes 5 bus cycles at 33 MHz, or 15 CPU cycles at 100 MHz, to access memory (16 byte cache line size, from comp.arch). From Anandtech, Skylake can take 100 cycles to access memory (64 byte cache line size, typical for x86_64).
Memory capacity: The 80486 is a 32-bit CPU and can address up to 4 GB of RAM. Again from Wikipedia, the 80486 accepts SIMM form factor RAM, with up to 2 GB capacity (but tens of MB may have been more commonly available). On the other hand, the 9900K supports a maximum memory capacity of 128 GB.
It would seem that improvements in MIPS have outpaced improvements in memory-related metrics, e.g. memory latency in units of cycles has gotten worse. What I don’t know is how sensitive Stockfish is to variations in memory performance. However I would conclude that I’d update the estimated SF8 ELO on older hardware upwards when additionally accounting for the effect of memory-related metrics in addition to MIPS. In other words, it seems more likely to me that the SF8 ELO curve is underestimated when you include both MIPS and memory-related effects.
(There is another direction one could follow: get Stockfish to compile on an 80486 or other old hardware that one has lying around, and report back with results.)
(a) The most recent data points are from CCRL. They use an i7-4770k and the listed tournament conditions. With this setup, SF11 has about 3500 ELOs. That’s what I used as the baseline to calibrate my own machine (an i7-7700k).
(b) I used the SF8 default which is 1 GB.
(c) Yes. However, the hardware details (RAM, memory bandwidth) are not all that important. You can use these SF9 benchmarks on various CPUs. For example, the AMD Ryzen 1800 is listed with 304,510 MIPS and gets 14,377,000 nodes/sec on Stockfish (i.e., 19.9 nodes per MIPS). The oldest CPU in the list, the Pentium-150 has 282 MIPS and reaches 5,626 nodes/sec (i.e., 47.2 nodes per MIPS). That’s about a factor of two difference, due to memory and related advantages. As we’re getting that much every 18 months due to Moore’s law, it’s a small (but relevant) detail, and decreases the hardware overhang slightly. Thanks for bringing that up!
Giving Stockfish more memory also helps, but not a lot. Also, you can’t give 128 GB of RAM to a 486 CPU. The 1 GB is probably already stretching it. Another small detail which reduces the overhang by likely less than one year.
There are a few more subtle details like endgame databases. Back then, these were small, constrained by disk space limitations. Today, we have 7-stone endgame databases through the cloud (they weigh in at 140 TB). That seems to be worth about 50 ELO.
Wow, thanks for the very comprehensive response. (Also fun to see someone has compiled a modern chess engine on early-mid-90s hardware and shared their results.)
Thanks for writing this post. I have a handful of quick questions: (a) What was the reference MIPS (or the corresponding CPU) you used for the c. 2019-2020 data point? (b) What was the constant amount of RAM you used to run Stockfish? (c) Do I correctly understand that the Stockfish-to-MIPS comparison is based on the equation [edit: not sure how to best format this LaTeX...]:
slowed down SF8 running timereference SF8 running time=historical MIPSreference MIPS
So, your post piqued my interest to investigate the Intel 80486 a bit more with the question in mind: how comparable are old vs. new CPUs according not just to MIPS but also to other metrics?
Instructions per second (MIPS): Using the Wikipedia MIPS source, the 80486 has 70 MIPS at 100 MHz, whereas a recent Skylake 8-core CPU (i9-9900K) has around 50,000 MIPS/core at a clock of 4.7 GHz, and closer to 40,000 MIPS/core at a sustained clock of 3.6 GHz.
Memory bandwidth: From Wikipedia, the 80486 has a 33 MHz bus and a data rate of up to 32 bits, so roughly 130 MB/s, whereas the 9900K is closer to 40 GB/s (from Intel).
Memory latency: The 80486 takes 5 bus cycles at 33 MHz, or 15 CPU cycles at 100 MHz, to access memory (16 byte cache line size, from comp.arch). From Anandtech, Skylake can take 100 cycles to access memory (64 byte cache line size, typical for x86_64).
Memory capacity: The 80486 is a 32-bit CPU and can address up to 4 GB of RAM. Again from Wikipedia, the 80486 accepts SIMM form factor RAM, with up to 2 GB capacity (but tens of MB may have been more commonly available). On the other hand, the 9900K supports a maximum memory capacity of 128 GB.
It would seem that improvements in MIPS have outpaced improvements in memory-related metrics, e.g. memory latency in units of cycles has gotten worse. What I don’t know is how sensitive Stockfish is to variations in memory performance. However I would conclude that I’d update the estimated SF8 ELO on older hardware upwards when additionally accounting for the effect of memory-related metrics in addition to MIPS. In other words, it seems more likely to me that the SF8 ELO curve is underestimated when you include both MIPS and memory-related effects.
(There is another direction one could follow: get Stockfish to compile on an 80486 or other old hardware that one has lying around, and report back with results.)
(a) The most recent data points are from CCRL. They use an i7-4770k and the listed tournament conditions. With this setup, SF11 has about 3500 ELOs. That’s what I used as the baseline to calibrate my own machine (an i7-7700k).
(b) I used the SF8 default which is 1 GB.
(c) Yes. However, the hardware details (RAM, memory bandwidth) are not all that important. You can use these SF9 benchmarks on various CPUs. For example, the AMD Ryzen 1800 is listed with 304,510 MIPS and gets 14,377,000 nodes/sec on Stockfish (i.e., 19.9 nodes per MIPS). The oldest CPU in the list, the Pentium-150 has 282 MIPS and reaches 5,626 nodes/sec (i.e., 47.2 nodes per MIPS). That’s about a factor of two difference, due to memory and related advantages. As we’re getting that much every 18 months due to Moore’s law, it’s a small (but relevant) detail, and decreases the hardware overhang slightly. Thanks for bringing that up!
Giving Stockfish more memory also helps, but not a lot. Also, you can’t give 128 GB of RAM to a 486 CPU. The 1 GB is probably already stretching it. Another small detail which reduces the overhang by likely less than one year.
There are a few more subtle details like endgame databases. Back then, these were small, constrained by disk space limitations. Today, we have 7-stone endgame databases through the cloud (they weigh in at 140 TB). That seems to be worth about 50 ELO.
Wow, thanks for the very comprehensive response. (Also fun to see someone has compiled a modern chess engine on early-mid-90s hardware and shared their results.)