I propose this conjecture: In any sufficiently complex physical system there exists a subsystem that can be interpreted as the mental process of an sentient being experiencing unbearable sufferings.
In this case, Eliezer’s goal is like avoiding crushing the ants while walking on the top of an anthill.
Jordan: “I imagine a distant future with just a smattering of paper clip maximizers—having risen in different galaxies with slightly different notions of what a paperclip is—might actually be quite interesting.”
That’s exactly how I imagine the distant future. And I very much like to point to the cyclic cellular automaton (java applet) as a visualization. Actually, I speculate that we live in a small part of the space-time continuum not yet eaten by a paper clip maximizer. Now you may ask: Why don’t we see huge blobs of paper clip maximizers expanding on the night sky? My answer is that they are expanding with the speed of light in every direction.
Note: I abused the term paper clip maximizer somewhat. Originally I called these things Expanding Space Amoebae, but PCM is more OB.