treating intelligence as a machine that optimizes things is missing the entire point of intelligence. If you had ever read Douglas Hofstadter’s “Godel Escher Bach”, or Christopher Alexander’s “The Nature of Order” series, you might have a greater appreciation for the role that abstract pattern recognition and metaphor plays in intelligence.
Eliezer has read GEB and praised it above the mountains (literally). So a charitable reader of him and his colleagues might suppose that they know the point about pattern recognition, but do not see the connection that you find obvious. And in fact I don’t know what you’re responding to, or what you think your second quoted sentence has to do with the first, or what practical conclusion you draw from it through what argument. Perhaps you could spell it out in detail for us mortals?
To focus on one problem with this, you write:
Eliezer has read GEB and praised it above the mountains (literally). So a charitable reader of him and his colleagues might suppose that they know the point about pattern recognition, but do not see the connection that you find obvious. And in fact I don’t know what you’re responding to, or what you think your second quoted sentence has to do with the first, or what practical conclusion you draw from it through what argument. Perhaps you could spell it out in detail for us mortals?