@Caledonian: A properly designed AI should care about the very things that we care. Or what would be the purpose of building it? Wipe out the human race trying to prove the Riemann hypothesis? I’m well aware of the huge mind-design space out there—but I think that we should aim for the right spot when designing our first AI (now, where is the spot, and how to aim for it is still an open question).
I would be interested in seeing what you think about how we should build an AI (and what its goals should be as well...)