I think we should figure out what our true utility function is, and then, if we have reason to be worried about bugs/errors in the AI, determine how to prevent or mitigate against such errors, which may include programming an utility function into the AI that’s not the true utility function, but an approximation that is less vulnerable to errors. I don’t see how we can do that without knowing the true utility function, or the likelihood of various kinds of errors. It seems like you’re jumping way too far ahead.
I think we know of at least one working idea of the utility function for one individual human in theory—show the human all the possible simulated futures and allow them to pick the best.
Of course that would be a complex vector utility function, and you couldn’t hope to map that down to a single simple scalar value that aggregates (even with voting).
The true optimal theoretical ‘utility function’ for humanity as a whole is probably something akin to simulating the evolution of human culture/civilization forward into many possible futures, disseminating all that knowledge, feeding that back in on itself recursively with numerous rounds of re-simulation and then allowing current political/economic systems to evolve forward with all that processed future-path knowledge.
I think we should figure out what our true utility function is, and then, if we have reason to be worried about bugs/errors in the AI, determine how to prevent or mitigate against such errors, which may include programming an utility function into the AI that’s not the true utility function, but an approximation that is less vulnerable to errors. I don’t see how we can do that without knowing the true utility function, or the likelihood of various kinds of errors. It seems like you’re jumping way too far ahead.
I think we know of at least one working idea of the utility function for one individual human in theory—show the human all the possible simulated futures and allow them to pick the best.
Of course that would be a complex vector utility function, and you couldn’t hope to map that down to a single simple scalar value that aggregates (even with voting).
The true optimal theoretical ‘utility function’ for humanity as a whole is probably something akin to simulating the evolution of human culture/civilization forward into many possible futures, disseminating all that knowledge, feeding that back in on itself recursively with numerous rounds of re-simulation and then allowing current political/economic systems to evolve forward with all that processed future-path knowledge.