Eliezer: You go on with the same morals as before, and the same moral arguments as before. There is no sudden Grand Overlord Procedure to which you can appeal to get a perfectly trustworthy answer.
‘Same moral arguments as before’ doesn’t seem like an answer, in the same sense as ‘you should continue as before’ is not a good advice for cavemen (who could benefit from being brought into modern civilization). If cavemen can vaguely describe what they want from environment, this vague explanation can be used to produce optimized environment by sufficiently powerful optimization process that is external to cavemen, based on the precise structure of current environment. It won’t go all the way there (otherwise, a problem of ignorant jinn will kick in), but it can really help.
Likewise, the problem of ‘metamorality’ is in producing a specification of goals that is better than vague explanations of moral philosophers. For that, we need to produce vague explanation of what we think morality is, and set an optimization process on these explanations to produce a better description of morality, based on current state of environment (or, specifically, humanity and human cognitive architecture).
These posts sure clarify something for the confused, but what is the content in the sense I described? I hope the above quotation was not a curiosity stopper.
we need to produce vague explanation of what we think morality is, and set an optimization process on these explanations to produce a better description of morality
Agreed, but this post isn’t it, and wasn’t meant to be. This post basically nailed down what form morality should take. Perhaps it could be expressed as a summation of all our thousand shards of desire. In order to actually compute this, we would use a function which Yudkowsky calls Coherent Extrapolated Volition. That’s what he describes as “what we would come to believe if we knew all empirical facts and had a million years to think about it”. Actually calculating morality is left as an exercise for the reader.
Eliezer: You go on with the same morals as before, and the same moral arguments as before. There is no sudden Grand Overlord Procedure to which you can appeal to get a perfectly trustworthy answer.
‘Same moral arguments as before’ doesn’t seem like an answer, in the same sense as ‘you should continue as before’ is not a good advice for cavemen (who could benefit from being brought into modern civilization). If cavemen can vaguely describe what they want from environment, this vague explanation can be used to produce optimized environment by sufficiently powerful optimization process that is external to cavemen, based on the precise structure of current environment. It won’t go all the way there (otherwise, a problem of ignorant jinn will kick in), but it can really help.
Likewise, the problem of ‘metamorality’ is in producing a specification of goals that is better than vague explanations of moral philosophers. For that, we need to produce vague explanation of what we think morality is, and set an optimization process on these explanations to produce a better description of morality, based on current state of environment (or, specifically, humanity and human cognitive architecture).
These posts sure clarify something for the confused, but what is the content in the sense I described? I hope the above quotation was not a curiosity stopper.
Agreed, but this post isn’t it, and wasn’t meant to be. This post basically nailed down what form morality should take. Perhaps it could be expressed as a summation of all our thousand shards of desire. In order to actually compute this, we would use a function which Yudkowsky calls Coherent Extrapolated Volition. That’s what he describes as “what we would come to believe if we knew all empirical facts and had a million years to think about it”. Actually calculating morality is left as an exercise for the reader.