No need to import human value judgements, this only confuses the argument for the reader.
On the one hand, I’d agree with you… but consider this excellent example of our “objective/unemotional” perceptions failing to communicate to us how game theory feels from the inside!
If told about how a machine that wanted to maximize A and minimize B ended up self-modifying to maximize a B-correlated C, most humans would not feel strongly about that, they’d hardly pay attention—but they’d wish they had if later told that, say, A was “hedonism”, B was “suffering” and C was “murder”. Such insensitivity plagues nearly everyone, even enlightened LW readers.
Generating drama so as to stir the unwashed masses sounds… suboptimal… and I say this as an avid drama-generator. Surely there are better ways to combat the plague of complacency?
On the one hand, I’d agree with you… but consider this excellent example of our “objective/unemotional” perceptions failing to communicate to us how game theory feels from the inside!
If told about how a machine that wanted to maximize A and minimize B ended up self-modifying to maximize a B-correlated C, most humans would not feel strongly about that, they’d hardly pay attention—but they’d wish they had if later told that, say, A was “hedonism”, B was “suffering” and C was “murder”. Such insensitivity plagues nearly everyone, even enlightened LW readers.
Generating drama so as to stir the unwashed masses sounds… suboptimal… and I say this as an avid drama-generator. Surely there are better ways to combat the plague of complacency?