But just as there can be laws governing mechanical systems … there can be laws governing the behavior of value-optimizing systems which are distinct from the other laws of nature.
Agreed. But then, I believe that my main point still stands: trying to build a value system other than M that does not result in its host mind being destroyed, would be as futile as trying to build a hot air balloon that goes to Mars.
And what I mean by “destructive” is that they tend to destroy.
Well, yes, but what if “destroy oneself as soon as possible” is a core value in one particular value system ?
Agreed. But then, I believe that my main point still stands: trying to build a value system other than M that does not result in its host mind being destroyed, would be as futile as trying to build a hot air balloon that goes to Mars.
Well, yes, but what if “destroy oneself as soon as possible” is a core value in one particular value system ?
We ought not expect to find any significantly powerful optimizers implementing that value system.