I think the big difference in expected complexity is between sampling the
space of possible singletons’ algorithms results and sampling the space of
competitive entities. I agree with Eliezer that an imprecisely
chosen value function, if relentlessly optimized, is likely to yield a dull
universe. To my mind the key is that the ability to relentlessly
optimize one function only exists if a singleton gets and keeps an
overwhelming advantage over everything else. If this does not happen, we
get competing entities with the computationally difficult problem of
outsmarting each other. Under this scenario, while I might not like the
detailed results, I’d expect them to be complex to much the same extent
and for much the same reasons as living organisms are complex.
@Jordan—agreed.
I think the big difference in expected complexity is between sampling the space of possible singletons’ algorithms results and sampling the space of competitive entities. I agree with Eliezer that an imprecisely chosen value function, if relentlessly optimized, is likely to yield a dull universe. To my mind the key is that the ability to relentlessly optimize one function only exists if a singleton gets and keeps an overwhelming advantage over everything else. If this does not happen, we get competing entities with the computationally difficult problem of outsmarting each other. Under this scenario, while I might not like the detailed results, I’d expect them to be complex to much the same extent and for much the same reasons as living organisms are complex.