How can this [isolated utility maximization] be countered?
If the values are faithfully noted this is no problem. It strictly is what you’d want.
I think the danger is that the utility maximizer optimizes for (possibly only slightly) wrong values. The complex value space might accidentally admit some such optimization because of e.g. leaving out some obscure value with was previously overlooked.
A utility optimizer shoud not optimize faster than errors in the value function can be found i.e. faster than humans can give collective feedback on it.
That is one reason companies (mentioned in your comment) sometimes produce goods that are in high demand until the secondary effects are noticed which then may take some time to fix e.g. by legislation.
I don’t think the speed of mere companes need to be slowed down to fix this (altough one might consider and model this). But more powerful utility maximizers definitely should time smooth their optimization process.
If the values are faithfully noted this is no problem. It strictly is what you’d want.
I think the danger is that the utility maximizer optimizes for (possibly only slightly) wrong values. The complex value space might accidentally admit some such optimization because of e.g. leaving out some obscure value with was previously overlooked.
A utility optimizer shoud not optimize faster than errors in the value function can be found i.e. faster than humans can give collective feedback on it.
That is one reason companies (mentioned in your comment) sometimes produce goods that are in high demand until the secondary effects are noticed which then may take some time to fix e.g. by legislation.
I don’t think the speed of mere companes need to be slowed down to fix this (altough one might consider and model this). But more powerful utility maximizers definitely should time smooth their optimization process.