This is a peculiar essay. If there are limits to how big, how small, or how stable you can make some object, that doesn’t mean it’s impossible to maximize the number of copies of the object. On the contrary, knowing those limits tells you what maximization looks like.
Perhaps you’re interpreting “maximize” to mean “increase without limit”? Maximization just means to increase as much as possible. If there’s a limit, maximization means you go up to the limit.
The most interesting issue touched upon, is the uncertainty over exactly what counts as a paperclip. This corresponds to a genuinely important issue in AI value systems, namely, how does the AI decide what counts as goodness or happiness or consent (etc.), when it encounters new situations and new possibilities. But the essay doesn’t explore that direction for long.
This is a peculiar essay. If there are limits to how big, how small, or how stable you can make some object, that doesn’t mean it’s impossible to maximize the number of copies of the object. On the contrary, knowing those limits tells you what maximization looks like.
Perhaps you’re interpreting “maximize” to mean “increase without limit”? Maximization just means to increase as much as possible. If there’s a limit, maximization means you go up to the limit.
The most interesting issue touched upon, is the uncertainty over exactly what counts as a paperclip. This corresponds to a genuinely important issue in AI value systems, namely, how does the AI decide what counts as goodness or happiness or consent (etc.), when it encounters new situations and new possibilities. But the essay doesn’t explore that direction for long.