I’m having a hard time figuring out how the examples of evolution and markets map on to the agent above. They’re both processes that seem to take advantage of the fact that there are agents inside them trying to optimize goals, and are the results of these agents and processes coming into conflict with each other and reality. While I can imagine the unselfaware agent simulating these instances, I don’t see a way to actually view it as a process made up of agents trying to achieve goals.
Yes, I kind of over-answered the question: I’m saying even if you have some optimization process that is not even an agent you can get instrumental “behaviors” that are not safe, let alone an agent that is not self-aware.
I’m having a hard time figuring out how the examples of evolution and markets map on to the agent above. They’re both processes that seem to take advantage of the fact that there are agents inside them trying to optimize goals, and are the results of these agents and processes coming into conflict with each other and reality. While I can imagine the unselfaware agent simulating these instances, I don’t see a way to actually view it as a process made up of agents trying to achieve goals.
Yes, I kind of over-answered the question: I’m saying even if you have some optimization process that is not even an agent you can get instrumental “behaviors” that are not safe, let alone an agent that is not self-aware.