But optimistic errors are catastrophic. The agent will systematically seek out the behaviors that receive the high reward, and will use loopholes to avoid penalties when something actually bad happens. So even if these errors are extremely rare initially, they can totally mess up my agent.
In other words, make sure your agent can’t wirehead! And from evolution’s point of view, not spending your time promoting your inclusive genetic fitness is wireheading.
We didn’t have the technology to literally wirehead until maybe very recently, so we went about it the long way round. But we still spend a lot of time and resources on e.g. consuming art, even absent signalling benefits, like watching TV alone at home. And evolution doesn’t seem to be likely to “fix” that given some more time.
We don’t behave in a “Malthusian” way, investing all extra resources in increasing the number or relative proportion of our descendants in the next generation. Even though we definitely could, since population grows geometrically. It’s hard to have more than 10 children, but if every descendant of yours has 10 children as well, you can spend even the world’s biggest fortune. And yet such clannish behavior is not a common theme of any history I’ve read; people prefer to get (almost unboundedly) richer instead, and spend those riches on luxuries, not children.
And evolution doesn’t seem to be likely to “fix” that given some more time.
Why would you suppose that?
We don’t behave in a “Malthusian” way, investing all extra resources in increasing the number or relative proportion of our descendants in the next generation. Even though we definitely could, since population grows geometrically. It’s hard to have more than 10 children, but if every descendant of yours has 10 children as well, you can spend even the world’s biggest fortune. And yet such clannish behavior is not a common theme of any history I’ve read; people prefer to get (almost unboundedly) richer instead, and spend those riches on luxuries, not children.
Isn’t that just due to the rapid advance of technology creating a world in disequilibrium? In the ancestral environment of pre-agricultural societies these behaviors you describe line up with maximizing inclusive genetic fitness pretty well; any recorded history you can read is too new and short to reflect what evolution intended to select for.
Exactly for the reason you give yourself—we now change our behavior and our environment on much shorter timescales than evolution operates on, due in large part to modern technology. We have a goal of circumventing evolution (see: this post) and we modify our goals to suit ourselves. Evolution is no longer fast enough to be the main determinant of prevailing behavior.
In the ancestral environment of pre-agricultural societies these behaviors you describe line up with maximizing inclusive genetic fitness pretty well
We don’t know almost anything about most relevant human behavior from before the invention of writing. Did they e.g. consume a lot of art (music, storytelling, theater, dancing)? How much did such consumption correlate with status or other fitness benefits, e.g. by conspicuous consumption or advertising wealth? We really don’t know.
Ok I see, I was just confused by the wording “given some more time”. I’ve become less optimistic over time about how long this disequilibrium will last given how quickly certain religious communities are growing with the explicit goal of outbreeding the rest of us.
In other words, make sure your agent can’t wirehead! And from evolution’s point of view, not spending your time promoting your inclusive genetic fitness is wireheading.
We didn’t have the technology to literally wirehead until maybe very recently, so we went about it the long way round. But we still spend a lot of time and resources on e.g. consuming art, even absent signalling benefits, like watching TV alone at home. And evolution doesn’t seem to be likely to “fix” that given some more time.
We don’t behave in a “Malthusian” way, investing all extra resources in increasing the number or relative proportion of our descendants in the next generation. Even though we definitely could, since population grows geometrically. It’s hard to have more than 10 children, but if every descendant of yours has 10 children as well, you can spend even the world’s biggest fortune. And yet such clannish behavior is not a common theme of any history I’ve read; people prefer to get (almost unboundedly) richer instead, and spend those riches on luxuries, not children.
Why would you suppose that?
Isn’t that just due to the rapid advance of technology creating a world in disequilibrium? In the ancestral environment of pre-agricultural societies these behaviors you describe line up with maximizing inclusive genetic fitness pretty well; any recorded history you can read is too new and short to reflect what evolution intended to select for.
Exactly for the reason you give yourself—we now change our behavior and our environment on much shorter timescales than evolution operates on, due in large part to modern technology. We have a goal of circumventing evolution (see: this post) and we modify our goals to suit ourselves. Evolution is no longer fast enough to be the main determinant of prevailing behavior.
We don’t know almost anything about most relevant human behavior from before the invention of writing. Did they e.g. consume a lot of art (music, storytelling, theater, dancing)? How much did such consumption correlate with status or other fitness benefits, e.g. by conspicuous consumption or advertising wealth? We really don’t know.
Ok I see, I was just confused by the wording “given some more time”. I’ve become less optimistic over time about how long this disequilibrium will last given how quickly certain religious communities are growing with the explicit goal of outbreeding the rest of us.