Aqueducts. Water “wants” to flow downhill. Humans want the water to flow from remote mountain springs into our cities and homes. So, we provide an incentive gradient: the water can go (locally) downhill fastest by following our aqueduct. Could the water go down faster by some other route? Well, it could spray through a leak in the pipe/channel, for instance.
Yudkowsky claims that every cause wants to become a cult—i.e. there is a positive feedback loop which amplifies cult-like aspects of causes. Leaders are incentivized to play along with this—i.e. to “give the cause what it wants” in exchange for themselves being in charge. Note that this incentive pressure applies regardless of whether the leaders actually want their cause to become a cult. What would it look like for a cause’s leaders satisfy this incentive via some other strategy? Basically, they could take the “extreme” members who want to push that positive feedback loop, and give them some position/outlet which satisfies the relevant group-status needs without actually pushing marginal people out of the group.
Filters (the physical kind, like coffee filters). Filters select for very small particles, so if we look at what makes it through, it’s “incentivized” to be small. But things could satisfy the incentive (i.e. sneak through the filter) in other ways—e.g. a microorganism could literally eat its way through, or weakly-soluble salts could dissolve and re-precipitate on the other side.
The second one is a good example for selection incentive: the incentives are there regardless of what we want. I like the counter intuitive actions in the third example: organisms are deliberately trying to achieve high reward by taking counterintuitive actions.
It’s also interesting to consider that the organisms that eat through filters aren’t always doing it deliberately. Some organisms may be designed to eat through filters without knowing what’s on the other side. They may have evolved to attack a particular filter, possibly after recognizing it, because filters tend to exist when there is some valuable resource on the other side.
Aqueducts. Water “wants” to flow downhill. Humans want the water to flow from remote mountain springs into our cities and homes. So, we provide an incentive gradient: the water can go (locally) downhill fastest by following our aqueduct.
Could the water go down faster by some other route? Well, it could spray through a leak in the pipe/channel, for instance.
Yudkowsky claims that every cause wants to become a cult—i.e. there is a positive feedback loop which amplifies cult-like aspects of causes. Leaders are incentivized to play along with this—i.e. to “give the cause what it wants” in exchange for themselves being in charge. Note that this incentive pressure applies regardless of whether the leaders actually want their cause to become a cult.
What would it look like for a cause’s leaders satisfy this incentive via some other strategy? Basically, they could take the “extreme” members who want to push that positive feedback loop, and give them some position/outlet which satisfies the relevant group-status needs without actually pushing marginal people out of the group.
Filters (the physical kind, like coffee filters). Filters select for very small particles, so if we look at what makes it through, it’s “incentivized” to be small.
But things could satisfy the incentive (i.e. sneak through the filter) in other ways—e.g. a microorganism could literally eat its way through, or weakly-soluble salts could dissolve and re-precipitate on the other side.
The second one is a good example for selection incentive: the incentives are there regardless of what we want. I like the counter intuitive actions in the third example: organisms are deliberately trying to achieve high reward by taking counterintuitive actions.
It’s also interesting to consider that the organisms that eat through filters aren’t always doing it deliberately. Some organisms may be designed to eat through filters without knowing what’s on the other side. They may have evolved to attack a particular filter, possibly after recognizing it, because filters tend to exist when there is some valuable resource on the other side.