Nice post—simple, clear, explains an important point better than the Yudkowsky-Ngo dialogue did.
The partially-baked theory of agency I’m working on is basically that agents are a kind of chain reaction: whereas fires are chain reactions in which a small amount of heat grows larger (by consuming fuel and oxygen) agents are chain reactions in which a small cluster of convergent instrumental resources (money, knowledge, energy) grows larger.
One of the reasons I like this theory is that it fits nicely with selection-based arguments like the one you are making here. It makes sense that we should expect competitions for knowledge, resources, and influence to be won by knowledge+resources+influence chain reactions!
Nice post—simple, clear, explains an important point better than the Yudkowsky-Ngo dialogue did.
The partially-baked theory of agency I’m working on is basically that agents are a kind of chain reaction: whereas fires are chain reactions in which a small amount of heat grows larger (by consuming fuel and oxygen) agents are chain reactions in which a small cluster of convergent instrumental resources (money, knowledge, energy) grows larger.
One of the reasons I like this theory is that it fits nicely with selection-based arguments like the one you are making here. It makes sense that we should expect competitions for knowledge, resources, and influence to be won by knowledge+resources+influence chain reactions!