> Failure mode: When B-cultured entities invest in “having more influence”, often the easiest way to do this will be for them to invest in or copy A’-cultured-entities/processes. This increases the total presence of A’-like processes in the world, which have many opportunities to coordinate because of their shared (power-maximizing) values. Moreover, the A’ culture has an incentive to trick the B culture(s) into thinking A’ will not take over the world, but eventually, A’ wins.
> In other words, the humans and human-aligned institutions not collectively being good enough at cooperation/bargaining risks a slow slipping-away of hard-to-express values and an easy takeover of simple-to-express values (e.g., power-maximization).
This doesn’t feel like other words to me, it feels like a totally different claim.
Hmm, perhaps this is indicative of a key misunderstanding.
For example, natural monopolies in the production web wouldn’t charge each other marginal costs, they would charge profit-maximizing profits.
Why not? The third paragraph of the story indicates that: “Companies closer to becoming fully automated achieve faster turnaround times, deal bandwidth, and creativity of negotiations.” In other words, at that point it could certainly happen that two monopolies would agree to charge each other lower cost if it benefitted both of them. (Unless you’d count that as instance of “charging profit-maximizing costs”?) The concern is that the subprocesses of each company/institution that get good at (or succeed at) bargaining with other institutions are subprocesses that (by virtue of being selected for speed and simplicity) are less aligned with human existence than the original overall company/institution, and that less-aligned subprocess grows to take over the institution, while always taking actions that are “good” for the host institution when viewed as a unilateral move in an uncoordinated game (hence passing as “aligned”).
At this point, my plan is try to consolidate what I think the are main confusions in the comments of this post, into one or more new concepts to form the topic of a new post.
At this point, my plan is try to consolidate what I think the are main confusions in the comments of this post, into one or more new concepts to form the topic of a new post.
Sounds great! I was thinking myself about setting aside some time to write a summary of this comment section (as I see it).
Hmm, perhaps this is indicative of a key misunderstanding.
Why not? The third paragraph of the story indicates that: “Companies closer to becoming fully automated achieve faster turnaround times, deal bandwidth, and creativity of negotiations.” In other words, at that point it could certainly happen that two monopolies would agree to charge each other lower cost if it benefitted both of them. (Unless you’d count that as instance of “charging profit-maximizing costs”?) The concern is that the subprocesses of each company/institution that get good at (or succeed at) bargaining with other institutions are subprocesses that (by virtue of being selected for speed and simplicity) are less aligned with human existence than the original overall company/institution, and that less-aligned subprocess grows to take over the institution, while always taking actions that are “good” for the host institution when viewed as a unilateral move in an uncoordinated game (hence passing as “aligned”).
At this point, my plan is try to consolidate what I think the are main confusions in the comments of this post, into one or more new concepts to form the topic of a new post.
Sounds great! I was thinking myself about setting aside some time to write a summary of this comment section (as I see it).