Yes, that’s right, though I can see how it’s confusing based on the economics literature. Any suggestions for an alternative phrase? I was considering “communication costs”, but there could also be costs from the fact that different parts have different competencies.
This is longer, but maybe “coordination costs that are unrelated to value differences”?
It’s not clear to me that principal-agent costs are more important than the ones I’m talking about here. My experience of working in large companies is not that I was misaligned with the company, it was that the company’s “plan” (to the extent that one existed) was extremely large and complex and not something I could easily understand. It could be that this is actually the most efficient way to work even with intent-aligned agents, since communicating the full plan could involve very large communication costs.
If companies had fully aligned workers and managers, they could adopt what Robin Hanson calls the “divisions” model where each division works just like a separate company except that there is an overall CEO that “looks for rare chances to gain value by coordinating division activities” (such as, in my view, internally charge each other efficient prices instead of profit-maximizing prices), so you’d still gain efficiency as companies merge or get bigger through organic growth. In other words, coordination costs that are unrelated to value differences won’t stop a single AGI controlling all resources from being the most efficient way to organize an economy.
While searching for that post, I also came across Firm Inefficiency which like Moral Mazes (but much more concisely) lists many inefficiencies that seem all or mostly related to value differences.
Seems reasonable, though I don’t think it is arguing against the main arguments in favor of CAIS (which to me are that CAIS seems more technically feasible than AGI).
I think it’s at least one of the main arguments that Eric Drexler makes, since he wrote this in his abstract:
Perhaps surprisingly, strongly self-modifying agents lose their instrumental value even as their
implementation becomes more accessible, while the likely context for
the emergence of such agents becomes a world already in possession of
general superintelligent-level capabilities.
(My argument says that a strongly self-modifying agent will improve faster than a self-improving ecosystem of CAIS with access to the same resources, because the former won’t suffer from principal-agent costs while researching how to self-improve.)
I don’t see how this suggests that our existing institutions to prevent centralization of power will go away, since even now monopolies could merge, often want to merge, but are prevented by law from doing so. (Though I’m not very confident in this claim, I’m mostly parroting back things I’ve heard.)
Yeah I’m not very familiar with this either, but my understanding is that such mergers are only illegal if the effect “may be substantially to lessen competition” or “tend to create a monopoly”, which technically (it seems to me) isn’t the case when existing monopolies in different industries merge.
If companies had fully aligned workers and managers, they could adopt what Robin Hanson calls the “divisions” model where each division works just like a separate company except that there is an overall CEO that “looks for rare chances to gain value by coordinating division activities”
Once you switch to the “divisions” model your divisions are no longer competing with other firms, and all the divisions live or die as a group. So you’re giving up the optimization that you could get via observing which companies succeed / fail at division-level tasks. I’m not sure how big this effect is, though I’d guess it’s small.
While searching for that post, I also came across Firm Inefficiency which like Moral Mazes (but much more concisely) lists many inefficiencies that seem all or mostly related to value differences.
Yeah, I’m more convinced now that principal-agent issues are significantly larger than other issues.
I think it’s at least one of the main arguments that Eric Drexler makes, since he wrote this in his abstract
Yeah, I agree it’s an argument against that argument from Eric. I forgot that Eric makes that point (mainly because I have never been very convinced by it)
Yeah I’m not very familiar with this either, but my understanding is that such mergers are only illegal if the effect “may be substantially to lessen competition” or “tend to create a monopoly”, which technically (it seems to me) isn’t the case when existing monopolies in different industries merge.
My guess would be that the spirit of the law would apply, and that would be enough, but really I’d want to ask a social scientist or lawyer.
Once you switch to the “divisions” model your divisions are no longer competing with other firms, and all the divisions live or die as a group.
Why? Each division can still have separate profit-loss accounting, so you can decide to shut one down if it starts making losses, and the benefits of having that division to the rest of the company doesn’t outweigh the losses. The latter may be somewhat tricky to judge though. Perhaps that’s what you meant?
Yeah, I’m more convinced now that principal-agent issues are significantly larger than other issues.
I should perhaps mention that I still have some uncertainty about this, mainly because Robin Hanson said “There are many other factors that influence coordination, after all; even perfect value matching is consistent with quite poor coordination.” But I haven’t been able to find any place where he wrote down what those other factors are, nor did he answer when I asked him about it.
Why? Each division can still have separate profit-loss accounting, so you can decide to shut one down if it starts making losses, and the benefits of having that division to the rest of the company doesn’t outweigh the losses. The latter may be somewhat tricky to judge though. Perhaps that’s what you meant?
That’s a good point. I was imagining that each division ends up becoming a monopoly in its particular area due to the benefits of within-firm coordination, which means that even if the division is inefficient there isn’t an alternative that the firm can go with. But that was an assumption, and I’m not sure it would actually hold.
This is longer, but maybe “coordination costs that are unrelated to value differences”?
If companies had fully aligned workers and managers, they could adopt what Robin Hanson calls the “divisions” model where each division works just like a separate company except that there is an overall CEO that “looks for rare chances to gain value by coordinating division activities” (such as, in my view, internally charge each other efficient prices instead of profit-maximizing prices), so you’d still gain efficiency as companies merge or get bigger through organic growth. In other words, coordination costs that are unrelated to value differences won’t stop a single AGI controlling all resources from being the most efficient way to organize an economy.
While searching for that post, I also came across Firm Inefficiency which like Moral Mazes (but much more concisely) lists many inefficiencies that seem all or mostly related to value differences.
I think it’s at least one of the main arguments that Eric Drexler makes, since he wrote this in his abstract:
(My argument says that a strongly self-modifying agent will improve faster than a self-improving ecosystem of CAIS with access to the same resources, because the former won’t suffer from principal-agent costs while researching how to self-improve.)
Yeah I’m not very familiar with this either, but my understanding is that such mergers are only illegal if the effect “may be substantially to lessen competition” or “tend to create a monopoly”, which technically (it seems to me) isn’t the case when existing monopolies in different industries merge.
Once you switch to the “divisions” model your divisions are no longer competing with other firms, and all the divisions live or die as a group. So you’re giving up the optimization that you could get via observing which companies succeed / fail at division-level tasks. I’m not sure how big this effect is, though I’d guess it’s small.
Yeah, I’m more convinced now that principal-agent issues are significantly larger than other issues.
Yeah, I agree it’s an argument against that argument from Eric. I forgot that Eric makes that point (mainly because I have never been very convinced by it)
My guess would be that the spirit of the law would apply, and that would be enough, but really I’d want to ask a social scientist or lawyer.
Why? Each division can still have separate profit-loss accounting, so you can decide to shut one down if it starts making losses, and the benefits of having that division to the rest of the company doesn’t outweigh the losses. The latter may be somewhat tricky to judge though. Perhaps that’s what you meant?
I should perhaps mention that I still have some uncertainty about this, mainly because Robin Hanson said “There are many other factors that influence coordination, after all; even perfect value matching is consistent with quite poor coordination.” But I haven’t been able to find any place where he wrote down what those other factors are, nor did he answer when I asked him about it.
That’s a good point. I was imagining that each division ends up becoming a monopoly in its particular area due to the benefits of within-firm coordination, which means that even if the division is inefficient there isn’t an alternative that the firm can go with. But that was an assumption, and I’m not sure it would actually hold.