To further your argument, Capital in the 21st Century makes the point that it becomes more efficient to generate capital the more capital you have (because many strategies to make more capital benefit from having more starting capital).
In that case, even if we had both 1) Only AI advisors for individual *humans*, and not any larger social entities, 2) All AI advisors are (initially) rate-limited to the same level of intelligence,
We might imagine that the AI advisors of the richest individuals could enact capital-intensive strategies that disproportionately increase their influence over the future, effectively disempowering >99% of humans simply for being poor.
Secondly,
I claim the clearest reason that AI advisors fail is simply that before the transition to a stable AI-driven civilization finishes, it seems almost certain at least one smart-human level AI will escape into the wild and start autonomously evolving, and this is likely to lead to disempowerment. Does this seem right to you? What am I missing?
Re: Humans are Not Alone.
Firstly,
To further your argument, Capital in the 21st Century makes the point that it becomes more efficient to generate capital the more capital you have (because many strategies to make more capital benefit from having more starting capital).
In that case, even if we had both
1) Only AI advisors for individual *humans*, and not any larger social entities,
2) All AI advisors are (initially) rate-limited to the same level of intelligence,
We might imagine that the AI advisors of the richest individuals could enact capital-intensive strategies that disproportionately increase their influence over the future, effectively disempowering >99% of humans simply for being poor.
Secondly,
I claim the clearest reason that AI advisors fail is simply that before the transition to a stable AI-driven civilization finishes, it seems almost certain at least one smart-human level AI will escape into the wild and start autonomously evolving, and this is likely to lead to disempowerment. Does this seem right to you? What am I missing?