I don’t agree “AI might be misaligned and agentic and that could be bad for humans” and “maybe the institutions of law / democracy / markets will break down” is sufficient summary. When large part of the debate is about distribution, it becomes important to discuss aligned to what/whom.
Eg in Phil’s accounting, any concentration of capital+intelligence which humans originally set in motion counts as human-aligned. Consider the option of “Ultimate Foundation for Scientific Progress”—a wealthy philanthropist set up an entity governed by AIs, which is aligned to the mission “make scientific progress”, keeps the resulting intellectual property, and reinvests. Because humans would be less reliable guardians, they are out of the loop. Maybe he changes his mind afterwards. From the perspective of humans, the income of the entity is not distributed to humans, or influenced by them. At the same time it is not “misaligned AI” in the mostly commonly used sense.
I’d guess what’s going on is if you use aligned in some extremely broad sense of “AIs taken in total are aligned to the good of collective humanity”, yes, many problems go away (but also likely the problems in the essay). If aligned means some more narrow thing, you can have humans owning ˜0% of capital without any local alignment constrain being violated (btw on the topic I highly recommend Beren’s post on difficulty of indexing AI economy)
Thanks, good point re the alignment constraints being complex here.
Re the Beren post: I agree with the post that the AI agents (/automated companies) involved in creating new businesses will be in a better position to pick up the best investments than entities that lack the access to and knowledge about new startup opportunities.
But it still seems like the AIs that do this could totally be investing on behalf of human principals? You only need a fairly minimal sort of alignment to ensure that the AI actually gives all the money it makes back to the human.
I don’t agree “AI might be misaligned and agentic and that could be bad for humans” and “maybe the institutions of law / democracy / markets will break down” is sufficient summary. When large part of the debate is about distribution, it becomes important to discuss aligned to what/whom.
Eg in Phil’s accounting, any concentration of capital+intelligence which humans originally set in motion counts as human-aligned. Consider the option of “Ultimate Foundation for Scientific Progress”—a wealthy philanthropist set up an entity governed by AIs, which is aligned to the mission “make scientific progress”, keeps the resulting intellectual property, and reinvests. Because humans would be less reliable guardians, they are out of the loop. Maybe he changes his mind afterwards. From the perspective of humans, the income of the entity is not distributed to humans, or influenced by them. At the same time it is not “misaligned AI” in the mostly commonly used sense.
I’d guess what’s going on is if you use aligned in some extremely broad sense of “AIs taken in total are aligned to the good of collective humanity”, yes, many problems go away (but also likely the problems in the essay). If aligned means some more narrow thing, you can have humans owning ˜0% of capital without any local alignment constrain being violated (btw on the topic I highly recommend Beren’s post on difficulty of indexing AI economy)
Thanks, good point re the alignment constraints being complex here.
Re the Beren post: I agree with the post that the AI agents (/automated companies) involved in creating new businesses will be in a better position to pick up the best investments than entities that lack the access to and knowledge about new startup opportunities.
But it still seems like the AIs that do this could totally be investing on behalf of human principals? You only need a fairly minimal sort of alignment to ensure that the AI actually gives all the money it makes back to the human.