An important part of the property story is that it smuggles in the assumption of intent-alignment to shareholders into the discussion. IE, the AI’s original developers or the government executives that are running the project adjust the model spec in such a way that it alignment is “do what my owners want”, where owners are anyone who owned a share in the AI company.
I find it somewhat plausible that we get intent alignment. [1] But I think the transmutation from “the board of directors/engineers who actually write the model spec are in control” to “voting rights over model values are distributed by stock ownership” is basically nonsense, because most of those shareholders will have no direct way to influence the AIs values during the takeoff period. What property rights do exist would be at the discretion of those influential executives, as well as managed by differences in hard power if there’s a multipolar scenario (ex: US/Chinese division of the lightcone).
--
As a sidenote, Tim Underwood’s The Accord is a well written look at what the literal consequences of locking in our contemporary property rights for the rest of time might look like.
It makes sense to expect the groups bankrolling AI development to prefer an AI that’s aligned to their own interests, rather than humanity at large. On the other hand, it might be the case that intent alignment is harder/less robust than deontological alignment, at which point you’d expect most moral systems to forbid galactic-level inequality.
An important part of the property story is that it smuggles in the assumption of intent-alignment to shareholders into the discussion. IE, the AI’s original developers or the government executives that are running the project adjust the model spec in such a way that it alignment is “do what my owners want”, where owners are anyone who owned a share in the AI company.
I find it somewhat plausible that we get intent alignment. [1] But I think the transmutation from “the board of directors/engineers who actually write the model spec are in control” to “voting rights over model values are distributed by stock ownership” is basically nonsense, because most of those shareholders will have no direct way to influence the AIs values during the takeoff period. What property rights do exist would be at the discretion of those influential executives, as well as managed by differences in hard power if there’s a multipolar scenario (ex: US/Chinese division of the lightcone).
--
As a sidenote, Tim Underwood’s The Accord is a well written look at what the literal consequences of locking in our contemporary property rights for the rest of time might look like.
It makes sense to expect the groups bankrolling AI development to prefer an AI that’s aligned to their own interests, rather than humanity at large. On the other hand, it might be the case that intent alignment is harder/less robust than deontological alignment, at which point you’d expect most moral systems to forbid galactic-level inequality.