I agree that you won’t get property rights if the ASI doesn’t wanna respect property rights. And I agree that if the ASI is either
Misaligned
Kind of CEV aligned, or aligned to the value handshake of a group of people
Corrigible / Aligned to follow the instructions of a single person
The ASI won’t care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.
But I don’t think its that strange to imagine the ASI aligned to follow the instructions of a group of people, and that the way those people “divide up” the ASI is using something like property rights. Like Tomas Bjartur wrote this on twitter, which is very similar to your post
Both Dwarkesh and his critics imagine an absurd world. Think of the future they argue about. Political economy doesn’t change rapidly, even as the speed of history increases, in the literal sense that the speed of thought of the actors who produce history will be thousands of times faster, not to mention way smarter. These are agents to whom we seem like toddlers walking in slow motion. It is complete insanity to expect your OAI stock certificates to be worth anything in this world, even if it is compatible with human survival.
So many can’t contend with the scope of what they project. They can’t hold in their mind that things are allowed to be DIFFERENT and so we get bizarre arguments about nonsense. Own a galaxy? What does this mean for a human to own a galaxy in an economy operated by minds running thousands to millions of times faster than ours? Children? What sort of children, Dwarkesh? Copies of your brain state? Are you even allowing yourself to think with the level of bizarreness required? Because emulations are table stakes, and even they will be economically obsolete curiosities by the time they’re created. Things will be much weirder than we can possibly comprehend. How often have property rights been reset throughout history? How quickly will history move in the transition period? Why shouldn’t it trample on your stock certificates, if not the air you breathe? But institutions are surprisingly robust? Maybe they are. How long have they existed in their current form? How fast will history be moving exactly, again?
Suppose OAI aligns AI, whatever the fuck that means. Will it serve the interests of the USG? The CCP? Will they align it to humanity weighted by wealth, to OAI stockholders, to Sama, to the coterie of engineers (who may well be AIs) who actually know wtf is going on, to the coding agent who implements it? Tax policy? Truly the important question.
What does it mean to be a human principal in this world? How robust are these institutions? How secure is a human mind? Extremely insecure given how easy humans are to scam. There is going to be a lot of incentive to break your mind if you own, checks notes, a whole galaxy? Oh? You will have a lil AI nanny to defend you? Wow. Isn’t that nice? Please return to the beginning of this paragraph. A human owning galaxies? That’s bad space opera. Treat the future with the respect it deserves. This scenario is not even close to science-fictional enough to happen.
And I responded by saying:
Doesn’t seem that hard to imagine for me. What do you find so implausible with a story going something like this?
group X builds ASI (through medium-fast recursive self-improvement from something like current systems)
Before this, they figured out alignment. Meaning: ~they can create ASIs that have the goals they want, while avoiding the catastrophes any sane person would expect to follow from that premise
The people inside group X with the power to say what goals should be put into the baby ASI, maybe the board, tells it something like “do what we say”.
They tell it to do all the things that stabilize their position. Sabotage runner-up labs, make sure the government doesn’t mess their stuff up, make sure other countries aren’t worried, or just ask the ASI to do stuff that cements their position depending on how the ASI alignment/corrigibility works exactly.
They now quickly control the whole world. And can do whatever sci-fi stuff they want to do.
The group in control of the ASI have value disagreements about what should be done with the world. They negotiate a little bit, figure out the best solution is something like, split everything (the universe) radially, and make some rules (maybe people can’t build torture sims in their universe slice). Enforce this by making the next generation of ASIs aligned to “listen to person [x,y,z] in their slice, don’t build torture sims, don’t allow yourself to be modified into something that builds torture sims, don’t mess with others in their pizza slice” etc etc. The original ASI can help them with the formulation.
This would give some people ownership of galaxies. I don’t see any issue posed by the ASI thinking super quickly. You kind of answer by saying the “AI nanny” idea is absurd. But the argument you present is ‘return to the beginning of this paragraph’, which reads
“What does it mean to be a human principal in this world? How robust are these institutions? How secure is a human mind? Extremely insecure given how easy humans are to scam. There is going to be a lot of incentive to break your mind if you own, checks notes, a whole galaxy?”
But like, these would not be concerns in the story I laid out right?
--- I mean to be clear 1) This is not how I expect the future to go, but I’d assign more than 1% to it. I don’t think its ridiculous. 2) I realize this notion of “ownership” is somewhat different from whats laid out in the essay. Which is fair, but theres a slightly different class of stories that seem maybe half as probably where more people end up with ownership / stake in the ASI’s value function.
Is stock ownership a good Schelling point for the god designers? Maybe more than I was imagining. You’re assuming very fine-grained alignment ability and Anthropic/OAI employees willing to consciously defect against humanity. Why not defect against stockholders, too? It seems an odd Schelling point. Doing this requires a conspiracy to take over the world. The natural Schelling point is surely an equal share to all conspirators, no? And why would the board be involved? Have they been so far in alignment decisions? It will be technical people who understand how things work doing the actual defection.
And even granting you that, this sort of ownership isn’t best characterized in the terms Dwarkesh was characterizing it. It is ownership in a sense, but Dwarkesh was talking about tax policy which is absurd in this scenario. The ownership Dwarkesh and Scott imagine is one embedded in a system of property rights and the existing political economy. The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy. Though one still has to wonder how god will extrapolate your preferences such that “you” can consume a galaxy. [1]
It all comes down to initial preferences and the competitive equilibrium of the AIs in the end; contextualizing it as tax policy still reads as absurd and not giving the future its due. Notions of identity and polity will predictably break down to the point that the frames Scott and Dwarkesh are using look very unlikely to be fruitful, is my main claim. And if you assume the existing political economy remains in some form (which Scott and Dwarkesh do), you should expect a coup at some point that breaks property rights. They do happen from time to time and history will accelerate. They imagine the dynamics that maintain property rights will remain but those that cause their occasional dissolution will not, even in unprecedented time that resembles more a speciation event than any common historical process, and one in which history is happening thousands of times faster[2] than it ever has before
Galaxies are far away, so what does it even mean to control one? It means you get to decide what it will spend its computation computing after your nanobots or whatever convert it into something useful, which is a different sort of wealth. I suppose you might send copies of your brain state there. Brain state? What exactly are “you” again? If you can “solve alignment” and point an AI god at your preferences, you might be able to own a galaxy in some sense. However, a lot of optimization pressure will now be centred on “you” and “your preferences” and what comes out of that might not look very much like you expect owning a galaxy to look like
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can’t tell if you’re being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon’s posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you’re not being sarcastic then I suspect we don’t disagree that much.
Wrt the first paragraph, I’m explicitly assuming away those things. 20 years after ASI I’d rank outcomes in descending (as it appears scrolling downards on page) order of probability something like this
AI kills everyone
AI is just running around freely optimizing the galaxy for the CEV of humanity, or some more limited group of people, or if not the true cev, still some form of values that express themselves differently from the intent alignment required for the above story
Maybe equally probable as (3), but ASI not killing everyone, but still disempowering them in some sense, yielding various bad to okay ish outcomes, many short great short stories about this, eg makers of mind, trojan sky, origami men.
But then after this, theres many % left in the probability space. I don’t know 20%. And in this space I’m assuming the ASI is not misaligned, and for that we have to have pretty good alignment abilities. Just we don’t immediately point them in the CEV direction. That’s not a crazy assumption, or do you disagree? I’m not saying thats a prediction for what obviously would’ve happened, just obviously false. I’d give it maybe 60% chance.
Then something like, in this situation the board is ASI-pilled, and decides they’re responsible for what the ASI will do, and decides to take a hands on approach wrt the alignment procedures the lab is using. I’m imagining like the slowdown ending in AI2027 where they say something like
The Oversight Committee formalizes that power structure. They set up a process for approving changes to the Spec, requiring sign-off from the full Oversight Committee, which now includes five to ten tech executives (from OpenBrain and its now-merged competitors) and five to ten government officials (including the President).18 Also, the Spec now emphasizes that AIs shouldn’t assist with any unapproved attempts to change future AIs’ goals. They also set up a simple measure designed to prevent committee members from getting superintelligent assistance in plotting against other members: the logs of all model interactions are viewable by all members of the Oversight Committee, their staff, and their AI assistants.19
Except the government officials are not on the oversight committee.
Then the committee members either decide
Make intent aligned ASI and use it ourselves, and do something like the radial split proposed above
Make intent aligned ASI and do the radial split over shareholders.
Why would they do this? One reason is that thats closer to what they’re legally supposed to do, right? Another is, they were voted in by shareholders, maybe they feel some allegiance to them? A third is, maybe they split ownership amongst the board in proportion to how many shares they have, that seems fair, and then locking out people because they’re not present in the room seems immediately unfair?
I think (1) is more likely if I had to bet, and it gets you the type of ownership I described. (2) is somewhat less likely, but not like 1% (conditional on getting here, in absolute terms it probably is around 1-2%), and it’d get you something closer to how we currently think of ownership.
I would also note that, just to be clear:
The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy.
Is not really how I’d characterize it. I described a system where people have areas of space where they are free to do whatever they want with the space and objects therein. Then there are some constraints like no torture sims and no initialize vacuum decay of universe. I don’t think this is so hard to implement, and is fairly natural.
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can’t tell if you’re being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon’s posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you’re not being sarcastic then I suspect we don’t disagree that much.
I was under the impression you considered it more plausible than you do and I was using a rhetorical tool where I slightly cede ground before arguing against what I ceded. Probably unvirtuous on reflection. Still, though I don’t think it’s a likely they would honor stock holders, it is a possible. It does seem very unnatural as conspirators would likely have very different stock allocations and the biggest beneficiaries may well be those outside the conspiracy, a strange sort of altruism.
Anyway, my tweet was not worded particularly well.
Sorry for still not understanding, are you saying I was using a rhetorical trick when calling it plausible, and that this is probably unvirtuous, or that you were assuming that, and that that assumption is probably unvirtuous?
I think I made it quite clear what I meant by plausible, by saying “And I agree that if the ASI is either (1,2,3) the ASI won’t care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.” in the beginning of my post.
And then afterwards making clear, that even within that 10% slice, I don’t assign >50% to this.
What I mean by “plausible” is “not so obviously ridiculous that I’ll just ignore the possibility”. Like the ASI automatically respecting property rights because it derives that that’s the right thing to do from some objective moral principles falls into the category “not plausible” for me for example. I think its ruled out apriori by several strong theoretical arguments. So I put the probability very low, not 1%, but like 1e-6. Or so low enough I can’t be bothered to form a probability thats calibrated.
Sorry, I was claiming I was using this rhetorical trick. I saw your “share ownership seems like a fair schelling” comment and felt like I should mention that in my reply. My “maybe more than I was imagining” was just rhetoric on my part.
To keep it short, I don’t think the story you present would likely mean that AI stock would be worth galaxies, but rather that the inner circle has control. Part of my writing (one of the pictures) is on that possibility. This inner circle would probably have to be very small or just 1 person such that nobody just quickly uses the intent-aligned ASI to get rid of the others. However, I still feel like debating future inequality in galaxy-distribution based on on current AI stock ownership is silly.
I take a bit of issue with saying that this is very similar to what Bjartur wrote, so much apparently that you don’t even need to write a response to my post but you can just copypaste your response to him. I read that post once like a week ago and don’t think the two posts are very similar, even though they are on the same topic with similar (obvious) conclusions. (I know Bjartur personally, I’d be very surprised he takes issue with me writing on the same topic)
First of all, I didn’t mean to insinuate that your posts are too similar, or that he’d take issue with you writing the post, or anything like that. I just, started writing up my response, and realized I was about to write the exact same thing I wrote in response to the Bjartur post, so copied it instead, and wouldn’t feel comfortable doing that without alerting people that’s what I was doing.
Now, I don’t think your response addresses my reply very well. Like, I feel your response is already addressed by my original response. Like when you say
, I don’t think the story you present would likely mean that AI stock would be worth galaxies, but rather that the inner circle has control
But like, the specific way of exercising that control was to split up the ASI is using something like property rights. like in point 6)
The group in control of the ASI have value disagreements about what should be done with the world. They negotiate a little bit, figure out the best solution is something like, split everything (the universe) radially, and make some rules (maybe people can’t build torture sims in their universe slice..
And like:
his inner circle would probably have to be very small or just 1 person such that nobody just quickly uses the intent-aligned ASI to get rid of the others.
is also addressed immediately after by
Enforce this by making the next generation of ASIs aligned to “listen to person [x,y,z] in their slice, don’t build torture sims, don’t allow yourself to be modified into something that builds torture sims, don’t mess with others in their pizza slice” etc etc. The original ASI can help them with the formulation.
Like the thing that was most similar by your and bjartur posts were acting exasperated and saying people lack imagination and are failing to grasp how different things could be. But I feel like you’re the one doing that, failing to imagine specific scenarios.
However, I still feel like debating future inequality in galaxy-distribution based on on current AI stock ownership is silly.
Well, I don’t. Interested to hear your argument. Like share ownership seems like a fair schelling point for the radial split described in the 1-6 story. (quick edit: I should not that this model of ownership, specifically based on owning current stocks, is less plausible than the already quite low probability story I wrote, but I still don’t think its obviously ridiculous. like there are not that many steps. 1) people on the board feel accountable to the shareholders and then 2) just do the splitting thing)
I think you are debating for something different than what I am attacking. You are defending the unlikely possibility that people align AI to a small group of people and they somehow share stuff with each other and use something akin to property rights. I guess this is a small variation of the thing i mention in the cartoon, where the CEO has all the power, perhaps it’s the CEO and a board member he likes. But still doesn’t really justify thinking that current property distributions will determine how many galaxies you’ll get and that we shall focus on this question now.
Like the thing that was most similar by your and bjartur posts were acting exasperated
This post is not designed to super carefully examine every argument I can think of, it’s certainly a bit polemic. It’s intended because I think the “owning galaxies for AI stock” thing is really dumb.
Not really, or, I think my story as I told it gets you to “Owning Galaxies”, but does not get you to all the way to “OpenAI shares entitle you to galaxies”.
But you don’t have to make much modification to get there. Or any really, just fill in a detail. Like I said in my previous comment, board of directors using ownership as a schelling point for divying up the gains. Not that far fetched. Do you disagree?
This post is not designed to super carefully examine every argument I can think of, it’s certainly a bit polemic. It’s intended because I think the “owning galaxies for AI stock” thing is really dumb.
I agree that you won’t get property rights if the ASI doesn’t wanna respect property rights. And I agree that if the ASI is either
Misaligned
Kind of CEV aligned, or aligned to the value handshake of a group of people
Corrigible / Aligned to follow the instructions of a single person
The ASI won’t care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.
But I don’t think its that strange to imagine the ASI aligned to follow the instructions of a group of people, and that the way those people “divide up” the ASI is using something like property rights. Like Tomas Bjartur wrote this on twitter, which is very similar to your post
And I responded by saying:
Is stock ownership a good Schelling point for the god designers? Maybe more than I was imagining. You’re assuming very fine-grained alignment ability and Anthropic/OAI employees willing to consciously defect against humanity. Why not defect against stockholders, too? It seems an odd Schelling point. Doing this requires a conspiracy to take over the world. The natural Schelling point is surely an equal share to all conspirators, no? And why would the board be involved? Have they been so far in alignment decisions? It will be technical people who understand how things work doing the actual defection.
And even granting you that, this sort of ownership isn’t best characterized in the terms Dwarkesh was characterizing it. It is ownership in a sense, but Dwarkesh was talking about tax policy which is absurd in this scenario. The ownership Dwarkesh and Scott imagine is one embedded in a system of property rights and the existing political economy. The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy. Though one still has to wonder how god will extrapolate your preferences such that “you” can consume a galaxy. [1]
It all comes down to initial preferences and the competitive equilibrium of the AIs in the end; contextualizing it as tax policy still reads as absurd and not giving the future its due. Notions of identity and polity will predictably break down to the point that the frames Scott and Dwarkesh are using look very unlikely to be fruitful, is my main claim. And if you assume the existing political economy remains in some form (which Scott and Dwarkesh do), you should expect a coup at some point that breaks property rights. They do happen from time to time and history will accelerate. They imagine the dynamics that maintain property rights will remain but those that cause their occasional dissolution will not, even in unprecedented time that resembles more a speciation event than any common historical process, and one in which history is happening thousands of times faster[2] than it ever has before
Galaxies are far away, so what does it even mean to control one? It means you get to decide what it will spend its computation computing after your nanobots or whatever convert it into something useful, which is a different sort of wealth. I suppose you might send copies of your brain state there. Brain state? What exactly are “you” again? If you can “solve alignment” and point an AI god at your preferences, you might be able to own a galaxy in some sense. However, a lot of optimization pressure will now be centred on “you” and “your preferences” and what comes out of that might not look very much like you expect owning a galaxy to look like
That is, there will be more history in a given year as the agents involved will be thinking vastly faster
Sorry, when you say
I genuinely can’t tell if you’re being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon’s posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you’re not being sarcastic then I suspect we don’t disagree that much.
Wrt the first paragraph, I’m explicitly assuming away those things. 20 years after ASI I’d rank outcomes in descending (as it appears scrolling downards on page) order of probability something like this
AI kills everyone
AI is just running around freely optimizing the galaxy for the CEV of humanity, or some more limited group of people, or if not the true cev, still some form of values that express themselves differently from the intent alignment required for the above story
Single person AI dictator like this this ai-futures story
Maybe equally probable as (3), but ASI not killing everyone, but still disempowering them in some sense, yielding various bad to okay ish outcomes, many short great short stories about this, eg makers of mind, trojan sky, origami men.
But then after this, theres many % left in the probability space. I don’t know 20%. And in this space I’m assuming the ASI is not misaligned, and for that we have to have pretty good alignment abilities. Just we don’t immediately point them in the CEV direction. That’s not a crazy assumption, or do you disagree? I’m not saying thats a prediction for what obviously would’ve happened, just obviously false. I’d give it maybe 60% chance.
Then something like, in this situation the board is ASI-pilled, and decides they’re responsible for what the ASI will do, and decides to take a hands on approach wrt the alignment procedures the lab is using. I’m imagining like the slowdown ending in AI2027 where they say something like
Except the government officials are not on the oversight committee.
Then the committee members either decide
Make intent aligned ASI and use it ourselves, and do something like the radial split proposed above
Make intent aligned ASI and do the radial split over shareholders.
Why would they do this? One reason is that thats closer to what they’re legally supposed to do, right? Another is, they were voted in by shareholders, maybe they feel some allegiance to them? A third is, maybe they split ownership amongst the board in proportion to how many shares they have, that seems fair, and then locking out people because they’re not present in the room seems immediately unfair?
I think (1) is more likely if I had to bet, and it gets you the type of ownership I described. (2) is somewhat less likely, but not like 1% (conditional on getting here, in absolute terms it probably is around 1-2%), and it’d get you something closer to how we currently think of ownership.
I would also note that, just to be clear:
Is not really how I’d characterize it. I described a system where people have areas of space where they are free to do whatever they want with the space and objects therein. Then there are some constraints like no torture sims and no initialize vacuum decay of universe. I don’t think this is so hard to implement, and is fairly natural.
I was under the impression you considered it more plausible than you do and I was using a rhetorical tool where I slightly cede ground before arguing against what I ceded. Probably unvirtuous on reflection. Still, though I don’t think it’s a likely they would honor stock holders, it is a possible. It does seem very unnatural as conspirators would likely have very different stock allocations and the biggest beneficiaries may well be those outside the conspiracy, a strange sort of altruism.
Anyway, my tweet was not worded particularly well.
Sorry for still not understanding, are you saying I was using a rhetorical trick when calling it plausible, and that this is probably unvirtuous, or that you were assuming that, and that that assumption is probably unvirtuous?
I think I made it quite clear what I meant by plausible, by saying “And I agree that if the ASI is either (1,2,3) the ASI won’t care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.” in the beginning of my post.
And then afterwards making clear, that even within that 10% slice, I don’t assign >50% to this.
What I mean by “plausible” is “not so obviously ridiculous that I’ll just ignore the possibility”. Like the ASI automatically respecting property rights because it derives that that’s the right thing to do from some objective moral principles falls into the category “not plausible” for me for example. I think its ruled out apriori by several strong theoretical arguments. So I put the probability very low, not 1%, but like 1e-6. Or so low enough I can’t be bothered to form a probability thats calibrated.
Sorry, I was claiming I was using this rhetorical trick. I saw your “share ownership seems like a fair schelling” comment and felt like I should mention that in my reply. My “maybe more than I was imagining” was just rhetoric on my part.
Thanks for clarifying.
Is this a rhetorical trick?
To keep it short, I don’t think the story you present would likely mean that AI stock would be worth galaxies, but rather that the inner circle has control. Part of my writing (one of the pictures) is on that possibility. This inner circle would probably have to be very small or just 1 person such that nobody just quickly uses the intent-aligned ASI to get rid of the others. However, I still feel like debating future inequality in galaxy-distribution based on on current AI stock ownership is silly.
I take a bit of issue with saying that this is very similar to what Bjartur wrote, so much apparently that you don’t even need to write a response to my post but you can just copypaste your response to him. I read that post once like a week ago and don’t think the two posts are very similar, even though they are on the same topic with similar (obvious) conclusions. (I know Bjartur personally, I’d be very surprised he takes issue with me writing on the same topic)
First of all, I didn’t mean to insinuate that your posts are too similar, or that he’d take issue with you writing the post, or anything like that. I just, started writing up my response, and realized I was about to write the exact same thing I wrote in response to the Bjartur post, so copied it instead, and wouldn’t feel comfortable doing that without alerting people that’s what I was doing.
Now, I don’t think your response addresses my reply very well. Like, I feel your response is already addressed by my original response. Like when you say
But like, the specific way of exercising that control was to split up the ASI is using something like property rights. like in point 6)
And like:
is also addressed immediately after by
Like the thing that was most similar by your and bjartur posts were acting exasperated and saying people lack imagination and are failing to grasp how different things could be. But I feel like you’re the one doing that, failing to imagine specific scenarios.
Well, I don’t. Interested to hear your argument. Like share ownership seems like a fair schelling point for the radial split described in the 1-6 story. (quick edit: I should not that this model of ownership, specifically based on owning current stocks, is less plausible than the already quite low probability story I wrote, but I still don’t think its obviously ridiculous. like there are not that many steps. 1) people on the board feel accountable to the shareholders and then 2) just do the splitting thing)
I think you are debating for something different than what I am attacking. You are defending the unlikely possibility that people align AI to a small group of people and they somehow share stuff with each other and use something akin to property rights. I guess this is a small variation of the thing i mention in the cartoon, where the CEO has all the power, perhaps it’s the CEO and a board member he likes. But still doesn’t really justify thinking that current property distributions will determine how many galaxies you’ll get and that we shall focus on this question now.
This post is not designed to super carefully examine every argument I can think of, it’s certainly a bit polemic. It’s intended because I think the “owning galaxies for AI stock” thing is really dumb.
Not really, or, I think my story as I told it gets you to “Owning Galaxies”, but does not get you to all the way to “OpenAI shares entitle you to galaxies”.
But you don’t have to make much modification to get there. Or any really, just fill in a detail. Like I said in my previous comment, board of directors using ownership as a schelling point for divying up the gains. Not that far fetched. Do you disagree?
Well, I don’t really like that. But fair enough.