My substack: https://tomasbjartur.substack.com/
EPUB of all my fiction: https://tomasbjartur.com/The_Origami_Men_and_Other_Stories.epub
Fiction:
My Twitter: https://x.com/bjarturtomas
LLM policy: None of my posts or comments are written by, or in collaboration with, an LLM. This includes Beauty and the Beast. Occasionally, I will use an LLM for proofreading/research but all words are my own.
My own contribution is equally telling: https://tomasbjartur.bearblog.dev/rational-teletubbies/
Better now?
I did copy and paste it from google to get the diacritics. Is it fixed now for you?
Yeah. It’s more something I wanted him to contend with. It isn’t my true objection. I should have fixed the essay after inkhaven.
IMO, it is worth it from a pure PR perspective. Chopping heads off gives people the ick.
Thanks. Fixed.
If anyone interested in my fiction prefers to read on their kindle, I have a website now which grabs my fiction from the LessWrong API and turns it into an ebook: https://tomasbjartur.com/.
It’s also easy to become someone else in the environments where ‘strategic silence’ seems useful. We hold our principles more lightly than we imagine. And they can change without us noticing.
This will convince approximately no one at OpenAI. I have no love for Altman, but even I can see you are being uncharitable. You can just make an honest case for him being an awful person. Some of the rhetoric here, such as his talk of the costs of training humans implying he has no empathy for them, is just not going to be effective for smart people, especially those who know him and give him the benefit of the doubt. He’s clearly just talking in the abstract in a way that isn’t going to offend an OpenAI employee. The bunker stuff, too. They know it won’t save him. He knows it won’t save him. So the rhetoric around that is negatively useful, though effective on a normal person.
I am not sure what an effective open letter of this type would look like, but it would not look so obviously uncharitable in its reading of the man your targets admire.
Sorry, I was claiming I was using this rhetorical trick. I saw your “share ownership seems like a fair schelling” comment and felt like I should mention that in my reply. My “maybe more than I was imagining” was just rhetoric on my part.
Sorry, when you say
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can’t tell if you’re being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon’s posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you’re not being sarcastic then I suspect we don’t disagree that much.
I was under the impression you considered it more plausible than you do and I was using a rhetorical tool where I slightly cede ground before arguing against what I ceded. Probably unvirtuous on reflection. Still, though I don’t think it’s a likely they would honor stock holders, it is a possible. It does seem very unnatural as conspirators would likely have very different stock allocations and the biggest beneficiaries may well be those outside the conspiracy, a strange sort of altruism.
Anyway, my tweet was not worded particularly well.
Is stock ownership a good Schelling point for the god designers? Maybe more than I was imagining. You’re assuming very fine-grained alignment ability and Anthropic/OAI employees willing to consciously defect against humanity. Why not defect against stockholders, too? It seems an odd Schelling point. Doing this requires a conspiracy to take over the world. The natural Schelling point is surely an equal share to all conspirators, no? And why would the board be involved? Have they been so far in alignment decisions? It will be technical people who understand how things work doing the actual defection.
And even granting you that, this sort of ownership isn’t best characterized in the terms Dwarkesh was characterizing it. It is ownership in a sense, but Dwarkesh was talking about tax policy which is absurd in this scenario. The ownership Dwarkesh and Scott imagine is one embedded in a system of property rights and the existing political economy. The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy. Though one still has to wonder how god will extrapolate your preferences such that “you” can consume a galaxy. [1]
It all comes down to initial preferences and the competitive equilibrium of the AIs in the end; contextualizing it as tax policy still reads as absurd and not giving the future its due. Notions of identity and polity will predictably break down to the point that the frames Scott and Dwarkesh are using look very unlikely to be fruitful, is my main claim. And if you assume the existing political economy remains in some form (which Scott and Dwarkesh do), you should expect a coup at some point that breaks property rights. They do happen from time to time and history will accelerate. They imagine the dynamics that maintain property rights will remain but those that cause their occasional dissolution will not, even in unprecedented time that resembles more a speciation event than any common historical process, and one in which history is happening thousands of times faster[2] than it ever has before
Galaxies are far away, so what does it even mean to control one? It means you get to decide what it will spend its computation computing after your nanobots or whatever convert it into something useful, which is a different sort of wealth. I suppose you might send copies of your brain state there. Brain state? What exactly are “you” again? If you can “solve alignment” and point an AI god at your preferences, you might be able to own a galaxy in some sense. However, a lot of optimization pressure will now be centred on “you” and “your preferences” and what comes out of that might not look very much like you expect owning a galaxy to look like
That is, there will be more history in a given year as the agents involved will be thinking vastly faster
Sam would have to be 99.99999th percentile megalomaniacal—rather than just the already-priced-in 99.99th—to try this crazy thing that could very likely land him in prison.
Would this land him in prison? Would it even be a crime to add “work towards making Sam Altman god emperor” to the spec?
It’s pretty bad.
I wrote a bit about Malthus maybe having less teeth than he seems here: https://tomasbjartur.bearblog.dev/contra-my-own-doomerism/
Not particularly well thought out and I may be rehashing things, but I haven’t seen it put this way before.