My substack: https://tomasbjartur.substack.com/
Fiction:
My Twitter: https://x.com/bjarturtomas
LLM policy: None of my posts or comments are written by, or in collaboration with, an LLM. This includes Beauty and the Beast. Occasionally, I will use an LLM for proofreading/research but all words are my own.
Sorry, when you say
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can’t tell if you’re being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon’s posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you’re not being sarcastic then I suspect we don’t disagree that much.
I was under the impression you considered it more plausible than you do and I was using a rhetorical tool where I slightly cede ground before arguing against what I ceded. Probably unvirtuous on reflection. Still, though I don’t think it’s a likely they would honor stock holders, it is a possible. It does seem very unnatural as conspirators would likely have very different stock allocations and the biggest beneficiaries may well be those outside the conspiracy, a strange sort of altruism.
Anyway, my tweet was not worded particularly well.
Is stock ownership a good Schelling point for the god designers? Maybe more than I was imagining. You’re assuming very fine-grained alignment ability and Anthropic/OAI employees willing to consciously defect against humanity. Why not defect against stockholders, too? It seems an odd Schelling point. Doing this requires a conspiracy to take over the world. The natural Schelling point is surely an equal share to all conspirators, no? And why would the board be involved? Have they been so far in alignment decisions? It will be technical people who understand how things work doing the actual defection.
And even granting you that, this sort of ownership isn’t best characterized in the terms Dwarkesh was characterizing it. It is ownership in a sense, but Dwarkesh was talking about tax policy which is absurd in this scenario. The ownership Dwarkesh and Scott imagine is one embedded in a system of property rights and the existing political economy. The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy. Though one still has to wonder how god will extrapolate your preferences such that “you” can consume a galaxy. [1]
It all comes down to initial preferences and the competitive equilibrium of the AIs in the end; contextualizing it as tax policy still reads as absurd and not giving the future its due. Notions of identity and polity will predictably break down to the point that the frames Scott and Dwarkesh are using look very unlikely to be fruitful, is my main claim. And if you assume the existing political economy remains in some form (which Scott and Dwarkesh do), you should expect a coup at some point that breaks property rights. They do happen from time to time and history will accelerate. They imagine the dynamics that maintain property rights will remain but those that cause their occasional dissolution will not, even in unprecedented time that resembles more a speciation event than any common historical process, and one in which history is happening thousands of times faster[2] than it ever has before
Galaxies are far away, so what does it even mean to control one? It means you get to decide what it will spend its computation computing after your nanobots or whatever convert it into something useful, which is a different sort of wealth. I suppose you might send copies of your brain state there. Brain state? What exactly are “you” again? If you can “solve alignment” and point an AI god at your preferences, you might be able to own a galaxy in some sense. However, a lot of optimization pressure will now be centred on “you” and “your preferences” and what comes out of that might not look very much like you expect owning a galaxy to look like
That is, there will be more history in a given year as the agents involved will be thinking vastly faster
Sam would have to be 99.99999th percentile megalomaniacal—rather than just the already-priced-in 99.99th—to try this crazy thing that could very likely land him in prison.
Would this land him in prison? Would it even be a crime to add “work towards making Sam Altman god emperor” to the spec?
It’s pretty bad.
Writing fiction on Substack being my hobby, I keep tabs on the rise of AI-written fiction. I came across this amusing example, a fiction story called Flagged as 97% AI, which is about a man unjustly accused of using AI to generate his thesis. It read off to me so I put it through Pangram. It flagged as 100% AI generated.
Being neither particularly intelligent, rational, or sociable I likely can’t offer the best advice.
However, if the goal is to feel less hate towards people you consider intellectually inferior to yourself, maybe you’re going about it backwards. Consider moves in the opposite direction. Find an environment where you are vastly outclassed, such that your contributions feel worthless, your efforts to improve yourself pointless, and you maybe even feel some bitter jealousy in your heart.
We have an intense desire to feel superior. Those blessed with intellect should have some noblesse oblige. To despise those who lack your genetic and mimetic gifts lacks grace. It is their very inferiority that provides you with the pleasure of feeling superior. Schopenhauer decries the Malthusian ocean he floated atop of, the people that let him live his life of the mind. What a dick.
Reflections on Inkhaven
Would I recommend applying next year? Yes.
Would I go again? It was a really good experience. I do wonder if it is the type of thing that would never be quite the same the second time. Despite thinking this, I am tempted.
Did it make me a measurably stronger writer?
I don’t think so. I am much prouder of the fiction I wrote before Inkhaven than what I produced during Inkhaven. A counterpoint to this: Ozy, Wales, Linch and Eneasz convinced me I should spend some time editing/revising my fiction—which I haven’t really been doing much. This could very well make me a much stronger fiction writer, but I haven’t really tried it yet. We will see what happens with my next short story.
Am I proud of what I wrote?
I am not sure. My first week, I wrote mostly terrible essays. I then started writing microfiction, and some of it still amuses me. I did get one longer bit of fiction done there, too: Lobsang’s Children. I am pretty happy with it, but it wasn’t a big hit.
These are my favourite of my short posts:
Harry Potter and the Rules of Quidditch
With Adjustable Chef’s Tool Drawer
Most valuable thing about Inkhaven?
Inkhaven is nothing without its people.
Which residents impressed me most?
Linch struck me as the most likely to become a professional blogger. Vishal’s blog was great and he is insanely well read. Though I didn’t really talk to him, Screwtape impressed me. He takes teaching/learning rationality seriously. By revealed preference, this isn’t something I care much about. But it is a noble pursuit and I am glad at least one person is still carrying the torch. I had a brief conversation with GeneSmith and was in awe at how many paths to intelligence augmentation he has considered, including really crazy ideas like methods for expanding adult skulls so as to have more room for brain grafts. He struck me as insanely smart.
Biggest regrets?
I was in a bad mood at the start and psyched myself out of enjoying the first week. I started drinking in an attempt to improve my mood and this was a poor decision. I was consequently rather rude to a few people. Interestingly, my tweets were (for whatever reason) much better than usual during this period.
The snack bar was so tempting, I gained about 10 pounds. I just don’t have willpower in regards to snacks. At home, I eat bland food to stay at the high end of “normal” BMI. I am very annoyed I did this to myself and am considering an extreme fast. I would have paid extra to be forbidden snacks, as stupid as this sounds. Perhaps Screwtape is right about this whole rationality thing.
What is Lighthaven like?
An excerpt from a tweet I wrote about this:
I am not the most visual person, but I can relay impressions. I would say it feels a bit Chestertonian. The Rose Garden Inn as patient zero, infecting various houses around it. The Chestertonian bit, in my imagination, is the implicit mad scheme to slowly undermine various stifling norms of city planning. One house at a time, seeds of freedom spread. I imagine this spirit infecting first the neighborhood and then the city of Berkeley itself. At a key moment, the planning commissioner will be turned and the revolution completed.
There is an artificial lawn. And though you’d think artificial grass would be tacky, it does somehow work. Once again, materials scientists have outdone God. There are “onions,” strange functional sculptures in which one can sit and have conversations. They look to me like large demonic hands transfigured from the artificial turf by some strange wizards, but in a pleasant way. There is a snack-bar, which is perhaps too generous with its offerings. I would recommend hiring Stephan J. Guyenet as a consultant to ruin everyone’s good time.
The aesthetics, then, I will describe as Post-Bureaucratic-Apocalypse chic. Planning laws and an aesthetically dead, inhuman, architectural tradition preventing any new beauty being constructed, those with taste must take the old and put it to new purposes. And this Lighthaven has done.
Favourite Visiting Writers? I am mostly interested in fiction, so talked with Daystar, Ozy and Alexander Wales. They were all great. Ozy’s feedback was particularly good. And they’re just a very pleasant, happy person. Daystar seemed cool and I should have asked for more feedback from him, though I did send him this very personal, autobiographical work.
How well run was it? It was very well run. I have no complaints. Ben Pace did a great job. I expect next year it will be even better.
He stomps on a rubber soul.
So long and good luck!
There is also the immediate, irresistible desire to have sex with yourself and the consequent shame afterwards.
Suppose you’re a billionaire and you want to get married. However, gold-digging people of the gender you prefer target you. They are good enough at faking attraction that you cannot tell. How should you act? One idea I had was this: pick 10000 random people and then select from there. You will at the very least likely remove most of the world-class dissemblers. Armstrong proposes a similar scheme in Siren worlds and the perils of over-optimised search.
Yeah. I prefer not fighting the thought experiment though and thinking of beauty qua beauty. Though perhaps what remains of beauty once it is stripped of all its correlates is perverse.
Tallness is zero sum. But I suspect beauty isn’t. If everyone was more beautiful but the relative differences remained, I think people would be happier. Am I wrong in this? This has policy implications, as once genetic engineering gets better taxing height is likely wise to avoid red-queens races into unhealthy phenotypes. Taxing beauty seems very horrible to me. As beauty is quite beautiful.
Yes. It did a lot of CoT. I don’t recall how long.
I am mostly surprised at the humor being more humory and less humor-like-substancy.
Normative claims describe desired world states. To the extent they are coherent and possible, they cannot be argued against. They are merely a shape the world could be. Descriptive statements make neutral claims about features of the world. Toy examples of each mistake:
“Having no legs is not good; I bet that crippled man has invisible legs.”
“That crippled man has no legs; this must be the way of things and so just and right”
In practice, this tends to be done with themes and processes and not at such a crude, silly level. The stereotypical reactionary might identify selective processes, themselves, as ends. The stereotypical liberal might pretend selective processes don’t exist. The reactionary may be right about his values, and then is not making a mistake here. But in many cases I think they are confusing the descriptive for the normative and do not value what they think they value.
Sorry, I was claiming I was using this rhetorical trick. I saw your “share ownership seems like a fair schelling” comment and felt like I should mention that in my reply. My “maybe more than I was imagining” was just rhetoric on my part.