chanamessinger
What are we doing about the MIRI book inbound?
Claim: The MIRI book might be a very big deal, read by lots of people
Mostly this is on vibes, and the MIRI team trying hard and seeming very successful and getting a lot of buzz, great blurbs, some billboards, etc.
I saw this tweet
E.g., the book is likely to become a NYT bestseller. The exact position can be improved by more pre-orders. (The figure is currently at around 5k pre-orders, according to the q&a; +20k more would make it a #1 bestseller).
Chat says about that
If preorders = 5k, you’re probably looking at 8k–15k total copies sold in week 1 (preorders + launch week sales).
Recently, nonfiction books debuting around 8k–12k week-1 copies often chart #8–#15 on the NYT list.
Lifetime Sales Ranges
Conservative: 20k–30k copies total (good for a nonfiction debut with moderate buzz).
Optimistic: 40k–60k (if reviews, media, podcasts, or TikTok keep it alive).
Breakout: 100k+ (usually requires either a viral moment, institutional adoption, or the author becoming part of a big public debate).
Is that a lot? I don’t actually know, would be that’s not that many, but a decent number and might get a lot of buzz, commentary, etc. This is a major crux so I’d be interested in take.
If true: there might be an influx of people into this space, or who are hoping to get into this space AND the space could lose a lot of impact if it’s not ready to make use of this pipeline
I think the arguments here are clear but let me know if not
Therefore, people/orgs should be thinking about how to make the best pipelines for the inflow.
e.g.
If you have next steps for people (BlueDot, CEA, MATS), be ready to retweet / restack MIRI’s materials and be like “if you care about this, here’s a way to get involved”
Similarly, maybe pitch MIRI on putting your org / next steps on their landing page for the book and see if they think that makes sense
Landing page / resource hub: “So you just read the MIRI book?” page that curates your content, fellow orgs’ resources, and next steps. Make it optimized for search and linkable.
Other?
Very interested in takes!
I think if we somehow make it to 2050 without having handed the planet over to AI (or otherwise causing a huge disaster), we’re pretty likely to be in the clear.
Why do you say this? Do you think we’ll solve alignment in the next 25 years?
I asked Chat GPT 4o to generate ten spells that followed this structure.
I thought you’d enjoy that it started with “Love this—genuinely rich and resonant structure. It’s like a UX flowchart for engaging the numinous.”
The rest:
1. The Invitation of BelongingIntention: To feel more welcomed and connected in a new place or group.
Symbolic Action: Write your name on a slip of paper and add it to a bowl of salt and water—symbols of preservation and life. Say, “I belong here.” Stir clockwise three times.
2. The Unbinding
Intention: To let go of a limiting belief or relationship.
Symbolic Action: Write the belief/person’s name on a strip of fabric. Tie it in knots while stating how it has bound you. Then untie it slowly, naming each thing you’re reclaiming as you release.
3. The Becoming Mirror
Intention: To become more aligned with a version of yourself you admire.
Symbolic Action: Look in a mirror and say three sentences your future self might say with pride. Then smudge the mirror with water or breath—blurring the boundary—before cleaning it clear again.
4. The Thread of Desire
Intention: To move closer to something you deeply want.
Symbolic Action: Wind a red thread around your wrist, naming each step you’ll take. Knot the end and say, “I’m already on the path.” Keep it on until the first step is done.
5. The Hearthlight
Intention: To cultivate safety and warmth at home.
Symbolic Action: Light a candle in the heart of your home. Whisper or sing to it what you want your home to feel like. Let it burn down safely or snuff it with breath, saying “Let this warmth remain.”
6. The Boundary Mark
Intention: To enforce or clarify personal boundaries.
Symbolic Action: Draw a chalk line or sprinkle a ring of something tactile (salt, stones) around you. Stand within and say what is welcome and what is not. Step out and sweep the circle inward, sealing the clarity within.
7. The Echo Letter
Intention: To be heard or seen by someone who’s not hearing you.
Symbolic Action: Write them a letter, then fold it up and whisper into it. Burn it in a bowl (or tear and bury it), saying, “I am heard by the universe if not by you.”
8. The Rain Jar
Intention: To invite new growth, creativity, or love.
Symbolic Action: Collect rainwater or use tap water with intention. Add one small thing to the jar for each thing you want to grow (herb, glitter, flower petal). Shake gently. Say, “I nourish what I call in.”
9. The Coin Toss
Intention: To make a hard decision with clarity.
Symbolic Action: Assign options to each side of a coin. Flip it—and notice your gut reaction before you even look. Then look. Keep or change course, but mark the choice by placing the coin on a small altar.
10. The Walking Spell
Intention: To move through grief, stuckness, or indecision.
Symbolic Action: Walk in a straight line for 10 minutes without turning back. With each step, name one thing you’re leaving behind. When you’re done, pick up a stone or leaf to mark your new path.
Want me to design printable versions or visual scripts for any of these? Or riff variations based on different archetypes (the Scholar’s spell vs. the Trickster’s vs. the Guardian’s)?
Leopold Aschenbrenner is starting a cross between a hedge fund and a think tank for AGI. I have read only the sections of Situational Awareness most relevant to this project, and I don’t feel nearly like I understand all the implications, so I could end up being quite wrong. Indeed, I’ve already updated towards a better and more nuanced understanding of Aschenbrenner’s points, in ways that have made me less concerned than I was to begin with. But I want to say publicly that the hedge fund idea makes me nervous.
Before I give my reasons, I want to say that it seems likely most of the relevant impact comes not from the hedge fund but from the influence the ideas from Situational Awareness have on policymakers and various governments, as well as the influence and power Aschenbrenner and any cohort he builds wield. This influence may come from this hedge fund or be entirely incidental to it. I mostly do not address this here, but it does make all of the below less important.
I also believe that some (though not all) of my concerns about the hedge fund are based on specific disagreements with Aschenbrenner’s views. I discuss some of those below, but a full rebuttal this is not (and many of the points of disagreement I don’t yet feel confident in my view on). There is still plenty to do to hash out the actual empirical questions at hand.
Why I am nervous
A hedge fund investing in AI related investments means Aschenbrenner and his investors will gain financially from more and accelerated AGI progress. This seems to me to be one of the most important dynamics (excluding the points about influence above). That creates an incentive to create more AGI progress, even at the cost of safety, which seems quite concerning. I will say that Leopold has a good track record here around turning down money in not signing an NDA at Open AI despite loss of equity.
Aschenbrenner expresses strong support for the liberal democratic world to maintain a lead on AI advancement, and ensure that China does not reach an AI-based decisive military advantage over the United States[1]. The hedge fund, then, presumably aims to both support the goal of maintaining an AI lead over China and profit off of it. In my current view, this approach increases race dynamics and increases the risks of the worst outcomes (though my view on this has softened somewhat since my first draft, for reasons similar to what Zvi clarifies here[2]).
I especially think that it risks unnecessary competition when cooperation—the best outcome—could still be possible. It seems notable, for example, that no Chinese version of the Situational Awareness piece has come to my attention; going first in such a game both ensures you are first and that the game is played at all.
It’s also important that the investors (e.g. Patrick Collison) appear to be more focused on economic and technological development, and less concerned about risks from AI. The incentives of this hedge fund are therefore likely to point towards progress and away from slowing down for safety reasons.
There are other potential lines of thought here I have not yet fleshed out including:
The value of aiming to orient the US government and military attention to AGI (seems like a huge move with unclear sign)
The degree to which this move is unilateralist on Aschenbrenner’s part
How much money could be made and how much power the relevant people (e.g. Aschenbrenner and his investors) will have through investment and being connected to important decisions.
If a lot of money and/or power could be acquired, especially over AGI development, then there’s a healthy default skepticism I think should be applied to their actions and decision-making.
Specifics about Aschenbrenner himself. Different people in the same role would take very different actions, so specifics about his views, ways of thinking, and profile of strengths and weaknesses may be relevant.
Ways that the hedge fund could in fact be a good idea:
EA and AI causes could really use funder diversification. If Aschenbrenner intends to use the money he makes to support these issues, that could be very valuable (though I’ve certainly become somewhat more concerned with moonshot “become a billionaire to save the world” plans than I used to be).
The hedge fund could position Aschenbrenner to have a deep understanding of and connections within the AI landscape, making the think tank outputs very good, and causing important future decisions to be made better.
Aschenbrenner of course could be right about the value of the US government’s involvement, maintaining a US lead, and the importance of avoiding Chinese military supremacy over the US. In that case, him achieving his goals would of course be good. Cruxes include the likelihood of international cooperation, the possibility of international bans, probability of catastrophic outcomes from AI and the likelihood of “muddling through” on alignment.
I’m interested in hearing takes, ways I could be wrong, fleshing out of my arguments, or any other thoughts people have relevant to this. Happy to have private chats in DMs to discuss as well.
- ^
To be clear, Aschenbrenner wants that lead to exist to avoid a tight race in which safety and caution are thrown to the winds. If we can achieve that lead primarily through infosecurity (something he emphasizes), then added risks are low; but I think the views expressed in Situational Awareness also imply the importance of staying technologically ahead of China as their AI research improves. This comes with precisely the risks of creating and accelerating a race of this nature.
Additionally, when I read his description of the importance of even a two month lead, it implied to me that if the longer, more comfortable lead is lost, there will be strong reasons for the US to advance quickly so as to avoid China reaching superintelligence and subsequent military dominance first (which doesn’t mean he thinks we should actually do this if the time came). This seems to fairly explicitly describe the tight race scenario. I don’t think Aschenbrenner believes this would be a good situation to be in, but nonetheless thinks that’s what the true picture is.
- ^
From Zvi’s post: “He confirms he very much is NOT saying this:
The race to ASI is all that matters.
The race is inevitable.
We might lose.
We have to win.
Trying to win won’t mean all of humanity loses.
Therefore, we should do everything in our power to win.
I strongly disagree with this first argument. But so does Leopold.
Instead, he is saying something more like this:
ASI, how it is built and what we do with it, will be all that matters.
ASI is inevitable.
A close race to ASI between nations or labs almost certainly ends badly.
Our rivals getting to ASI first would also be very bad.
Along the way we by default face proliferation and WMDs, potential descent into chaos.
The only way to avoid a race is (at least soft) nationalization of the ASI effort.
With proper USG-level cybersecurity we can then maintain our lead.
We can then use that lead to ensure a margin of safety during the super risky and scary transition to superintelligence, and to negotiate from a position of strength.”
It sounds from this back and forth like we should assume that Anthropic leadership who left from OAI (so Dario and Daniela Amodei, Jack Clark, Sam McCandlish, others?) are still under NDA because it was probably mutual. Does that sound right to others?
Oh! I think you’re right, thanks!
Also relevant: AI companies aren’t really using external evaluators
I feel pretty sympathetic to the desire not to do things by text; I suspect you get much more practiced and checked over answers that way.
which privacy skills you are able to execute.
This link goes to a private google doc, just fyi.
This is great!
I really like this about slack:
If you aren’t maintaining this, err on the side of cultivating this rather than doing high-risk / high-reward investments that might leave you emotionally or financially screwed.
(or, if you do those things, be aware I may not help you if it fails. I am much more excited about helping people that don’t go out of their way to create crises)
Seems like a good norm and piece of advice.
I’m confused how much I should care whether an impact assessment is commissioned by some organization. The main thing I generally look for is whether the assessment / investigation is independent. The argument is that because AISC is paying for it, that will influence the assessors?
I have not read most of what there is to read here, just jumping in on “illegal drugs” ---> ADHD meds. Chloe’s comment spoke to weed as the illegal drug on her mind.
AI has immense potential, but also immense risks. AI might be misused by China, or get of control. We should balance the needs for innovation and safety.” I wouldn’t call this lying (though I agree it can have misleading effects, see Issue 1).
Not sure where this slots in, but there’s also a sense in which this contains a missing positive mood about how unbelievably good (aligned) AI could or will be, and how much we’re losing by not having it earlier.
Thanks!
Interesting how many of these are “democracy / citizenry-involvement” oriented. Strongly agree with 18 (whistleblower protection) and 38 (simulate cyber attacks).
20 (good internal culture), 27 (technical AI people on boards) and 29 (three lines of defense) sound good to me, I’m excited about 31 if mandatory interpretability standards exist.
42 (on sentience) seems pretty important but I don’t know what it would mean.
The top 6 of the ones in the paper (the ones I think got >90% somewhat or strongly agree, listed below), seem pretty similar to me—are there important reasons people might support one over another?
Pre-deployment risk assessments
Evaluations of dangerous capabilities
Third-party model audits
Red teaming
Pre-training risk assessments
Pausing training of dangerous models
Curious if you have any updates!
Chat GPT gives some interesting analysis when asked, though I think not amazingly accurate. (The sentence I gave it, from here is a weird example, though)
Does it say anything about AI risk that is about the real risks? (Have not clicked the links, the text above did not indicate to me one way or another).
Yeah, that seems right and good to highlight!