I work at the Alignment Research Center (ARC). I write a blog on stuff I’m interested in (such as math, philosophy, puzzles, statistics, and elections): https://ericneyman.wordpress.com/
Eric Neyman
Karma: 5,332
I’m not sure if this is the answer you’re looking for, but: most things that could exist don’t. The space of ideas is wide, and few of them are implemented in practice. Is this idea particularly privileged in the space of possible governance ideas, in such a way where you would have expected it to have been tried?
A couple of other things that stand out to me as particularly egregious:
My understanding is that Trump is far more corrupt than past presidents (including Trump in his first term). An example of this is allowing export controls of Nvidia’s AI chips to China in exchange for gifts from Jensen Huang.
The Trump administration has launched criminal investigations against political opponents at an unprecedented rate, most recently against Jerome Powell yesterday.
And of course, the fake electors plot to steal the 2020 presidential election (not to be confused with January 6th—I think his conduct on January 6th was really bad, but the fake electors plot is a much greater indictment of Trump’s character and much stronger evidence of his authoritarianism).
We finally did it, we found the median voter!
I think that by far the most important thing in this space is for a Democrat to win the 2028 presidential election. And I think the most important thing for making that happen is to nominate a Democrat whose positions on the issues are relatively close to the median voter.
We can get a sense of this by seeing how much potential Democratic candidates outperformed fundamentals (i.e. what you would have predicted given the state they were running in and the political environment that year). Some candidates who have done well on this metric include:
Andy Beshear (governor of Kentucky, a really red state)
Josh Shapiro (governor of Pennsylvania, a swing state, where he won his election by a large margin)
Amy Klobuchar (senator from Minnesota)
Ruben Gallego (senator from Arizona)
Mark Kelly (senator from Arizona)
Raphael Warnock (senator from Georgia)
Some candidates who have not done well on this metric include:
Gavin Newsom
Kamala Harris
Tim Walz
AOC
[Edited to add] Elizabeth Warren fares particularly badly on this metric, though I don’t think she’ll run in 2028
Anthropic: “we expect powerful AI systems will emerge in late 2026 or early 2027… Intellectual capabilities matching or exceeding that of Nobel Prize winners across most disciplines… The ability to navigate all interfaces available to a human doing digital work today… The ability to autonomously reason through complex tasks over extended periods—hours, days, or even weeks… The ability to interface with the physical world”
This is kind of annoyingly phrased, because it sounds like they’re saying that AIs will be making Nobel Prize-level discoveries by late 2026 or early 2027. But in fact that’s not what they’re saying. They’re only claiming that AIs will be capable of doing tasks that Nobel Prize winners could do in hours or days (or maybe even weeks) -- whereas Nobel Prize-level discoveries take years.
Bores repeatedly addressed concerns about regulatory burden by saying that frontier AI developers’ own memos said this bill would add 1 full time employee, and so wasn’t that burdensome.
I’d be surprised if this was true and very surprised if it’s what frontier developers said even if it was true, given their incentives. I’m waiting to hear back on the memo
As far as I can tell, Bores never said that frontier AI developers’ own memos said this; rather, it was that an opposition memo said this. Bores mentions this memo a few times during the 90 minutes; here’s a typical quote:
I’ll note that what came in as an opposition memo said that they estimated that this would require one full-time employee to comply with.
I believe that this is the memo that Bores was talking about. It was written by Will Rinehart of the American Entreprise Institute, which opposed the bill.
I just found myself here (two years later) because of a discussion about Control AI. I feel conflicted but am closer to agreeing than disagreeing with your comment. I think that my original comment was somewhat written in soldier mindset.
Thanks for the suggestion!
For what it’s worth, we believe that a mechanistic estimator can beat all sampling-based methods, no matter how sophisticated they are. The philosophical reason for this is that sophisticated sampling-based methods outperform simple Monte Carlo by exploiting structure in the function whose average value they’re estimating—but a mechanistic estimator can exploit that same structure, too.
In fact, I think it almost follows from the MSP that we can beat any sampling-based method. To see this, suppose you have some sophisticated estimator , which is given a neural net and some random coin flips as input, and produces a sophisticated, unbiased, low-variance estimate of using . Now, define the architecture as: . The MSP says that we need to be able to estimate the average output of (which is the same as the average output of ) with squared error less than or equal to the variance of , in the time that it takes to run . (We’re taking here.) In other words, given any sophisticated sampling algorithm for estimating the average output of , there needs to be a corresponding mechanistic estimator that gets lower (or equal) error in the same amount of time.
(I think this argument isn’t perfectly tight, because it’ll probably run into the same uniformity issues that I discussed in the “getting rid of ” appendix, which is why I said “almost follows” rather than “follows”.)
ARC progress update: Competing with sampling
Note that “toss-up” races are races where the general election (i.e. between the Democratic and Republican candidates) is a toss-up. By guess is that in such races, an extra $2,500 spent on TV ads is necessary to net a candidate one extra vote. This is because the pool of persuadable voters is much smaller: most voters will vote for the Democrat no matter what or vote for the Republican no matter what. By contrast, spending goes a lot further in primary elections.
There’s a cottage industry that thrives off of sneering, gawking, and maligning the AI safety community. This isn’t new, but it’s probably going to become more intense and pointed now that there are two giant super PACs that (allegedly) see safety as a barrier to [innovation/profit, depending on your level of cynicism]. Brace for some nasty, uncharitable articles.
One such article came out yesterday; I think it’s a fairly representative example of the genre.
My guess for Bores was:
25% better to donate on first day than second day
2x better to donate in late 2025 than 2026
I think that similarly for Wiener, I don’t think it makes a huge difference (maybe 15% or so?) whether you donate today vs. late December. Today vs. tomorrow doesn’t make much difference; think of it as a gradual decay over these couple months. But I think it’s much better (1.3x?) to donate in late December than early January, because having an impressive Q4 2025 fundraising number will be helpful for consolidating support. (Because Wiener is more of a known quantity to voters and party elites than Bores is, this is a less important factor for Wiener than it is for Bores.)
Nancy Pelosi is retiring; consider donating to Scott Wiener.
[Link to donate; or consider a bank transfer option to avoid fees, see below.]
Nancy Pelosi has just announced that she is retiring. Previously I wrote up a case for donating to Scott Wiener, an AI safety champion in the California legislature who is running for her seat, in which I estimated a 60% chance that Pelosi would retire. While I recommended donating on the day that he announced his campaign launch, I noted that donations would look much better ex post in worlds where Pelosi retires, and that my recommendation to donate on launch day was sensitive to my assessment of the probability that she would retire.
I know some people who read my post and decided (quite reasonably) to wait to see whether Pelosi retired. If that was you, consider donating today!
How to donate
You can donate through ActBlue here (please use this link rather than going directly to his website, because the URL lets his team know that these are donations from people who care about AI safety).
Note that ActBlue charges a 4% fee. I think that’s not a huge deal; however, if you want to make a large contribution and are already comfortable making bank transfers, shoot be a DM and I’ll give you instructions for making the bank transfer!
Oh yup, thanks, this does a good job of illustrating my point. I hadn’t seen this graphic!
This would require a longer post, but roughly speaking, I’d want the people making the most important decisions about how advanced AI is used once it’s built to be smart, sane, and selfless. (Huh, that was some convenient alliteration.)
Smart: you need to be able to make really important judgment calls quickly. There will be a bunch of actors lobbying for all sorts of things, and you need to be smart enough to figure out what’s most important.
Sane: smart is not enough. For example, I wouldn’t trust Elon Musk with these decisions, because I think that he’d make rash decisions even though he’s smart, and even if he had humanity’s best interests at heart.
Selfless: even a smart and sane actor could curtail the future if they were selfish and opted to e.g. become world dictator.
And so I’m pretty keen on interventions that make it more likely that smart, sane, and selfless people are in a position to make the most important decisions. This includes things like:
Doing research to figure out the best way to govern advanced AI once it’s developed, and then disseminating those ideas.
Helping to positively shape internal governance at the big AI companies (I don’t have concrete suggestions in this bucket, but like, whatever led to Anthropic having a Long Term Benefit Trust, and whatever could have led to OpenAI’s non-profit board having actual power to fire the CEO).
Helping to staff governments with competent people.
Helping elect smart, sane, and selfless people to elected positions in governments (see 1, 2).
People are underrating making the future go well conditioned on no AI takeover.
This deserves a full post, but for now a quick take: in my opinion, P(no AI takeover) = 75%, P(future goes extremely well | no AI takeover) = 20%, and most of the value of the future is in worlds where it goes extremely well (and comparatively little value comes from locking in a world that’s good-but-not-great).
Under this view, an intervention is good insofar as it affects P(no AI takeover) * P(things go really well | no AI takeover). Suppose that a given intervention can change P(no AI takeover) and/or P(future goes extremely well | no AI takeover). Then the overall effect of the intervention is proportional to ΔP(no AI takeover) * P(things go really well | no AI takeover) + P(no AI takeover) * ΔP(things go really well | no AI takeover).
Plugging in my numbers, this gives us 0.2 * ΔP(no AI takeover) + 0.75 * ΔP(things go really well | no AI takeover).
And yet, I think that very little AI safety work focuses on affecting P(things go really well | no AI takeover). Probably Forethought is doing the best work in this space.
(And I don’t think it’s a tractability issue: I think affecting P(things go really well | no AI takeover) is pretty tractable!)
(Of course, if you think P(AI takeover) is 90%, that would probably be a crux.)
If you donate through the link on this post, he will know! The /sw_ai at the end is ours—that’s what lets him know.
(The post is now edited to say this, but I should have said it earlier, sorry!)
Just so people are aware, I added the following note to the cost-effectiveness analysis. I intend to return to it later:
[Edit: the current cost-effectiveness analysis fails to account for the opportunity cost of Scott Wiener remaining in the State Senate for another two years -- 2027-2028 -- until he needs to leave due to term limits. I think this is an important consideration. My current all-things-considered belief is that this consideration is almost canceled out by the other neglected effect of strengthening ties between AI alignment advocates and Wiener in worlds where he loses and remains in the State Senate for those two years. However, this analysis is subject to change.]
Thank you!
My threat model is actually
More of the lightcone will be controlled by the Chinese government (or its successor).
My current guess is that the long-term future looks better if American actors have more bargaining power over the long-term future than Chinese actors. If space is subdivided amongst the key actors (including the Chinese government, the U.S. government, and possibly others), I worry that the parts controlled by the Chinese government would be illiberal, in the same kinds of ways that China is now. One particularly bad version of this is an AI-enabled surveillance state that locks in something like current Chinese ideology, curtailing the potential for moral improvement. I think this is less likely in the parts of space controlled by the U.S. government, because I think those are reasonably likely to be founded on fairly liberal values, perhaps similar in spirit to the U.S. constitution.