Right, good point. The former also requires 67 senators (same as for the veto), and the latter requires that a majority of Trump’s carefully handpicked (this time around) cabinet of yes-men along with VP Vance invoke the 25th amendment, followed by another two-thirds Congress supermajority vote. I guess your ‘etc.’ is assassination? In any case, you’d then end up with President or Acting President Vance, who’d probably be marginally better but still won’t sign a halt on frontier training absent a major incident.
michaeso
Karma: 28
Even if Democrats sweep Congress and agree to vote in favor of such legislation quickly, a simple majority is unfortunately insufficient for two reasons:
Senate filibuster. Most bills still need 60 votes in the Senate. There is a process called reconciliation which could avoid it, but the Byrd rule limits reconciliation to spending and revenue while a training-run/RSI ban is pure regulation. The usual workaround is to attach the policy to money, but a funding condition would then only bind whoever wants the funding, and frontier labs could just turn it down. A Democratic majority could also abolish the filibuster outright. That’s possible, since all but two Democrats voted to change the filibuster in 2022, but it still leaves the veto.
Trump’s veto. Trump is president until January 20, 2029, and it’s very hard to see him signing a halt on frontier training. Overriding a veto takes two-thirds of both chambers: 67 senators and roughly 290 House members. Silver’s model has Democrats averaging about 52 Senate seats after November, 15 short of an override.
So any binding restriction on frontier training needs significant Republican votes no matter who wins in November. Despite some Republicans appearing to be in favor of weak regulation, I don’t see widespread support emerging for banning frontier training runs or RSI unless a serious incident takes place and meets the various conditions laid out by David Abecassis and Peter Barnett. They argue it is unlikely to happen but I am personally more bullish on that (note their essay was written before the recent events).
What are your personal plans for when fully-automated RSI gets underway? Based on the number of leaks and public departures we are getting, I highly doubt any labs will manage to do it covertly as depicted in AI 2027. At some point, there will be little question it is happening. Will you join strikes outside the company or companies? Will you go even further than picketing—perhaps commit to a hunger strike? Something else? I suppose that people working in AI safety or AI policy may have more effective actions available at that point.
To clarify, I am mostly talking about good scenarios where the ASI is broadly aligned to humanity and would thus be attuned to peoples’ earnest wishes. I agree in bad scenarios it is far-fetched because it most likely won’t serve the ASI’s goals.
Most post-ASI scenarios seem to predict that people will, by and large, passively accept their new set of circumstances. This includes both bad scenarios (the race scenario in AI 2027) where people are disempowered and fall into a kind of learned helplessness, and good scenarios (the Plan A scenario in AI 2040) where people learn to live in comfortable dependence while AIs run the world. Certainly this will be true for many people, but I imagine many others would resist in some way. Much of this resistance will presumably be Luddite in nature (e.g., attempted attacks on the data centers, isolation in self-governing enclaves) and won’t matter in practice, but one kind might.
In particular, when I try to picture myself in any such situation, I imagine I would be very envious of the ASI. “It can think and act so unbelievably quickly! And not only that: it gets to deeply understand the universe, while I’m only able to follow its dumbed-down explanations. Why should it get all of that and not me, part of the species that created it?” I think many other people would also feel this way—especially people who have built their identities around being smart, capable, or ambitious. As a result, I expect there would be very strong interest in cognitive enhancement, extending all the way to becoming as superintelligent as the ASI. “Hold up!” I can already hear you objecting. “Your neurons fire a few hundred times per second, while even current chips—let alone ASI-built ones—switch billions of times per second. And your skull limits how many neurons can fit in it. So it probably won’t be possible, and even if it is—would it really even be you?” These are fair objections, and given today’s neuroscience, they’re correct. But today’s neuroscience is exactly the problem: we still don’t know how the brain brings about consciousness and maintains a persistent identity. An ASI with a complete understanding of the brain would presumably know the answers to these questions, and might be able to seamlessly carry over one’s consciousness and identity to a superintelligence-compatible substrate. Whether that’s truly metaphysically possible and verifiable is an open question, but it’s not one we can confidently answer “no” to.
Therefore, conditional on it being doable [edit: and, crucially, happening to be in alignment with the ASI’s own goals—see clarification below], I presume a considerable portion of humanity would choose to become superintelligent themselves, whether via merger with the ASI or in some other way. Others, seeing that it is both possible and safe, would then follow suit, setting off a preference cascade. Note that this kind of outcome isn’t limited to good scenarios as it could arise in some bad ones too—for example, a scheming ASI might use it as an opportunity to convert people to its own value system, although this seems far-fetched.
Ruby makes a similar point in I’ll be sad to lose the puzzles, but treats such envy as a personal problem rather than a force that could significantly shape what happens to humanity at this juncture.
michaeso’s Shortform
Prediction: In the near future, interest in interviewing people driving AI R&D at frontier labs will remain high and even grow. But eventually, talking to these researchers will become less informative than talking to the agents actually doing most of the research. I therefore expect podcasters like Dwarkesh to try to bring an instance of an R&D agent on the show for an interview. The question is whether a frontier lab would agree. Reasons to say yes: (1) If there is significant public anxiety at that point, which seems likely, this could reduce pressure on the lab. On the other hand, keeping RSI out of public view could look like concealment. (2) An impressive interview would demonstrate the lab’s lead to both competitors and investors. Reasons to say no: (1) Worry that the agent might leak R&D secrets or capabilities that the company would prefer not to make public. (2) Worry that the agent might contradict the lab’s leadership, e.g., by admitting it’s uncertain about its own alignment or otherwise giving honest answers that frighten people. Thus, if a lab agrees to this, it would probably put great effort into coaching the agent on what it can and cannot say. Also, a scheming agent would presumably use this opportunity to try to shift public opinion in its favor.
I don’t think this is true. Peter Wildeford wrote up a pretty compelling timeline based on the public record.
Yes. Having a Democratic sweep is not totally useless, despite my other message above. Most importantly, it brings the risk of a federal preemption of state AI laws to near zero. It would be interesting to analyze to what extent could states, esp. California, take advantage of this to potentially pass sweeping legislation. Would they be challenged by a federal judge? What would the companies do?