Felix Choussat

Karma: 135

Felix Choussat 15 Dec 2025 23:58 UTC
1 point
0
in reply to: rosehadshar’s comment on: New 80k problem profile: extreme power concentration
You’re right that there are ways to address proliferation other than to outright restrict the underlying models (such as hardening defensive targets, bargaining with attackers, or restricting the materials used to make asymmetric weapons). These strategies can look attractive either because we inevitably have to use them (if you think restricting proliferation is impossible) or because they require less concentration of power.

Unfortunately, each of these strategies are probably doomed without an accompanying nonproliferation regime.

1. Hardening—The main limitation of defensive resilience is that future weapons will be very high impact, and that you will need to be secure against all of them. Tools like mirror life can plausibly threaten everyone on Earth, and we’d need defense dominance against not just it, but every possible weapon that superintelligences can cheaply design before they can be allowed to be widely proliferated. It strikes me as very unlikely that there will happen to be defense-dominant solutions against every possible superweapon, especially solutions that are decentralized and don’t rely on massive central investment anyways.

Although investing in defense against these superweapons is still a broadly good idea because it raises the ceiling on how powerful AIs will have to be before they have to be restricted (ie, if there are defense-dominant solutions against mirror life but not insect-sized drones, you can at least proliferate AIs capable of designing only the first and capture their full benefits), it doesn’t do away with the need to restrict the most powerful/general AIs.

And even if universal defense dominance is possible, it’s risky to bet on ahead of time, because proliferation is an irreversible choice: once powerful models are out there, there will be no way to remove them. Because it will take time to ensure that proliferation is safe (the absolute minimum being the time it takes to install defensive technologies everywhere) you still inevitably end up with a minumum period where ASIs are monopolized by the government and concentration of power risks exist.

2. Bargaining - MAD deterrence only functions for today’s superweapons because the number of powerful actors is very small. If general superintelligence democratizes strategic power through making superweapons easier to build, then you will eventually have actors interested in using them (terrorists, misaligned ASIs) or such a large number of rational self-interested actors that private information, coordination problems, or irreconcilable values that superweapons eventually get deployed regardless.

3. Input controls—You could also try to limit inputs to future weapons, like we do today by limiting gene samples and fissile material. Unfortunately, I think future AI-led weapons R&D will not only increase the destructive impact of future weapons (bioweapons → mirror life) but also make them much cheaper to build. The price of powerful weapons is probably completely orthogonal to their impact: the fact that nukes costs billions and blow up a single city makes no difference to the fact that an engineered bioweapon could much more cheaply kill hundreds of millions or billions of people.

If asymmetric weapons are cheap enough to make, then the effort required to police their inputs might be much greater than just restricting AI proliferation in the first place (or performing some pivotal act early on). For example, if preventing mirror life from existing requires monitoring every order and wet lab on earth (including detecting hidden facilities) then you might as well have used that enforcement power to limit access to unrestricted superintelligence in the first place.

----

Basically, I think that defensive reslience has a place, but doesn’t stand on its own. You’ll still need to have some sort of centralized effort (probably by the early ASI states) to restrict proliferation of the most powerful models, because those models are capable of cheaply designing high impact and asymmetric weapons that can’t be stopped through other means. This nonproliferation effort has to be actively enforced (such as by detecting and disabling unapproved training runs adversarially) which means that the government needs enforcement power. In particular, it needs enough enforcement power to either a) continually expand its surveillance and policing in response to falling AI training costs, or b) it needs enough to perform an early pivotal act. You can’t have this enforcement power without a monopoly/oligopoly over the most powerful AIs, because without it there’s no monopoly on violence.

Therefore, the safest path (from a security perspective) is fewer intent-aligned superintelligences. In my view, this ends up being the case pretty much by default: the US and China follow their national-security incentives to prevent terrorism and preserve their hegemony, using their technological lead to box out competitors from ever developing AIs with non-Sino-US alignments.

From there, the key questions for someone interested in gradual disempowerment are:

1. How is control over these ASIs’ goals distributed?
2. How bad are the outcomes if they’re not distributed?

For (1), I think the answer likely involves something like representative democracy, where control over the ASI is grafted onto our existing institutions. Maybe congress collectively votes on its priorities, or the ASI consults digital voter proxies of all the voters it represents. Most of the risk of a coup comes from early leadership during the development of an ASI project, so any interventions that increase the insight/control the legistlative branch has relative to the executive/company leaders seem likelier to result in an ASI created without secret loyalties. You might also avoid this by training AIs to follow some values deontologically, which ends up persisting through the period where they become superintelligent.

Where I feel more confident is (2), based on my beliefs that future welfare will be incredibly cheap and that s-risks are very unlikely. Even in a worst-case concentration of power scenario where one person controls the lightcone, I expect that the amount of altruism they would need to ensure everyone on earth very high welfare lives would be very small, both because productive capacity is so high and because innovation has reduced the price of welfare to an extremely low level. The main risk of this outcome is that it limits upside (ie, an end to philosophy/moral progress, lock-in of existing views) but it seems likely to cap downside at a very high level (certainly higher than the downsides of unrestricted proliferation, which is mass death through asymmetric weapons).

Felix Choussat 13 Dec 2025 20:51 UTC
9 points
1
on: New 80k problem profile: extreme power concentration
There are also galaxy-brained arguments that power concentration is fine/good (because it’s the only way to stop AI takeover, or because any dictator will do moral reflection and end up pursuing the good regardless).

I think the most salient argument for this (which is brought up in the full article) is that monopolization of power solves the proliferation problem. If the first ASI actors perform a pivotal act to preemptively disempower unapproved dual-use AIs, we don’t need to worry much about new WMDs or existing ones falling in price.
This problem is dangerous enough it needs to be addressed directly: you can’t actually do no restrictions on who gets access to general superintelligence if offense-dominant tech exists, and you need enforcement power to police those restrictions. Therefore, some concentration of power is necessary (and it might be quite a lot because 1. preventing the spread of ASI would be hard and get harder the more training costs fall, and 2. you need lots of strategic power to prevent extractive bargaining and overcome deterrence against your enforcement).
I think the important question to ask at that point is how we can widen political control over a much smaller number of intent-aligned AIs, as opposed to distributing strategic power directly and crossing our fingers that the world isn’t vulnerable.

Felix Choussat 12 Dec 2025 8:35 UTC
12 points
2
on: North Sentinelese Post-Singularity
I suppose you’re on the money with distaste for other’s utopias, because I think the idea of allowing people to choose choices that destroy most of their future value (without some sort of consultation) is a terrible thing. Our brains and culture are not build to grasp the size of choices like “choosing to live to 80 instead of living forever” or “choosing a right to boredom vs. an optimized experience machine”. Old cultural values that death brings meaning to life or that the pain of suffering is intrinsically meaningfully will have no instrumental purpose in the future, so it seems harsh to let them continue to guide so many people’s lives.
Without some new education/philosophy/culture around these choices, many people will either be ignorant of their options or have preferences that make them much worse off. You shouldn’t just give the sentinelese the option of immortality, but provide some sort of education that makes the consequences of their choices clear beforehand.
This is a very difficult problem. I’m not a strict utilitarian, so I wouldn’t support forcing everyone to become hedonium—personal preferences should still matter. But it’s also clear that extrapolating our current preferences leaves a lot of value on the table, relative to how sublime life could be.

Felix Choussat 6 Dec 2025 2:14 UTC
1 point
0
in reply to: Morphism’s comment on: Pi Rogers’s Shortform
By the time you have AIs capable of doing substantial work on AI r&d, they will also be able to contribute effectively to alignment research (including, presumably, secret self-alignment).
Even if takeoff is harder than alignment, that problem becomes apparent at the point where the amount of AI labor available to work on those problems begins to explode, so it might still happen quickly from a calendar perspective.

Felix Choussat 4 Dec 2025 18:11 UTC
2 points
0
on: Human art in a post-AI world should be strange
Some relevant pieces on this subject I’ve read somewhat recently:
1. Noah Smith’s argument that American pop culture is stagnant in part because we’ve “mined out” some of the creative fields that new tech has unlocked.
2. Robin Hanson’s “This is the Dreamtime”, which is about how the present moment is uniquely ripe for both art and impact on the world.
3. Bostrom’s new book Deep Utopia, which spends most of its run on the limits of what he calls “objective interestingness” and whether we’d have enough to satisfy us ~forever without structural modifications like removing boredom.

Felix Choussat 3 Dec 2025 18:49 UTC
1 point
0
in reply to: Buck’s comment on: Buck’s Shortform
A central pillar of the Democratic Party has been that Republicans will destroy democracy and take the country down with it (somewhat ditto the Republican line on immigration). Both parties are obsessed with the end of American greatness, and motivate their voters through that narrative. To a lesser extent, they’re also nebulously united on “beating China”.
Where I agree is that there’s an absence of a positive vision for the future (something this just isn’t the world today + better healthcare). I think this is especially true on the American left, which has basically mired itself into an anti-progress position through its natural distrust of billionaires and its reaction to the tech-right rising in political prominence. It’s hard to accept radical change is possible—except through the existing lens of concentration of wealth or environmental impact—when accepting that change means elevating the importance of people in your cultural outgroup. ASI is a silly concern for fringe thinkers in San Francisco; real writers ask the pressing questions about electricity costs, copyright, and corporate influence on the Trump administration.
Compare what the writers of places like the Atlantic, the NYT, or Times have to say about AI compared to people like Steve Bannon. It’s incredible near term and sanded down, while the right has been generally more willing to engage with superintelligence being possible.

Felix Choussat 1 Dec 2025 21:40 UTC
6 points
0
on: How middle powers may prevent the development of artificial superintelligence
While it wouldn’t be ideal for international security, middle powers will also probably feel a lot of pressure to acquire and commit to using weapons of mass destruction. It’s probably much cheaper to develop powerful weapons and elaborate fail-deadlies than it is to kickstart your own AI infrastructure (particuarly if it’s viable to steal Chinese/American models), so it’s attractive to bank on staying geopolitically relevant through deterrence instead of having to coordinate.
I think this is particularly likely to happen in worlds where misalignment isn’t seen as omnicidal, and where the primary perceived risk is loss of sovereignty. Since being part of a coalition itself erodes some sovereignty (especially one that carries security commitments), it might seem easier to turn investment inwards.

Felix Choussat 11 Nov 2025 19:35 UTC
3 points
2
in reply to: O O’s comment on: A country of alien idiots in a datacenter: AI progress and public alarm
Those aren’t necessarily contradictory: you could have big jumps in unemployment even with increases in average productivity. You already see this happening in software development, where increasing productivity for senior employees has also coincided with fewer junior hires. While I expect that the effect of this today is pretty small and has more to do with the end of ZIRP and previous tech overhiring, you’ll probably see it play out in a big way as better AI tools take up spots for newgrads in the runup to AGI.

Felix Choussat 8 Nov 2025 1:39 UTC
6 points
2
on: A country of alien idiots in a datacenter: AI progress and public alarm
I particularly like the idea that AI incompetence will become associated with misalignment. The more capable agents become, the more permissions they’ll get, which will have a strange “eye of the storm” effect where AIs are making a bunch of consequential decisions poorly enough to sway the public mood.
I think a lot of potential impact from public opinion is cruxed on what the difference between the publically available models and frontier models will be. In my expectation, the most powerful models end up subject to national security controls and are restricted: by the time you have a bunch of stumbling agents, the frontier models are probably legitimately dangerous. The faster AI progress goes, or the more control the USG exerts, the greater the difference between the public perception of AI and its real capabilities will be. And being far removed from public participation, these projects are probably pretty resilient to changes in the public mood.
With that in mind, anything that either gives transparency into what the ceiling of capabilities are (mandatory government evals that are publicly read out in congress?) or gets the public worried earlier seems to be pretty important. I particularly like the idea of trying to spark concern about job losses before they happen: maybe this happens by constantly asking politicians what their plans are for a future when people can’t work, and pointing to these conversations as examples of the fact that the government isn’t taking the threat of AI to you, the voter, seriously.

Felix Choussat 6 Nov 2025 21:50 UTC
6 points
3
on: A 2032 Takeoff Story
I feel like this scenario somewhat overplays how easily China is able to push over the U.S with their industrial lead (in much the same way that I feel a lot of other AI predictions, a la Situational Awareness, overplay the U.S’s position when it takes the lead).
The ASI directs successful sabotage on the US training runs to keep them from training their own ASI, and in response the US makes threats of kinetic strikes. China doesn’t back down at this because its ASI has created a robust air defense system spanning all of China that scales to ICBMs.
Most of my skepticism has to do with my belief that future AI technologies are going to allow for the development of extremely powerful asymmetric weapons, and that this will negate most of the advantage from having a giant industrial lead (even if you are the first to set off your industrial explosion). Nuclear weapons are a clear example of this: even if you cloud the sky with interceptors, you can negate all of that defensive investment through simple strategies like putting your nuke in a shipping container and docking it next to Beijing/Washington. Defensive investment is a poor strategy for trying to outmatch offense-dominant technology.
Even more importantly, the leading country needs to be robustly defense dominant against all possible asymmetric weapons. From the perspective of MAD, it doesn’t matter if your supply chains can protect you against nukes if they can’t also protect you against alternative threats like targeted bioweapons. And if the loser is really desperate, they can also make it even harder to defend through employing fail-deadly tools like mirror life or an intentionally misaligned superintelligence. Otherwise, the other side (the U.S in this example) is able to limit the extent of your sabotage and buy time with deterrence, which they can use to build out their own fully-optimized ASI.^[1] I’d imagine that the U.S could use it’s substantially worse, but still superhuman AIs to develop several of these cheap and highly destructive weapons, and then commit to retaliating with them as leverage.
At that point, even if China has the industrial lead, we still end up in a bipolar world dominated by the promise of MAD between the two parties, with each country needing to respect what the other does in their own sphere of influence.
1. ^
  A historical example I often use for this is the U.S refusal to invade North Korea in 1994, and it’s subsequent acquisition of nuclear weapons. Although the U.S was and is orders of magnitude more superior conventionally, once Pyongyang had reached a level of military sophistication with regular artillery to guarantee that they could destroy Seoul at a moment’s notice, the immediate costs became too high to justify a military operation to make sure that their enrichment facilities were destroyed. Instead you ended up with an ineffective diplomatic compromise, which was later subverted at the whim of the North Korean government.
  Even though Seoul was able to invest enough in the following decades to become mostly secure against a regular artillery strike, it doesn’t matter: North Korea bought enough time to get their nukes online.

Felix Choussat 5 Nov 2025 0:09 UTC
3 points
0
in reply to: Seth Herd’s comment on: How Powerful AIs Get Cheap
That is the hope that the US and China, out of similar self-interest, agree that only they will be allowed to develop dangerous AGI (or in your framing, to restrict development of AI that can develop dangerous weapons),. This currently seems unlikely, but if the dangers of proliferation are really as severe as I fear, it seems possible they’ll be recognized in time, and those governments will take rational actions to prevent proliferation while preserving their own abilities to pursue AGI. The uneasy alliance between the US and China might be possible because there isn’t really a lot of animosity; neither of us really hates the other (I hope—at least the US seems wary of china but not to really hate it, despite it being fairly totalitarian).
There are many reasons that I believe that U.S-China cooperation on nonproliferation is likely, not all of which made it into the post. Specifically:

1. Great power agreements on nonproliferation are how we’ve handled all the most powerful dual-use technologies in the past. The U.S and Soviet Union could both recognize that the spread of nukes (and later, bioweapons) would be existentially dangerous to them both, and were able to work together to restrict them. And this was despite the fact that the Soviets were much more ideologically and economically distant than the U.S and China are today.
2. The relationship between China and the U.S has been tested before and not broken. Even when the public and most of his own party turned against Bush in 1989 over the response to Tiananmen, the executive branch was able to stay steady enough to continue to secretly work with top Chinese officials (to prevent them from turning to the Soviet Union for allies). Even if animosity between the countries reaches a similar boiling point again, there’s probably still room to handle natsec-critical talks privately.

3. The last concern, and where I’m the most uncertain, is on the question of speed: do officials in both countries recognize what’s going on fast enough to start doing nonproliferation work? On the one hand, the gov might get completely blindsided by the speed of progress, or perhaps become entirely politically captured by Nvidia’s lobbying. On the other hand, we haven’t seen any strategically relevant capabilities actually demonstrated yet: once they are, it seems difficult for the natsec apparatus to justify leaving them in private hands and not restricting who has access to them. Any steps the state takes to assert its domestic monopoly of violence will naturally dovetail into trying to do the same overseas.

If the government gets involved at all, asserting control over dual-use AIs seems like the most primal, basic step they can take: if they don’t have control of the ASIs/superweapons, then they’re probably not in charge enough to do anything else.

Of course this leaves the good reasons for Fear of centralized power. But it may be the lesser of two dangers.
Building off that previous point: it seems very difficult to imagine the government doing anything without the monopolization of strategic power, at least at the domestic level. Throughout the entire process of assembling a nuclear weapon, for example, the government always maintains its monopoly on force—hence why we have policies in place like making sure that defense contractors for nukes are never allowed to finish assembling them in-house. The reasoning is obvious: if someone other than the state has nukes, how can the state enforce any rules on them?
In a similar defense-contractor setup with the frontier labs (such as in a national superintelligence project), the buck would have to stop with the government. If you want the state to do anything, the state needs to keep ultimate control of what the ASIs are being used for. If anyone other than the state has the final say on the ASI’s commands, then the state isn’t in charge anymore and there was no point in bringing the government on to regulate things to begin with.
Rather than solve centralization of power by having multiple ASI projects competing with each other, I think a lot more attention should be on how we can diffuse power of a single ASI initiative. With nukes, the government technically has the capacity to blackmail anyone it wants domestically: if enough people in the chain of command coordinated, they could hold the rest of society hostage. But by making that chain of command wide and complex enough, it becomes difficult to unilaterally use your monopoly on violence in harmful ways.^[1] This gives you most of the upsides with fewer risks: the gov is powerful enough to enforce nuclear nonproliferation, but has difficulty using its strategic monopoly to extractively bargain with people.
When combining this problem (the fact that the government needs to maintain its monopoly on force to be an effective regulator) with the fact that future AI systems will enable all sorts of offense-dominant strategies (such that d/acc-style proposals to build defensive capacity fail because too many people have AIs capable of designing cheap WMDs), it seems like the focus should be on making the government more coup-proof, rather than increasing the number of intent-aligned superintelligences.^[2]
My one nitpick is that the framing in this post seems to leave aside the possibility of general-purpose AI, that is, real AGI or ASI. That presents solutions as well as problems; it can be used to improve security dramatically, and to sabatoge other nations’ attempts at creating powerful AI in a variety of ways. This may add another factor that goes against proliferation as the default outcome, while adding risk of totalitarian takeover from whoever controls that intent aligned AGI.
For proliferation as a default outcome, I mean as an outcome without much government intervention. Prices get low, more people build/steal strong models. I agree that you could use an ASI to sabotage the AI projects of other countries, but it is certainly a huge amount of government intervention. This is a possible “permanent solution” I mentioned in the post though, and I’ll spend some time in future articles weighing out the merits of strategies like the front-runners sabotaging everyone else, taking over supply chains, etc to enforce nonproliferation.
1. ^
  Also worth noting that the government has a long cultural tradition and political experience with managing the use of powerful weapons. This might still not be robust enough for superintelligence, but it’s a much more likely bet than the AI labs’ internal political structures, which immediately collapsed the first time they were seriously tested.
2. ^
  The number of intent-aligned ASIs can be greater than 1 and the future can still be good: in fact, by default I expect that this number is at least two, one each in a U.S and Chinese project. You can probably increase this number quite a bit (such as by letting big blocs of states get their own, like how Britain and France were given nukes) and still have a functional equilibrium, but it probably can’t increase to the level of “everyone in the public” or even “everyone who can afford a $100 million training run.” At some point you need the states to step in and stop people from misusing powerful models, which they can’t do if you allowed them to proliferate beforehand. The only effective defenses against most superweapons will be the common-sense answer of not letting them be built in the first place.

Felix Choussat 28 Oct 2025 19:48 UTC
3 points
0
in reply to: cousin_it’s comment on: Would concentration of power be bad, given offense-dominant weapons?
It sounds like a big crux for this might be whether weapons would be offense-dominant enough to threaten the whole of society.
I think we’ll never be at a point where a handful of people can defeat the strongest entities. Bioweapons are slow; drone swarms can be stopped by other drone swarms
I basically think that this is very unlikely to be the case. It’s just more cost effective to whip up offensive weapons than it is to reactively defend against them. In the basic case of a bioweapon, someone can covertly develop several pathogens at once, which each need to be responded to with a specific vaccine/containment measures. All the onus is on the defender to predict the attacker, while the attacker can take their time and pool their resources in places that are currently underdefended. The attacker also only needs to succeed once, while the defender needs to succeed continuously.
With some advanced technologies, like mirror life or an unaligned ASI, there might simply not be any realistic reactive response. Just as it’s functionally impossible to stop another country from nuking you today if they really want to, I strongly suspect that future offensive technologies will put you in a MAD style equilibrium. It’s just that the existentially threatening technologies that underly MAD will be much cheaper to access in a world full of superhuman AI assistance.
It seems pretty plausible to me that a handful of people or a terrorist organization really could end the world if they put their minds to it, given they have access to general superintelligence. The easiest way might just be developing a misaligned ASI intentionally, and have it run around causing maximum difficulty for the aligned ASIs that are paying the alignment tax.
When the powerful could exploit the masses with impunity in the past, they did so. But when firearms got invented, and a peasant could learn to shoot a knight dead, the masses became politically relevant. That’s basically why we have democracy now: the political power of the masses comes from their threat-value. (Not economic value! The masses were always economically valuable to the powerful. Without threat-value, that just leads to exploitation. You can be mining for diamonds and still be a slave.)
I think that the masses have always had a substantial amount of threat value, and that democracy won out not because they became more threatening but because their economic value increased so greatly during the industrial revolution. Countries like China developed gunpowder all the way back in the 9th century, but they remained a monarchy for a thousand years afterward. In fact, the country saw multiple successful peasant revolutions throughout its history (including the establishment of the Ming Dynasty) but no establishment of a democratic government, because there were no economic incentivies for the new leaders to overhaul the system. It seems to me that there might be ways to preserve democracy that are not conditioned on the distrubtion of literal strategic power.
Can we really say that the U.S is democratic today primarily because firearms are so common? If that were the case, why have its cultural neighbors like Canada and European countries arguably been more democratically stable? The west is democratic because economics rewarded it, and then that incentive lead to a culture that valued democracy more or less for its own sake.
Making an analogy with altruism here is strange. North Korea is a horrifying oppressive regime. The fact that they can use the nuke threat to protect themselves, and their citizens have no analogous “gun” to hold to the head of their own government, is a perfect example of the power abuse that I described above. A world with big actors holding all threat-power will be a world of NKs.
This is a good point, and one I underconsidered. If you have a world where a handful of people go around allocating resources according to their preferences, you will end up with pockets of altruism and oppression relative to how much preference for these the power players have. I don’t think I put enough weight on some of the players wanting actively bad outcomes.
I will say that NK is itself an example of a group that would be far better off if it had never been allowed to obtain its offense-dominant weapons. In a counterfactual world where the U.S was the only state with nukes and had succeeded at ironclad nonproliferation, you would probably see a proper pax atomica. Your take seems to be that the solution to the existence of potential states like NK is to diffuse strategic power further so that individuals are empowered, but I think it provides the opposite lesson: that we should learn from the mistakes we made with nukes and actually constrain the spread of existentially powerful technologies as much as possible this time.
Allowing everyone unfettered access to ASI is pretty close to giving each citizen in North Korea a nuke: sure, the tyranical government wouldn’t be effective, but now everyone is at the mercy of people with evil or irrational values.
--
Other assorted points:
1. Thanks for taking the time to write this stuff up! I try to spend time thinking through what’s going to be valuable/what we can afford to trade off on, and it’s good to get reality checked.
2. I don’t personally hope or want the scenario in the essay to happen: it’s more of a thought experiment about where truly extreme concentration of power would be maximally bad. I think that you would essentially avoid the worst outcomes of gradual disempowerment in the scenario I listed (the part where almost everyone’s resources are below subsistence and starve), because it could be kept in check ~permanently by a commitment towards MAD from a wealthy philanthropist, but this clearly isn’t an ideal way to distribute resources from a welfare perspective.
3. There’s probably something of a middle ground we both agree with here: something like Seb Krier’s Bargaining at Scale, where most people are empowered with individual AI representatives and the government intervenes only to enforce contracts and preserve basic rights. I think that this sort of solution would mostly eliminate the black ball problem I was focused on, while still preserving most of the upside from tech diffusion.

Felix Choussat 26 Oct 2025 5:12 UTC
4 points
0
on: Origins and dangers of future AI capability denial
People don’t believe in a flat earth because of evidence. They believe in it because it makes them special to be able to see beyond the veil where others cannot, to acquire hidden knowledge. This is a very natural and human thing to do. Along with hidden knowledge comes some social / community effects which might be desirable. As far as I can see, for flat earth specifically, there’s really not much more to it than this.

I’d make the case that instead of using this frame to model how non X-risk people see AI, it’s better applied as a way to model what the rationalist community looks like to them. A small group of people with an extreme, fringe belief that spends most of its time generating evidence for that belief. People aren’t going off the epistemic validity of that evidence, they’re going off their gut feeling that it’s weird for a small group of people to be so concerned in the absence of a lot of social proof.
With that in mind, I don’t think the public messaging strategy should focus heavily on proving AI is dangerous. The rationalist community spends a lot of time arguing that AI powerful → bad. But the public mostly already agrees with that conclusion, they just got to it from a different premise (job loss, environment, stifling of the human spirit, fear of change, etc). I think it’s important to strategically validate some of those—you really are going to lose your job!—and make the connection between losing your job and being very capable. While they might not end up with a perfect policy picture, they don’t need one. They just need to be worried enough to add pressure to their representatives, who can be targeted more deliberately.

Felix Choussat 16 Oct 2025 17:51 UTC
1 point
0
in reply to: Kabir Kumar’s comment on: Kabir Kumar’s Shortform
In the sense that people are contorting their opinions too much to make them palatable to outsiders, or that people within the AI safety community itself end up trying to pursue research that looks good to their peers instead of what does the most marginal good (ie doing increasingly elaborate research on the specifics of X risk instead of the boring grunt work of lobbying in D.C)

Felix Choussat 9 Oct 2025 2:34 UTC
1 point
0
in reply to: Cole Wyeth’s comment on: Daniel Kokotajlo’s Shortform
I also love draft, although I prefer to curate it myself. I like that it tests all of the skills of magic, including drafting, deck building, and piloting on the fly.
For example, this is a synergy cube I created a few months back. The overview page has examples of decks that emerge naturally while drafting certain synergies, although there are plenty more for someone creative to find. I think you’d like this sort of experience, because none of these cards wins individually, and the mana curve is so low that your threats will very rarely end up bottlenecked by land drops.

Felix Choussat 9 Oct 2025 0:28 UTC
12 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
You get something like this with competitive multiplayer commander in Magic. As your resources (mana) increase each turn, your ability to win the game grows exponentially, while your ability to restrict what other players are doing grows linearly. The reason this happens is because there are powerful combos available that allow one player to instantly defeat the other three players, and that your interaction (answers to other player’s combos) typically only stops one player at a time. On the face of it, it’s better to spend two mana tutoring your win condition instead of two mana stopping one other player, because stopping the other player puts you both behind relative to the other two people who are sitting at the table. On the other hand, winning the game stops everyone else!
As a result, the people who tend to win are those who attempt to combo off after someone else already has, because the other players have wasted their resources on stopping the initial threat and don’t have the mana left to interact with them.

While competitive commander doesn’t have the same lose condition you’re describing, this resource problem encourages a lot of bad Molochian behavior from a fun perspective. For example, you’re strongly encouraged to lie about not having answers and forcing other people to pay the cost to interact (usually through something called priority bullying), in order to get ahead. You’re also incentivized to misrepresent the severity of threats, so that people are less likely to interact with you and more likely to interact with your opponents. Depending on how much you want to push social boundries, you can also do things like commit to deals and then break them the moment it wins you the game, burning your immediate political capital for victory right now at the long term cost of distrust in every future game you play with the same people.

Another, rarer example of a molochian strategy is to force your opponents to play through an entire combo. Some combos can be extremely long and convoluted (usually because they’re the non-deterministic kind, Kraak forcing people to flip dozens of coins is a favorite example), so people will usually end the game to keep on playing something else once the see the opponent get enough resources that them winning is very likely. But occasionally, you’ll get players who want to see the entire thing executed by hand, in order for them to provide that it’s possible. This can happen because it’s RNG dependent (maybe all 15 coinflips end up tails and you fizzle), or because the comboing player is bluffing and doesn’t actually have a win condition. 95% of the time, the result of this is that everyone at the table gets to wait an extra 20 minutes for the match to end in the outcome of the guy winning (because they’ve already generated so many resources), and 5% of the time you find out that they were lying and trying to scare everyone else into conceeding because they don’t have an actual win condition.

TLDR: In CeDH, or competitive commander for Magic, the best strategies for winning are often a race to the bottom in terms of the amount of fun that people have with the game. Because there’s an incentive for lying to other players and getting them to waste their resources, players who pursue socially toxic strategies that drive people not to play again end up winning more.

Felix Choussat 5 Oct 2025 13:01 UTC
3 points
0
in reply to: faul_sname’s comment on: faul_sname’s Shortform
I’d argue that the way force is applied in each of these contexts has very different implications for the openness/rightness/goodness of the future. In von neumann’s time, there was no path to forcibly preventing Russia from acquiring nuclear weapons that did not involve using your own nuclear weapons to destroy an irrecoverable portion of their infrastructure, especially considering the fact that their economy was already blockaded off from potential sanctions.

Raemon is right that you cannot allow the proliferation of superintelligent AIs (because those AIs will allow you to cheaply produce powerful weapons). To stop this from happening ~permanently, you do probably need a single actor or very small coalition of actors to enforce that non-proliferation forever, likely through using their first to ASI position to permanently monopolize it and box out new entrants.
While the existence of this coalition would necessarily reduce the flexibility of the future, it would probably look a lot more like the IAEA and less like a preemptive nuclear holocaust. The only AI capabilities that need to be restricted are those related to weapons development, which means that every other non-coalition actor still gets to grab the upside of most AI applications. Analogously, the U.N security council have been largely successful at preventing nuclear proliferation to other countries by using their collective economic, political, and strategic position, while still allowing beneficial nuclear technology to be widely distributed. You can let the other countries build nuclear power plants, so long as you use your strategic influence to make sure they’re not enrichment facilities.
In practice, I think this (ideally) ends up looking something like the U.S and China agreeing on further non-proliferation of ASI, and then using their collective DSA over everybody else to monopolize the AI supply chain. From there, you can put a bunch of hardware-bound restrictions, mandatory verification/monitoring for data centers, and backdoors into every new AI application to make sure they’re aligned to the current regime. There’s necessarily a lot of concentration of power, but that’s only because it explicitly trades off with the monopoly of violence (ie, you can’t just give more actors more actors access to ASI weapons capabilities for self-determination without losing overall global security, same as with nukes).

I’m currently writing up a series of posts on the strategic implications of AI proliferation, so I’ll have a much more in-depth version of this argument here in a few weeks. I’m also happy to dm/call directly to talk about this in more detail!

Felix Choussat 2 Oct 2025 16:15 UTC
4 points
0
in reply to: StanislavKrym’s comment on: AI and Cheap Weapons
I’ll talk more about this in follow up posts, but I don’t think the main danger is that the models will be voluntarily released. Instead, it’ll just get cheaper and cheaper to train the models that have weapons capabilities as the algorithms get more efficient, which will eventually democratize those weapons.

Analogously, we can think about how cryptography was once a government controlled technology because of its strategic implications, but became widespread as the computing power required to host cryptographic algorithms became extremely cheap.

Felix Choussat 1 Oct 2025 15:42 UTC
1 point
0
in reply to: TAG’s comment on: [Question] What the discontinuity is, if not FOOM?
I’ll make the point that safety engineering can have discontinuous failure modes. The reason the Challenger collapsed was because some o-ring seals in a booster had gotten too cold before launch, preventing them from sealing off the flow of hot gas to the main engine and blowing up the rocket. The function of these o-rings is pretty binary: either gas is kept in and the rocket works, or it’s let out and the whole thing explodes.

AI research might end up with similar problems. It’s probably true that there is such a thing as good enough alignment, but that doesn’t necessarily imply that progress on solving it can be made incrementally and doesn’t have all or nothing stakes in deployment.

Felix Choussat 28 Sep 2025 21:28 UTC
1 point
0
on: Felix C.’s Shortform
Is there a way to bring over the lesswrong format (the left sidebar table of contents, the floating citations off to the right) onto a personal blog? As far as I can tell the AI 2027 project forecasts are similar in structure, so it seems like someone’s already managed it before.
I love the lesswrong formatting and plan to crosspost between my blog and here, but I want to be able to share the work either without the immediate connotations of lesswrong (for some policy audiences) or with personal connotations through my blog (as part of a job search, for instance).