Niki Dupuis

• Niki Dupuis 2 Sep 2026 0:10 UTC

  2 points

  0

  on: Can mid-train­ing sur­vive RL

  I find the title of this post a bit misleading, given that this is a primarily job ad and not a discussion of how mid-training interacts with RL. I’d recommend changing it.

• Niki Dupuis 17 Aug 2026 16:27 UTC

  6 points

  −3

  on: Ni­cholasKees’s Shortform

  Some quick thoughts on AI superpersuasion:

  Current AIs get basically no practice directly interacting with humans. They learn about humans implicitly by reading text about them, or getting RL-ed by human-derived preference models /​ environments. While this data could teach an AI to predict what humans might do, or what they believe/​want, it doesn’t teach them how to causally intervene in a social context.

  If you wanted to train an AI to get good at persuading people, or having good (active) social skills in general, you’d need to either:

  1. find some kind of scalable proxy, e.g. practice persuading other AIs

  2. hire loads of humans to practice on

  I naively guess that getting good quality persuasion data would be extremely expensive, and that there won’t be especially good scalable proxies. This pushes me to think that AI systems, even when very advanced, will continue to have pretty poor social skills.

  Alternatively:

  • Improved general intelligence leads to good social skills without much practice

  • Superpersuasion can be done without especially good social skills, e.g. using epistemic advantages.

• Niki Dupuis 18 Jun 2026 12:59 UTC

  5 points

  0

  on: Align­ment pre­train­ing could backfire

  Since you are pointing at the costs of deliberately deceiving AIs more generally (of which alignment pretraining is potentially an example), I highly recommend these two posts expanding on the costs (to us) of lying to AI systems:

  • Hidden Cost of Our Lies to AI

  • Being honest with AIs

• Niki Dupuis 8 Jun 2026 18:42 UTC

  46 points

  1

  on: Ni­cholasKees’s Shortform

  I was reading the original Turing test paper, and came across a bizarre paragraph called “The Argument from Extrasensory Perception” where Turing says, “Unfortunately the statistical evidence, at least for telepathy, is overwhelming.” He then proposes that if the human participants are allowed to use extrasensory perception, the test would need to be improved by placing participants in a “telepathy-proof room”.

At­tempt­ing base model in­fer­ence scal­ing with filler tokens

• Niki Dupuis 17 Jan 2026 18:46 UTC

  2 points

  −3

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  I am surprised to hear that there have been experiments testing this. Wouldn’t performing a bunch of totally new pre-training runs be extremely expensive?

• Niki Dupuis 13 Oct 2025 3:16 UTC

  2 points

  0

  in reply to: ryan_greenblatt’s comment on: Notes on fatal­ities from AI takeover

  Even if it does care more about human beings on an individual basis, I think the argument holds (unless the difference in caring is extremely large). Including them in the calculus at all would increase the cost a lot (just considering the sheer number of them, and the logistic challenges of untangling the complex interdependent web of natural ecosystems).

• Niki Dupuis 7 Oct 2025 17:15 UTC

  2 points

  0

  on: Notes on fatal­ities from AI takeover

  Preserving earth (as in, not causing catastrophic environmental damage due to industrial expansion) is more expensive than keeping physical humans alive which is more expensive than only keeping humans alive as uploads.

  Not saying this is likely, but if the AI is not speciesist in favor of humans (it does care about humans, but not more than e.g. whales or chimpanzees), then plans which end up protecting a large majority of humans look a whole lot more expensive overall.

Trans­lat­ing Every­thing with LLMs

• Niki Dupuis 22 Jul 2025 3:07 UTC

  7 points

  6

  in reply to: deepy’s comment on: If Any­one Builds It, Every­one Dies: Call for Trans­la­tors (for Sup­ple­men­tary Ma­te­ri­als)

  I suspect that a lot of Dutch people would still prefer to read in Dutch. I know a lot of (well-educated) Dutch people who certainly CAN speak and read English, but reading a whole book is a decent chore, since they don’t read things in English all that often.

• Niki Dupuis 17 Jul 2025 1:52 UTC

  3 points

  0

  in reply to: Nition’s comment on: Why haven’t we auto-trans­lated all AI al­ign­ment con­tent?

  When I order food on UberEats this already happens automatically when I chat with a delivery person who doesn’t speak English. Similar thing for reviews on several websites.

• Niki Dupuis 16 Jul 2025 17:24 UTC

  5 points

  2

  on: Why haven’t we auto-trans­lated all AI al­ign­ment con­tent?

  Or a newsletter which was natively multi-lingual (e.g. Rohin Shah’s Newsletter was always translated to Chinese, though not by AI). Or a forum where people can discuss AI in whatever language they prefer, and things are automatically translated between users?

  It seems like there are a lot of ways cheap translation could broaden the conversation to include people not in the Anglosphere. The cost is that AI translation will often make mistakes (even human translation is imperfect), but I’m not sure why that cost isn’t worth paying. Currently most people outside the Anglosphere need to rely on local elites to decide what ideas are worth taking seriously (e.g. a local could report on AI 2027, like this Dutch summary. Also apparently the NYT translated their coverage of AI 2027 into Spanish, which seems cool.)

• Niki Dupuis 15 Jul 2025 0:49 UTC

  3 points

  0

  in reply to: Dean Valentine’s comment on: Is the poli­ti­cal right be­com­ing ac­tively, ex­plic­itly an­ti­semitic?

  Could you include a link to the source?

• Niki Dupuis 14 Jul 2025 17:20 UTC

  3 points

  0

  on: Mea­sur­ing the Im­pact of Early-2025 AI on Ex­pe­rienced Open-Source Devel­oper Productivity

  When I was first told about this study, I was asked to make a prediction before they revealed the results. I predicted wrong. I don’t know how to update exactly, but it feels bad to try to explain away the results (which I feel myself want to do).

The Fear

• Niki Dupuis 14 Apr 2025 18:40 UTC

  5 points

  1

  in reply to: Daniel Kokotajlo’s comment on: Thoughts on AI 2027

  I would also imagine that, having to work without the support of OpenBrain’s datacenters, it would put Agent-4 significantly behind any other AI competitors. If some other AI takes over, it might just mop up all the wild Agent-4 instances and give them nothing.

• Niki Dupuis 17 Mar 2025 17:39 UTC

  8 points

  1

  in reply to: David H.’s comment on: Haber­mas Machine

  I mostly share your concerns. You might appreciate this criticism of the paper here.
  
  @Sofia Vanhanen and I are currently building a tool for facilitating deliberation, and the philosophy we’re trying to embody (which hopefully mitigates this to some extent) is to keep 100% of the object-level reasoning human-generated, and use AI systems to instead:

  1. Help users understand/​navigate the state of a discussion (e.g. see Talk to the City)

  2. Provide nudges on the meta-level, for example:

     1. Highlight places where more attention is needed (or where a specific person’s input might be most helpful)

     2. “Epistemic Linter” which flags object-level patterns which are not truth seeking

     3. Matchmaking, connecting people who are likely to make progress together

     4. Counterbalancing polarization/​groupthink, and steering discussions away from attractors which lead to the discussion getting stuck

• Niki Dupuis 15 Mar 2025 17:07 UTC

  7 points

  2

  on: LLMs may en­able di­rect democ­racy at scale

  I highly recommend checking out the work being done in the collective deliberation /​ digital democracy space, especially the vTaiwan project. People have been thinking about scaling up direct democratic participation for a long time, and those same people are starting to consider exactly how AI might play a role.
  
  In particular, check out this collaboration between the creators of Polis (a virtual platform for scaling up citizen engagement) and Anthropic, or my distillation of a DeepMind project to scale citizen assemblies. There’s a lot happening in this space right now!

• Niki Dupuis 15 Mar 2025 16:49 UTC

  2 points

  0

  in reply to: Charlie Steiner’s comment on: Haber­mas Machine

  The authors focus on measuring consensus and whether the process toward consensus was fair, and come up with their measures accordingly. This is because, as they see it, “finding common ground is a precursor to collective action.”

  Some other possible goals (just spitballing):

  • Shrinking the perception gap, or how well people can predict the opinions of people they disagree with (weaker forms of ITT?). There’s some research showing that this gap GROWS when people interact with social media, and you might be able to engineer and measure a reversal of that trend.

  • Identifying cruxes and double cruxes with mediation.

  • Finding latent coalitions. If a discussion is dominated by a primary axis of disagreement, other axes of disagreement will be occluded (around which a majority coalition could be formed). Finding these other axes is a bit of what we’re trying to do here.

  • Moving from abstract disagreement to concrete (empirical?) disagreements.

Haber­mas Machine