Marco Bazzani

• Marco Bazzani 30 Sep 2025 3:05 UTC

  −1 points

  −5

  in reply to: Eliezer Yudkowsky’s comment on: A non-re­view of “If Any­one Builds It, Every­one Dies”

  Here’s my attempted phrasing, which might be avoid some of the common confusions:
  
  Suppose we have a model $M$ with utility function $\varphi$, where $M$ is not capable of taking over the world. Assume that thanks to a bunch of alignment work, $\varphi$ is within $\delta$ (by some metric) of humanity’s collective utility function. Then in the process of maximizing $\varphi$, $M$ ends up doing a bunch of vaguely helpful stuff.
  
  Then someone releases model $M^{\prime }$ with utility function ${\varphi }^{\prime }$, where $M^{\prime }$ is capable of taking over the world. Suppose that our alignment techniques generalize perfectly. That is, ${\varphi }^{\prime }$ is also within $\delta$ of humanity’s collective utility function. Then in the process of maximizing ${\varphi }^{\prime }$, $M^{\prime }$ gets rid of humans and rearranges their molecules to satisfy ${\varphi }^{\prime }$ better.
  
  Does this phrasing seem accurate and helpful?