My new concept for “pivotal act that stops the world from getting to ASI, even though we get to AGI” is a soft-power act of better coordination. Get help from AGI to design and deploy decentralized governance tech that allows humanity (and AIs) to coordinate on escaping the trap of suicide-race.
Once you start getting involved with governance, you’re going to need law enforcement and defense, which is an adversarial context and thus means the whole instrumental goal niceness argument collapse.
My new concept for “pivotal act that stops the world from getting to ASI, even though we get to AGI” is a soft-power act of better coordination. Get help from AGI to design and deploy decentralized governance tech that allows humanity (and AIs) to coordinate on escaping the trap of suicide-race.
Once you start getting involved with governance, you’re going to need law enforcement and defense, which is an adversarial context and thus means the whole instrumental goal niceness argument collapse.