You need at least a certain amount of transfer in order to actually do your pivotal act. An “AI” with literally zero transfer is just a lookup table. The point of this principle is that you want as little transfer as possible while still managing a pivotal act. I used a theorem proving AI as an example where it’s really easy to see what would count as unnecessary transfer. But even with something whose pivotal act would require a lot more transfer than a theorem prover (say, a nanosystem builder AI), you’d still want to avoid transfer to domains such as deceiving humans or training other AIs.
You need at least a certain amount of transfer in order to actually do your pivotal act. An “AI” with literally zero transfer is just a lookup table. The point of this principle is that you want as little transfer as possible while still managing a pivotal act. I used a theorem proving AI as an example where it’s really easy to see what would count as unnecessary transfer. But even with something whose pivotal act would require a lot more transfer than a theorem prover (say, a nanosystem builder AI), you’d still want to avoid transfer to domains such as deceiving humans or training other AIs.