Because agents with good values can, quote, “steal the strategies” that other agents pursue. This is a term from an essay by Paul Christiano, I think…
The term “strategy stealing” originated in pure math (specifically, Combinatorial Game Theory). See Strategy-stealing argument (Wikipedia). That wikipedia article doesn’t say exactly when the term was coined … probably the 1960s? Then of course Paul brought that term into AGI discourse in 2019.
The term “strategy stealing” originated in pure math (specifically, Combinatorial Game Theory). See Strategy-stealing argument (Wikipedia). That wikipedia article doesn’t say exactly when the term was coined … probably the 1960s? Then of course Paul brought that term into AGI discourse in 2019.