Now this is an interesting thought. Even a satisficer with several goals but no upper bound on each will use all available matter on the mix of goals it’s working towards. But a limited goal (make money for GiantCo, unless you reach one trillion, then stop) seems as though it would be less dangerous. I can’t remember this coming up in Eliezer’s CFAI document, but suspect it’s in there with holes poked in its reliability.
Now this is an interesting thought. Even a satisficer with several goals but no upper bound on each will use all available matter on the mix of goals it’s working towards. But a limited goal (make money for GiantCo, unless you reach one trillion, then stop) seems as though it would be less dangerous. I can’t remember this coming up in Eliezer’s CFAI document, but suspect it’s in there with holes poked in its reliability.