I think proponents of the instrumental convergence thesis would expect a consequentialist chess program to exhibit instrumental convergence in the domain of chess. So if there were some (chess-related) subplan that was useful in lots of other (chess-related) plans, we would see the program execute that subplan a lot. The important difference would be that the chess program uses an ontology of chess while unsafe programs use an ontology of nature.
First, Nick Bostrom has an example where a Riemann hypothesis solving machine converts the earth into computronium. I imagine he’d predict the same for a chess program, regardless of what ontology it uses.
Second, if instrumental convergence was that easy to solve (the convergence in the domain of chess is harmless), it wouldn’t really be an interesting problem.
I think proponents of the instrumental convergence thesis would expect a consequentialist chess program to exhibit instrumental convergence in the domain of chess. So if there were some (chess-related) subplan that was useful in lots of other (chess-related) plans, we would see the program execute that subplan a lot. The important difference would be that the chess program uses an ontology of chess while unsafe programs use an ontology of nature.
First, Nick Bostrom has an example where a Riemann hypothesis solving machine converts the earth into computronium. I imagine he’d predict the same for a chess program, regardless of what ontology it uses.
Second, if instrumental convergence was that easy to solve (the convergence in the domain of chess is harmless), it wouldn’t really be an interesting problem.