Of course, ‘how we want the calculator to work’ is just a stand-in that represents not a subgoal of our utility function, but a referent to something outside us we perceive as more objective than what we can see of ourselves. A broken calculator is not wrong because the number it spits out isn’t the number we were hoping it would spit out when we started calculating our finances, nor because being misled about such a number would endanger our finances further. (That doesn’t mean we should turn the universe into a big calculator, of course; unless it’s a calculator that knows how to find and calculate the the most objective and elegant calculations, and not just arbitrary addition. Then maybe.)
Of course, ‘how we want the calculator to work’ is just a stand-in that represents not a subgoal of our utility function, but a referent to something outside us we perceive as more objective than what we can see of ourselves. A broken calculator is not wrong because the number it spits out isn’t the number we were hoping it would spit out when we started calculating our finances, nor because being misled about such a number would endanger our finances further. (That doesn’t mean we should turn the universe into a big calculator, of course; unless it’s a calculator that knows how to find and calculate the the most objective and elegant calculations, and not just arbitrary addition. Then maybe.)