To reason sensibly under abnormal conditions like mind copying or dealing with perfect predictors, we first separate the things that we control and which can be kept constant through all the situations in the problem (the program) from the things which we can’t control or which vary in the course of the problem (the state/data).
Which things we want to hold constant and which things vary depend on the problem we’re considering. If we’re trying to choose a strategy for a game—the Iterated Prisoner’s Dilemma, for example—then the program is a complete strategy which we assume is memorized before the beginning and followed perfectly, and the state is all the observations made between the start of the game and some decision point within it. Problems may force us to move things that are normally part of the program into the state, by taking them out of our control. For example, when reasoning about how a company should act in relation to a market, we treat everything that decides what the corporation does as a black box program, and the observations it makes of the market as its input data. If internal politics matter, then we have to narrow the black-boxing boundary to only ourselves. If we’re worried about procrastination, then we draw the boundary inside our own mind.
Whether something is Program or Data is not a property of the object itself, but rather of how we reason about it. If it can be fully modeled as a black box function, then it’s part of the program; otherwise it’s data.
To reason sensibly under abnormal conditions like mind copying or dealing with perfect predictors, we first separate the things that we control and which can be kept constant through all the situations in the problem (the program) from the things which we can’t control or which vary in the course of the problem (the state/data).
Which things we want to hold constant and which things vary depend on the problem we’re considering. If we’re trying to choose a strategy for a game—the Iterated Prisoner’s Dilemma, for example—then the program is a complete strategy which we assume is memorized before the beginning and followed perfectly, and the state is all the observations made between the start of the game and some decision point within it. Problems may force us to move things that are normally part of the program into the state, by taking them out of our control. For example, when reasoning about how a company should act in relation to a market, we treat everything that decides what the corporation does as a black box program, and the observations it makes of the market as its input data. If internal politics matter, then we have to narrow the black-boxing boundary to only ourselves. If we’re worried about procrastination, then we draw the boundary inside our own mind.
Whether something is Program or Data is not a property of the object itself, but rather of how we reason about it. If it can be fully modeled as a black box function, then it’s part of the program; otherwise it’s data.