Suppose we have an agent A trying to optimise for a reward R in an environment S.
How can we tell that the presence of the agent does not affect the environment and the measurement(observation) is not only subject to the agent but the environment?
This is related with the measurement problem in quantum computing, we have an agent (a particle ) entangled in a quantum superposition, consider an electron with two possible configurations: up and down, , when we measure the state, it collapses the wavefunction to a particular classical state, up or down.
Moreover, the observer effect, notes that measurements of certain systems cannot be made without affecting the system.
While the uncertainty principle argues that we cannot predict the value of a quantity with arbitrary certainty.
Another way to state the problem is, does measuring the state of the action affect the state of the environment?
How are the observations in physics different from the observations we make in RL?
Is the environment state at causal to the state in ?
Simply asking “does it affect the environment” is not enough here.
Leaking, say, 1pJ of thermal energy to the surroundings in a slightly different manner in the two cases technically affects the surroundings quite significantly—thermal motion in a gas is chaotic after all—but in practice we would tend to call this minimal[1] effects on the environment.
Assuming a human-livable environment at least. Obviously this might be different if e.g. your environment is all at 2 picoKelvin or somesuch.