Excellent point. It will be a few posts (if the audience is interested) before I can answer you in a way that is both intuitive and fully convincing.
The technical answer is that the belief update caused by an action is deterministically contractive. It never increases the amount of information.
A more intuitive answer (but perhaps not yet convincing) is that, proximally, your action of asking your friend did not change the location of your laptop, only your friend’s mental state. And the effect it had on your friend’s mental state is that you are now less sure of what it is. You took some of the things you knew about it (“she was probably going to keep sitting around”) and made them no longer true.
Regarding information about the future (of the location of your laptop), it is always included in information about the present. Your own mental state is independent of the future given the present. Put another way, you can’t update without more observations. In this case, the location of your laptop merely becomes entangled with information that you already had on your friend’s mental state (“this is where she thinks my desk is”).
What is ‘amount of information’ ? If I do not know if coin is heads or tails, then I have 0 bits of information about state of the coin, if I open my eyes and see it is heads, I have 1 bit. The information is in narrowing of the possibilities. That is conventional meaning. edit: though of course the information is not increased until next perception—perhaps that is what you meant? edit: still, there is a counter example—you can have axially magnetized coin, and electromagnet that can make the coin flip to heads up when its powered. You initially don’t know which way the coin is up, but if the action is to magnetize the electromagnet, you will have the coin be heads up. (Of course the overall entropy of world still did go up, but mostly in form of heat). One could say that it doesn’t increase knowledge of environment, but decreases the entropy in environment.
You are expressing a number of misconceptions here. I may address some in future posts, but in short:
By information I mean the Shannon information (see also links in OP). Your example is correct.
By the action of powering the electromagnet you are not increasing your information on the state of the world. You are increasing your information on the state of the coin, but through making it dependent on the state of the electromagnet which you already knew. This point is clearly worth a future post.
There is no “entropy in environment”. Entropy is subjective to the viewer.
I think it is mostly a matter of definitions. I am not familiar with your terminology. Also, if I have an atom that is in unknown alignment, and I align it using magnetic field, then take away the resulting heat, then the entropy (number of states) of that subsystem decreases, and this is used to attain extremely low temperatures. I am more familiar with the physical notion of entropy.
edit: Also, after powering electromagnet, I know that the direction of coin and direction of electromagnet relate in a particular way, which I did not know before. At the same time, I have physically restricted the number of states that the environment can be in—the coin can not now be other way around. It’s in this sense that entropy of environment (as seen on large scale) decreases . (and it is of course subjective, that the number of states that environment can be in, decreases. It does not decrease according to agent that already knows which way the coin is)
To pick a trivial case: A blind person with acute hearing taps a cane on the floor in order to ascertain, from echoes, the relative positions of nearby objects.
The issue is that “action” and “observation” can be entangled; your description of observation makes it into a passive process, ignoring the role of activity in observation. “Step one of my plan: Figure out where the table is so I don’t run into it.” Which is to say, your pattern is overly rigid.
You might argue that the tapping of the cane is itself an observation, in which case you’d also have to treat walking into a room to see what’s in it as an observation; the former removes no information, but the latter reduces your certainty of the positions of objects in the room you’ve just left, meaning either actions can generate information, or observations can reduce it. You could preserve the case that actions cannot generate information if you instead treat hearing the echoes as a secondary observation, but this still leaves you with the case that an action did not, in fact, eliminate any information.
I realize now that an example would be helpful, and yours is a good one.
Any process can be described on different levels. The trick is to find a level of description that is useful. We make an explicit effort to model actions and observation so as to separate the two directions of information flow between the agent and the environment. Actions are purely “active” (no information is received by the agent) while observations are purely “passive” (no information is sent by the agent). We do this because these two aspects of the process have very different properties, as I hope to make clear in future posts.
The process of “figuring out where the table is” involves information flowing in both directions, and so is neither an action nor an observation. Some researchers call such a thing “a subgoal”. We should break it down further, for example we could have taps as actions and echoes as observations, as you suggest.
If you want to argue that no information is lost by tapping, then fine, I won’t be pedantic and point out the tiny bits of information that do get lost. The point is that some information being lost is a representing feature of the process of taking an action. Over time, if you don’t take in new information through observations, your will have less and less information about the world, even if some actions you take have a high probability of not losing any information.
This would be true regardless of whether you engaged in any action at all, however. The passing of time since your last verification of a piece of information is that by which information is lost.
I’m assuming this model is AI-related, so my responses are going to be in line with information modeling with that in mind. If this isn’t accurate, let me know.
I would, indeed, suggest time since last verification as the mechanism in your model for information contraction, rather than action; assigning a prior probability that your information will remain accurate does a good job of completing the model. Imagine memorizing a room, closing your eyes, and firing a canon into the room. Contemporaneous to your action, your information is still valid. Shortly thereafter, it ceases to be in a rather dramatic way. Importantly for your model, I think, this is so regardless of whether you fire the canon, or another agent does. If it’s a soundproof room, and you close the door with another agent inside, your information about the state of the room can contract quite violently through no action of your own.
Excellent point. It will be a few posts (if the audience is interested) before I can answer you in a way that is both intuitive and fully convincing.
The technical answer is that the belief update caused by an action is deterministically contractive. It never increases the amount of information.
A more intuitive answer (but perhaps not yet convincing) is that, proximally, your action of asking your friend did not change the location of your laptop, only your friend’s mental state. And the effect it had on your friend’s mental state is that you are now less sure of what it is. You took some of the things you knew about it (“she was probably going to keep sitting around”) and made them no longer true.
Regarding information about the future (of the location of your laptop), it is always included in information about the present. Your own mental state is independent of the future given the present. Put another way, you can’t update without more observations. In this case, the location of your laptop merely becomes entangled with information that you already had on your friend’s mental state (“this is where she thinks my desk is”).
I’m interested.
I’m interested too, mostly because I don’t don’t find royf’s current explanation convincing.
What is ‘amount of information’ ? If I do not know if coin is heads or tails, then I have 0 bits of information about state of the coin, if I open my eyes and see it is heads, I have 1 bit. The information is in narrowing of the possibilities. That is conventional meaning. edit: though of course the information is not increased until next perception—perhaps that is what you meant? edit: still, there is a counter example—you can have axially magnetized coin, and electromagnet that can make the coin flip to heads up when its powered. You initially don’t know which way the coin is up, but if the action is to magnetize the electromagnet, you will have the coin be heads up. (Of course the overall entropy of world still did go up, but mostly in form of heat). One could say that it doesn’t increase knowledge of environment, but decreases the entropy in environment.
You are expressing a number of misconceptions here. I may address some in future posts, but in short:
By information I mean the Shannon information (see also links in OP). Your example is correct.
By the action of powering the electromagnet you are not increasing your information on the state of the world. You are increasing your information on the state of the coin, but through making it dependent on the state of the electromagnet which you already knew. This point is clearly worth a future post.
There is no “entropy in environment”. Entropy is subjective to the viewer.
I think it is mostly a matter of definitions. I am not familiar with your terminology. Also, if I have an atom that is in unknown alignment, and I align it using magnetic field, then take away the resulting heat, then the entropy (number of states) of that subsystem decreases, and this is used to attain extremely low temperatures. I am more familiar with the physical notion of entropy.
edit: Also, after powering electromagnet, I know that the direction of coin and direction of electromagnet relate in a particular way, which I did not know before. At the same time, I have physically restricted the number of states that the environment can be in—the coin can not now be other way around. It’s in this sense that entropy of environment (as seen on large scale) decreases . (and it is of course subjective, that the number of states that environment can be in, decreases. It does not decrease according to agent that already knows which way the coin is)
To pick a trivial case: A blind person with acute hearing taps a cane on the floor in order to ascertain, from echoes, the relative positions of nearby objects.
The issue is that “action” and “observation” can be entangled; your description of observation makes it into a passive process, ignoring the role of activity in observation. “Step one of my plan: Figure out where the table is so I don’t run into it.” Which is to say, your pattern is overly rigid.
You might argue that the tapping of the cane is itself an observation, in which case you’d also have to treat walking into a room to see what’s in it as an observation; the former removes no information, but the latter reduces your certainty of the positions of objects in the room you’ve just left, meaning either actions can generate information, or observations can reduce it. You could preserve the case that actions cannot generate information if you instead treat hearing the echoes as a secondary observation, but this still leaves you with the case that an action did not, in fact, eliminate any information.
I realize now that an example would be helpful, and yours is a good one.
Any process can be described on different levels. The trick is to find a level of description that is useful. We make an explicit effort to model actions and observation so as to separate the two directions of information flow between the agent and the environment. Actions are purely “active” (no information is received by the agent) while observations are purely “passive” (no information is sent by the agent). We do this because these two aspects of the process have very different properties, as I hope to make clear in future posts.
The process of “figuring out where the table is” involves information flowing in both directions, and so is neither an action nor an observation. Some researchers call such a thing “a subgoal”. We should break it down further, for example we could have taps as actions and echoes as observations, as you suggest.
If you want to argue that no information is lost by tapping, then fine, I won’t be pedantic and point out the tiny bits of information that do get lost. The point is that some information being lost is a representing feature of the process of taking an action. Over time, if you don’t take in new information through observations, your will have less and less information about the world, even if some actions you take have a high probability of not losing any information.
This would be true regardless of whether you engaged in any action at all, however. The passing of time since your last verification of a piece of information is that by which information is lost.
I’m assuming this model is AI-related, so my responses are going to be in line with information modeling with that in mind. If this isn’t accurate, let me know.
I would, indeed, suggest time since last verification as the mechanism in your model for information contraction, rather than action; assigning a prior probability that your information will remain accurate does a good job of completing the model. Imagine memorizing a room, closing your eyes, and firing a canon into the room. Contemporaneous to your action, your information is still valid. Shortly thereafter, it ceases to be in a rather dramatic way. Importantly for your model, I think, this is so regardless of whether you fire the canon, or another agent does. If it’s a soundproof room, and you close the door with another agent inside, your information about the state of the room can contract quite violently through no action of your own.