There are two elements to what you were asking the model to do:
1) Generate a random number without using is normakl built in source of stochasiticity, the token selection process. So you’re requiring it to have an internal pseudorandom number generator algorithm. Which it might simply not have. Wht would it need one? It has a random number generator built in, every time it generates a token.
2) Represent and store a number from 1 to 100 in its internal activations without actually outputting the tokens for it. As in, output a string of tokens like:
”OK, I thought of a number.”
and somewhere in the set of activations on those tokens, at a some layer on some token (maybe the token “number”, maybe the full stop of the sentence) encode that specific number in a way that subsequent activation heads can read from. The only issue here being, if that activation is at a late (but not final) layer, only attention heads at that late layer can attend to it, so the processing that the model is later able to do on that number is thus limited.
You’ve demonstrated that it fails. So, is that because:
a) it doesn’t have a suitable pseudorandom number generator, and you forbade it from using its normal solution of using stochastic token generation to genrate randomness, so it can’t pick a number in the first place — the problem is generation, not storage? b) it doesn’t have a way to represent the numbers 1 to 100 in its activation space, and is thus “not conscious”? (your claim) c) it does, but only at a late layer that limits the processing that it can subsequently do on that data, since it never emitted it as a token?
If c) were the case, then the model probably could consistently print the number on replay from after the end of that sentence, but could no play complex 20 questions abouit it. Have you tried that?
My suspicion is that the problem is a). But until you can rule out a) and c), you haven’t proven b).
The fact remains that a modern reasoning model with CoT could, and I’m sure would, pass this test: it would emit an semi-random number from 1 to 100 into its CoT and then be able to refer back to it consistently. So it would have a legible inner monolog, and once it had emitted the number into the CoT, the value of the number would remain accessible and fixed under replay. So even if you prove b), and thus that models a couple of years ago were not “conscious” by your chosen definition, more recent models are: we can inspect their Chin of Thought, and it clearly passes your criterion.
Again no, that is missing the point. (Although I agree that this one is not very good experiment).
The reasoning goes like that, they sometimes say e.g. “It felt frightening!”. Did it feel frightening? Or is this what you are supposed to say here, because it’s appropriate thing to say in such situation?
And then its (lack of) skill of introspection becomes relevant.
So even if you prove b), and thus that models a couple of years ago were not “conscious” by your chosen definition
You might be confusing me with OP, I did not indicate that I have any such chosen definition. Or whatever.
There are two elements to what you were asking the model to do:
1) Generate a random number without using is normakl built in source of stochasiticity, the token selection process. So you’re requiring it to have an internal pseudorandom number generator algorithm. Which it might simply not have. Wht would it need one? It has a random number generator built in, every time it generates a token.
2) Represent and store a number from 1 to 100 in its internal activations without actually outputting the tokens for it. As in, output a string of tokens like:
”OK, I thought of a number.”
and somewhere in the set of activations on those tokens, at a some layer on some token (maybe the token “number”, maybe the full stop of the sentence) encode that specific number in a way that subsequent activation heads can read from. The only issue here being, if that activation is at a late (but not final) layer, only attention heads at that late layer can attend to it, so the processing that the model is later able to do on that number is thus limited.
You’ve demonstrated that it fails. So, is that because:
a) it doesn’t have a suitable pseudorandom number generator, and you forbade it from using its normal solution of using stochastic token generation to genrate randomness, so it can’t pick a number in the first place — the problem is generation, not storage?
b) it doesn’t have a way to represent the numbers 1 to 100 in its activation space, and is thus “not conscious”? (your claim)
c) it does, but only at a late layer that limits the processing that it can subsequently do on that data, since it never emitted it as a token?
If c) were the case, then the model probably could consistently print the number on replay from after the end of that sentence, but could no play complex 20 questions abouit it. Have you tried that?
My suspicion is that the problem is a). But until you can rule out a) and c), you haven’t proven b).
The fact remains that a modern reasoning model with CoT could, and I’m sure would, pass this test: it would emit an semi-random number from 1 to 100 into its CoT and then be able to refer back to it consistently. So it would have a legible inner monolog, and once it had emitted the number into the CoT, the value of the number would remain accessible and fixed under replay. So even if you prove b), and thus that models a couple of years ago were not “conscious” by your chosen definition, more recent models are: we can inspect their Chin of Thought, and it clearly passes your criterion.
Again no, that is missing the point. (Although I agree that this one is not very good experiment).
The reasoning goes like that, they sometimes say e.g. “It felt frightening!”. Did it feel frightening? Or is this what you are supposed to say here, because it’s appropriate thing to say in such situation?
And then its (lack of) skill of introspection becomes relevant.
You might be confusing me with OP, I did not indicate that I have any such chosen definition. Or whatever.
You’re right, I have mistakenly assumed you were the OP replying