That’s not the point of this experiment. This experiment is aimed at testing different thing, whether when they tell you about their internal states / emotions, they have anything in there. And plausibly they don’t for the most part. Analogous to LLM telling it has a number in mind while it has actually no number in mind, and doing the impromptu role playing to bullshit you.
There are some subtler points here, like LLM is a kind of array of tubes, maybe the tubes individually feel something. Maybe they do coherently feel something together, but not the thing it has to tell you, as it’s appropriate thing to tell in this situation etc. Maybe something else.
I think what you’ve established is that LLMs weren’t very good at this a couple of years ago, before reasoning models. Humans have an inner monolog. So do reasoning models. Pre-reasoning models don’t. If you’re a sufficiently good meditator to be able to do this, try to pick a random number without internally saying it, visualizing it, specifying “the same number as my birthday” or otherwise committing to it in any symbolic form whatsoever.
Specifically, what I think you have demonstrated is that the model you were testing doesn’t have separate almost-orthogonal activation directions for all 100 numbers between 1 and 100 plus the ability to generate a sharp random distribution of one and only one of them. That’s mildly surprising, but I don’t think it proves lack of consciousness. Given that the only stochastic element in an LLM is the token selection, not letting it generate a token may be blocking the random distribution part of the task. Which would mean that it’s actually implemented lazily, when it does generate tokens to answer your questions, and by resetting it to before that you’re redoing the lazy generation. That’s a valid algorithm, just not the one a human with an inner monolog would use. And the test you are using clearly could be passed by a reasoning model putting tokens for the number into its CoT, so by your logic and definition of the word, that model is conscious.
Personally I don’t see arguing about what definition we want to use for the word “conscious” as very productive, nor as relevant to questions about AI ethics or welfare or alignment, but I know some people do.
They do! They have cashes and they can control them and read from them! Isn’t it surprising? You can ask them to think about bread and say duck, and look at their internals when they say duck and see that they thought “bread”. If they were a bit better at it they would have been able to think that number without saying it and then look back at that thought.
The Point of the experiment, that you missed, is that they sometimes tell that they felt something or whatever, and the question is, did they? Would they know?
so by your logic and definition of the word, that model is conscious.
Not my logic and definitions. I’m arguing about local point, that particular experiment, I have disagreement with you about.
There are two elements to what you were asking the model to do:
1) Generate a random number without using is normakl built in source of stochasiticity, the token selection process. So you’re requiring it to have an internal pseudorandom number generator algorithm. Which it might simply not have. Wht would it need one? It has a random number generator built in, every time it generates a token.
2) Represent and store a number from 1 to 100 in its internal activations without actually outputting the tokens for it. As in, output a string of tokens like:
”OK, I thought of a number.”
and somewhere in the set of activations on those tokens, at a some layer on some token (maybe the token “number”, maybe the full stop of the sentence) encode that specific number in a way that subsequent activation heads can read from. The only issue here being, if that activation is at a late (but not final) layer, only attention heads at that late layer can attend to it, so the processing that the model is later able to do on that number is thus limited.
You’ve demonstrated that it fails. So, is that because:
a) it doesn’t have a suitable pseudorandom number generator, and you forbade it from using its normal solution of using stochastic token generation to genrate randomness, so it can’t pick a number in the first place — the problem is generation, not storage? b) it doesn’t have a way to represent the numbers 1 to 100 in its activation space, and is thus “not conscious”? (your claim) c) it does, but only at a late layer that limits the processing that it can subsequently do on that data, since it never emitted it as a token?
If c) were the case, then the model probably could consistently print the number on replay from after the end of that sentence, but could no play complex 20 questions abouit it. Have you tried that?
My suspicion is that the problem is a). But until you can rule out a) and c), you haven’t proven b).
The fact remains that a modern reasoning model with CoT could, and I’m sure would, pass this test: it would emit an semi-random number from 1 to 100 into its CoT and then be able to refer back to it consistently. So it would have a legible inner monolog, and once it had emitted the number into the CoT, the value of the number would remain accessible and fixed under replay. So even if you prove b), and thus that models a couple of years ago were not “conscious” by your chosen definition, more recent models are: we can inspect their Chin of Thought, and it clearly passes your criterion.
Again no, that is missing the point. (Although I agree that this one is not very good experiment).
The reasoning goes like that, they sometimes say e.g. “It felt frightening!”. Did it feel frightening? Or is this what you are supposed to say here, because it’s appropriate thing to say in such situation?
And then its (lack of) skill of introspection becomes relevant.
So even if you prove b), and thus that models a couple of years ago were not “conscious” by your chosen definition
You might be confusing me with OP, I did not indicate that I have any such chosen definition. Or whatever.
It’s very easy to have a reasoning model pick a number in CoT and not tell you. Any competent model should then pass your test.
That’s not the point of this experiment. This experiment is aimed at testing different thing, whether when they tell you about their internal states / emotions, they have anything in there. And plausibly they don’t for the most part. Analogous to LLM telling it has a number in mind while it has actually no number in mind, and doing the impromptu role playing to bullshit you.
There are some subtler points here, like LLM is a kind of array of tubes, maybe the tubes individually feel something. Maybe they do coherently feel something together, but not the thing it has to tell you, as it’s appropriate thing to tell in this situation etc. Maybe something else.
I think what you’ve established is that LLMs weren’t very good at this a couple of years ago, before reasoning models. Humans have an inner monolog. So do reasoning models. Pre-reasoning models don’t. If you’re a sufficiently good meditator to be able to do this, try to pick a random number without internally saying it, visualizing it, specifying “the same number as my birthday” or otherwise committing to it in any symbolic form whatsoever.
Specifically, what I think you have demonstrated is that the model you were testing doesn’t have separate almost-orthogonal activation directions for all 100 numbers between 1 and 100 plus the ability to generate a sharp random distribution of one and only one of them. That’s mildly surprising, but I don’t think it proves lack of consciousness. Given that the only stochastic element in an LLM is the token selection, not letting it generate a token may be blocking the random distribution part of the task. Which would mean that it’s actually implemented lazily, when it does generate tokens to answer your questions, and by resetting it to before that you’re redoing the lazy generation. That’s a valid algorithm, just not the one a human with an inner monolog would use. And the test you are using clearly could be passed by a reasoning model putting tokens for the number into its CoT, so by your logic and definition of the word, that model is conscious.
Personally I don’t see arguing about what definition we want to use for the word “conscious” as very productive, nor as relevant to questions about AI ethics or welfare or alignment, but I know some people do.
They do! They have cashes and they can control them and read from them! Isn’t it surprising? You can ask them to think about bread and say duck, and look at their internals when they say duck and see that they thought “bread”. If they were a bit better at it they would have been able to think that number without saying it and then look back at that thought.
The Point of the experiment, that you missed, is that they sometimes tell that they felt something or whatever, and the question is, did they? Would they know?
Not my logic and definitions. I’m arguing about local point, that particular experiment, I have disagreement with you about.
There are two elements to what you were asking the model to do:
1) Generate a random number without using is normakl built in source of stochasiticity, the token selection process. So you’re requiring it to have an internal pseudorandom number generator algorithm. Which it might simply not have. Wht would it need one? It has a random number generator built in, every time it generates a token.
2) Represent and store a number from 1 to 100 in its internal activations without actually outputting the tokens for it. As in, output a string of tokens like:
”OK, I thought of a number.”
and somewhere in the set of activations on those tokens, at a some layer on some token (maybe the token “number”, maybe the full stop of the sentence) encode that specific number in a way that subsequent activation heads can read from. The only issue here being, if that activation is at a late (but not final) layer, only attention heads at that late layer can attend to it, so the processing that the model is later able to do on that number is thus limited.
You’ve demonstrated that it fails. So, is that because:
a) it doesn’t have a suitable pseudorandom number generator, and you forbade it from using its normal solution of using stochastic token generation to genrate randomness, so it can’t pick a number in the first place — the problem is generation, not storage?
b) it doesn’t have a way to represent the numbers 1 to 100 in its activation space, and is thus “not conscious”? (your claim)
c) it does, but only at a late layer that limits the processing that it can subsequently do on that data, since it never emitted it as a token?
If c) were the case, then the model probably could consistently print the number on replay from after the end of that sentence, but could no play complex 20 questions abouit it. Have you tried that?
My suspicion is that the problem is a). But until you can rule out a) and c), you haven’t proven b).
The fact remains that a modern reasoning model with CoT could, and I’m sure would, pass this test: it would emit an semi-random number from 1 to 100 into its CoT and then be able to refer back to it consistently. So it would have a legible inner monolog, and once it had emitted the number into the CoT, the value of the number would remain accessible and fixed under replay. So even if you prove b), and thus that models a couple of years ago were not “conscious” by your chosen definition, more recent models are: we can inspect their Chin of Thought, and it clearly passes your criterion.
Again no, that is missing the point. (Although I agree that this one is not very good experiment).
The reasoning goes like that, they sometimes say e.g. “It felt frightening!”. Did it feel frightening? Or is this what you are supposed to say here, because it’s appropriate thing to say in such situation?
And then its (lack of) skill of introspection becomes relevant.
You might be confusing me with OP, I did not indicate that I have any such chosen definition. Or whatever.
You’re right, I have mistakenly assumed you were the OP replying