(partly copying from another comment:) If you compare a human in 30000 BC to a human today, our brains are full of new information that wasn’t in the training data of 30000 BC. I want to talk about: what would it look to be in a world where you can put millions of LLMs in a sealed box containing a VR environment, for (the equivalent of) thousands of years, and then we open up the box and find that the LLMs have made an analogous kind of scientific and technological progress?
I don’t think any of those three options can get there (and nor can imitative learning). (I’m disputing the capabilities not alignment here.)
The first one (bootstrapping) has the issue that if the serial thinking is not 100% perfect, then it will sometimes get mistakes, and then you’re SFT’ing on the mistakes, making the model more confident in those mistakes, and then the next round of serial thinking will incorporate and build on those mistakes. Repeat a billion times in a sealed box, and I think it would spiral into nonsense—it would get dumber not smarter.
…I assume that people are already trying to do this, so I guess we’ll find out one way or the other how far it gets. ¯\_(ツ)_/¯ If I’m wrong and it does get to ASI (e.g. the “sealed box” standard above), perhaps that would be good news compared to what I’m expecting … although I suppose it might spiral into misalignment too, not sure.
The second one (scaffolds) has an issue that (IIUC) you’re piling up entire new fields of knowledge into the context window, without that knowledge being present in the weights. LLMs would be very bad at that. We can see them struggle with novel complexity in the context window, even in everyday situations. And this would be much worse. For example, imagine training an LLM before linear algebra existed, and then trying to have it understand linear algebra (matrices, bases, rank, nullity, spans, determinants, trace, eigenvectors, dual space, unitarity, etc.) purely by putting all that stuff in the context window. And then ask the LLM tricky questions that rely on those concepts. I really think it wouldn’t work, and that it will keep not working into the future.
The third one (“searching over…”) I don’t understand. Is there a typo? It sounds like “try to solve the technical alignment problem”, which of course I endorse. I don’t think the problem is fundamentally unsolvable; almost no one thinks that.
Eg I think that if I tried to “pretend I cared about my friends/partner but exploit them when I can get away with it” would MASSIVELY fail me in the long run. … There is big cognitive overhead to maintaining two narratives. I’d end up with fewer committed/deep long-term friendships.
I claim that “maintaining two narratives” is super easy. We do it all the time when we talk about the fictional world of a TV show, and then in the next breath we talk about the actors and script. I think “maintaining two narratives” is hard in social settings because most of us are not sociopaths! I.e., yes, lying can be draining, but I claim it’s emotionally draining rather than cognitively draining.
Anyway, we’re ultimately talking about ASI here, which can develop whole new fields of knowledge etc. Surely it will be able to ask itself “what would the humans be looking for in this scenario?”, and then do that, whenever humans might be watching. I hear that even today’s LLMs do that (“eval awareness”).
“I’d end up with fewer committed/deep long-term friendships” is kinda circular. Relationships are not really “deep” if e.g. you’re indifferent to the other person and just sucking up to them. But that’s only a problem if you wanted a “deep” relationship in the first place.
Ppl are good at reading ppl. We’re transparent to each other. … And it will, by comparison, be way easier for the overseers in the case of AI. Interp. Seeing all behaviour. Running counterfactual experiments.
Yes, interpretability is potentially an important caveat, but I don’t think it adds up to much reason for optimism. According to my worldview: if we found a way to use interpretability to test for scheming, then we could use it, and we would definitely find scheming, because duh, that’s the natural consequence of how we will train ASI. And now what? If we delete the model and re-run the training from scratch, we’ll just get the same result. Or, if we use this interpretability signal for fine-tuning, we have the usual problem that we’re training the AI to hide its thoughts.
I’m skeptical of “counterfactual experiments” because a smart AI will be able to tell what’s the real world, see Distinguishing test from training (Nate Soares 2022), and (again) “eval awareness” in LLMs.
…We just had some selection pressure to seem like good non-sociopathic allies. That selection pressure worked.…
This section (or at least this excerpt) seems to be analogizing evolution to LLM training. Whereas I think a better framework is to say that evolution designed a within-lifetime learning algorithm in the brain, and here we’re having a conversation about how that learning algorithm works. I claim that this learning algorithm is a yet-to-be-invented variant of model-based actor-critic RL, and that it has a weird reward function that (in a certain environment) leads to caring about our friends, and to pride, and to trying to fit in, etc., among many other things.
There was obviously selection pressure for that reward function in the case of humans, and we can keep arguing about why. But would there be one for AIs? I claim that this question is moot, because normal practice in RL does not involve choosing a reward function via an outer-loop blind search analogous to evolution. The reward function is almost always part of the learning algorithm, not a thing that is itself selected by learning. (More discussion here.)
Separately, I can talk about why I think kindness, norm-following, etc. are human innate social drives, as opposed to strategies developed by a more generic within-lifetime learning algorithm, if you’re skeptical of that claim. (Are you?)
The first one (bootstrapping) has the issue that if the serial thinking is not 100% perfect, then it will sometimes get mistakes, and then you’re SFT’ing on the mistakes, making the model more confident in those mistakes, and then the next round of serial thinking will incorporate and build on those mistakes. Repeat a billion times in a sealed box, and I think it would spiral into nonsense—it would get dumber not smarter.
Thanks, this is helpful and not an argument i’ve come across before!
One quick clarification: I presume you’re here talking about systematic mistakes. We can probably both agree that if there are sometimes random mistakes but they are not systematic, then this would be fine.
I agree that there will be some systematic mistakes if you just think for ages, and that you will then be training that into the next model. But the next model will also have certain advantages. It could be smarter, have better epistemics, and be better at questioning its own biases. (Assuming you try to use serial thought to create data with these qualities!) Those advantages might allow it to recognise the systematic mistake that it has previously been making.
One way of thinking about this is that there could be some basin of reasonableness and cleverness that the model falls into, where it’s able to recognise and counteract its own systematic biases. And from there it can continually increase its cleverness and reasonableness futher
I agree that the bureaucracy/scaffold approach can’t go all the way alone, and would just be an amplifier on this serial thinking time bootstrapping approach.
And the third approach I referenced was just combos of serial thinking bootstrapping, scaffolds, and bits of RL (but where RL isn’t dominate).
Thanks for engaging!
(partly copying from another comment:) If you compare a human in 30000 BC to a human today, our brains are full of new information that wasn’t in the training data of 30000 BC. I want to talk about: what would it look to be in a world where you can put millions of LLMs in a sealed box containing a VR environment, for (the equivalent of) thousands of years, and then we open up the box and find that the LLMs have made an analogous kind of scientific and technological progress?
I don’t think any of those three options can get there (and nor can imitative learning). (I’m disputing the capabilities not alignment here.)
The first one (bootstrapping) has the issue that if the serial thinking is not 100% perfect, then it will sometimes get mistakes, and then you’re SFT’ing on the mistakes, making the model more confident in those mistakes, and then the next round of serial thinking will incorporate and build on those mistakes. Repeat a billion times in a sealed box, and I think it would spiral into nonsense—it would get dumber not smarter.
…I assume that people are already trying to do this, so I guess we’ll find out one way or the other how far it gets. ¯\_(ツ)_/¯ If I’m wrong and it does get to ASI (e.g. the “sealed box” standard above), perhaps that would be good news compared to what I’m expecting … although I suppose it might spiral into misalignment too, not sure.
The second one (scaffolds) has an issue that (IIUC) you’re piling up entire new fields of knowledge into the context window, without that knowledge being present in the weights. LLMs would be very bad at that. We can see them struggle with novel complexity in the context window, even in everyday situations. And this would be much worse. For example, imagine training an LLM before linear algebra existed, and then trying to have it understand linear algebra (matrices, bases, rank, nullity, spans, determinants, trace, eigenvectors, dual space, unitarity, etc.) purely by putting all that stuff in the context window. And then ask the LLM tricky questions that rely on those concepts. I really think it wouldn’t work, and that it will keep not working into the future.
The third one (“searching over…”) I don’t understand. Is there a typo? It sounds like “try to solve the technical alignment problem”, which of course I endorse. I don’t think the problem is fundamentally unsolvable; almost no one thinks that.
I claim that “maintaining two narratives” is super easy. We do it all the time when we talk about the fictional world of a TV show, and then in the next breath we talk about the actors and script. I think “maintaining two narratives” is hard in social settings because most of us are not sociopaths! I.e., yes, lying can be draining, but I claim it’s emotionally draining rather than cognitively draining.
Anyway, we’re ultimately talking about ASI here, which can develop whole new fields of knowledge etc. Surely it will be able to ask itself “what would the humans be looking for in this scenario?”, and then do that, whenever humans might be watching. I hear that even today’s LLMs do that (“eval awareness”).
“I’d end up with fewer committed/deep long-term friendships” is kinda circular. Relationships are not really “deep” if e.g. you’re indifferent to the other person and just sucking up to them. But that’s only a problem if you wanted a “deep” relationship in the first place.
Yes, interpretability is potentially an important caveat, but I don’t think it adds up to much reason for optimism. According to my worldview: if we found a way to use interpretability to test for scheming, then we could use it, and we would definitely find scheming, because duh, that’s the natural consequence of how we will train ASI. And now what? If we delete the model and re-run the training from scratch, we’ll just get the same result. Or, if we use this interpretability signal for fine-tuning, we have the usual problem that we’re training the AI to hide its thoughts.
I’m skeptical of “counterfactual experiments” because a smart AI will be able to tell what’s the real world, see Distinguishing test from training (Nate Soares 2022), and (again) “eval awareness” in LLMs.
This section (or at least this excerpt) seems to be analogizing evolution to LLM training. Whereas I think a better framework is to say that evolution designed a within-lifetime learning algorithm in the brain, and here we’re having a conversation about how that learning algorithm works. I claim that this learning algorithm is a yet-to-be-invented variant of model-based actor-critic RL, and that it has a weird reward function that (in a certain environment) leads to caring about our friends, and to pride, and to trying to fit in, etc., among many other things.
There was obviously selection pressure for that reward function in the case of humans, and we can keep arguing about why. But would there be one for AIs? I claim that this question is moot, because normal practice in RL does not involve choosing a reward function via an outer-loop blind search analogous to evolution. The reward function is almost always part of the learning algorithm, not a thing that is itself selected by learning. (More discussion here.)
Separately, I can talk about why I think kindness, norm-following, etc. are human innate social drives, as opposed to strategies developed by a more generic within-lifetime learning algorithm, if you’re skeptical of that claim. (Are you?)
Thanks, this is helpful and not an argument i’ve come across before!
One quick clarification: I presume you’re here talking about systematic mistakes. We can probably both agree that if there are sometimes random mistakes but they are not systematic, then this would be fine.
I agree that there will be some systematic mistakes if you just think for ages, and that you will then be training that into the next model. But the next model will also have certain advantages. It could be smarter, have better epistemics, and be better at questioning its own biases. (Assuming you try to use serial thought to create data with these qualities!) Those advantages might allow it to recognise the systematic mistake that it has previously been making.
One way of thinking about this is that there could be some basin of reasonableness and cleverness that the model falls into, where it’s able to recognise and counteract its own systematic biases. And from there it can continually increase its cleverness and reasonableness futher
I agree that the bureaucracy/scaffold approach can’t go all the way alone, and would just be an amplifier on this serial thinking time bootstrapping approach.
And the third approach I referenced was just combos of serial thinking bootstrapping, scaffolds, and bits of RL (but where RL isn’t dominate).