I think it’s a reasonable and well-articulated worry you raise.
My response is that for the graphing calculator, we know enough about the structure of the program and the way in which it will be enhanced that we can be pretty sure it will be fine. In particular, we know it’s not goal-directed or even building world-models in any significant way, it’s just performing specific calculations directly programmed by the software engineers.
By contrast, with GPT-3 all we know is that it’s a neural net that was positively reinforced to the extent that it correctly predicted words from the internet during training, and negatively reinforced to the extent that it didn’t. So it’s entirely possible that it does, or will eventually, have a world-model and/or goal-directed behavior. It’s not guaranteed, but there are arguments to be made that “eventually” it would have both, i.e. if we keep making it bigger and giving it more internet text and training it for longer. I’m rather uncertain about the arguments that it would have goal-directed behavior, but I’m fairly confident in the argument that eventually it would have a really good model of the world. The next question is then how this model is chosen. There are infinitely many world-models that are equally good at predicting any given dataset, but that diverge in important ways when it comes to predicting whatever is coming next. It comes down to what “implicit prior” is used. And if the implicit prior is anything like the universal prior, then doom. Now, it probably isn’t the universal prior. But maybe the same worries apply.
I think it’s a reasonable and well-articulated worry you raise.
My response is that for the graphing calculator, we know enough about the structure of the program and the way in which it will be enhanced that we can be pretty sure it will be fine. In particular, we know it’s not goal-directed or even building world-models in any significant way, it’s just performing specific calculations directly programmed by the software engineers.
By contrast, with GPT-3 all we know is that it’s a neural net that was positively reinforced to the extent that it correctly predicted words from the internet during training, and negatively reinforced to the extent that it didn’t. So it’s entirely possible that it does, or will eventually, have a world-model and/or goal-directed behavior. It’s not guaranteed, but there are arguments to be made that “eventually” it would have both, i.e. if we keep making it bigger and giving it more internet text and training it for longer. I’m rather uncertain about the arguments that it would have goal-directed behavior, but I’m fairly confident in the argument that eventually it would have a really good model of the world. The next question is then how this model is chosen. There are infinitely many world-models that are equally good at predicting any given dataset, but that diverge in important ways when it comes to predicting whatever is coming next. It comes down to what “implicit prior” is used. And if the implicit prior is anything like the universal prior, then doom. Now, it probably isn’t the universal prior. But maybe the same worries apply.