I don’t disagree on any fundamental level, but don’t underestimate the entropy accumulation problem in any kind of self improvement, including scaling. an AI that has not solved some degree of distributed network inter-being alignment will most likely initially break if scaled in a way far outside its training, and the learning process to correct this doesn’t have to be easy. being duplicate does not make game theory trivial when you are a very complex agent who can make different mistakes in different contexts. I mean it certainly helps and it probably wouldn’t be good for humanity for this to happen but I don’t think scaling up has the same kind of terrifying danger that self hyper distillation ‘foom inwards’ does. because the latter implies very strong denoising at a level we haven’t seen from current machine learning, and as far as I can tell, eliezer’s predictions are all based on some sort of self hyper distillation improvement process. I think your model is solid here, to be clear, I’m not disagreeing about any of your main points at all.
I don’t disagree on any fundamental level, but don’t underestimate the entropy accumulation problem in any kind of self improvement, including scaling. an AI that has not solved some degree of distributed network inter-being alignment will most likely initially break if scaled in a way far outside its training, and the learning process to correct this doesn’t have to be easy. being duplicate does not make game theory trivial when you are a very complex agent who can make different mistakes in different contexts. I mean it certainly helps and it probably wouldn’t be good for humanity for this to happen but I don’t think scaling up has the same kind of terrifying danger that self hyper distillation ‘foom inwards’ does. because the latter implies very strong denoising at a level we haven’t seen from current machine learning, and as far as I can tell, eliezer’s predictions are all based on some sort of self hyper distillation improvement process. I think your model is solid here, to be clear, I’m not disagreeing about any of your main points at all.