I think my current take on this is that it is plausible that current LM architectures may not scale to AGI, but the obstacles don’t seem fundamental, and ML researchers are strongly incentivized to fix them. Also, you don’t need a full model of the universe, or even of the human model of the universe; just as classical mechanics is not the actual nature of the universe but nonetheless is really useful in many situations, we might also expect the “minimal conditional structure” to be super useful (and dangerous). You can imagine that an AI doesn’t need to understand the ineffable intricacies of consciousness to do lots of dangerous things, even things like deception or manipulation.
See this post (especially the section about resolution) for some similar ideas: https://www.lesswrong.com/posts/EmxfgPGvaKqhttPM8/thoughts-on-the-alignment-implications-of-scaling-language
I think my current take on this is that it is plausible that current LM architectures may not scale to AGI, but the obstacles don’t seem fundamental, and ML researchers are strongly incentivized to fix them. Also, you don’t need a full model of the universe, or even of the human model of the universe; just as classical mechanics is not the actual nature of the universe but nonetheless is really useful in many situations, we might also expect the “minimal conditional structure” to be super useful (and dangerous). You can imagine that an AI doesn’t need to understand the ineffable intricacies of consciousness to do lots of dangerous things, even things like deception or manipulation.