As often when Paul C and Nate or Eliezer debate a concept, I find myself believing that the most accurate description of the world is somewhere between their two viewpoints.
I think the path described by Paul is something in my mind like the average or modal path. And I think that’s good, because I think there’s some hope there that more resources will get funneled into alignment research and sufficient progress will be made there that things aren’t doom when we hit the critical threshold.
I think there’s still some non-trovial chance of a foom-ier path. I think it’s plausible for a safety-oblivious or malicious researcher to deliberately set up an iteratively self-improving ml model system. I think there’s some critical performance threshold for such a system where it could foom at some point sooner than the critical threshold for the ‘normal’ path. I don’t have any solution in mind for that scenario other than to hope that the AI governance folks can convince governments to crack down on that sort of experimentation.
Edit 6 months later: I have done more thinking and research on this, and am now convinced that a foom path within < 3 years is high probability. Unclear how sharp the takeoff will be, but I believe there will be at least one AI-improvement speed doubling due to AI-improving-AI. And I do not expect it to plateau there.
As often when Paul C and Nate or Eliezer debate a concept, I find myself believing that the most accurate description of the world is somewhere between their two viewpoints. I think the path described by Paul is something in my mind like the average or modal path. And I think that’s good, because I think there’s some hope there that more resources will get funneled into alignment research and sufficient progress will be made there that things aren’t doom when we hit the critical threshold. I think there’s still some non-trovial chance of a foom-ier path. I think it’s plausible for a safety-oblivious or malicious researcher to deliberately set up an iteratively self-improving ml model system. I think there’s some critical performance threshold for such a system where it could foom at some point sooner than the critical threshold for the ‘normal’ path. I don’t have any solution in mind for that scenario other than to hope that the AI governance folks can convince governments to crack down on that sort of experimentation.
Edit 6 months later: I have done more thinking and research on this, and am now convinced that a foom path within < 3 years is high probability. Unclear how sharp the takeoff will be, but I believe there will be at least one AI-improvement speed doubling due to AI-improving-AI. And I do not expect it to plateau there.