I continue to insist that we have a threshold problem, not just a speed problem. We currently have no way to constrain what will happen when AI becomes strictly superhuman at creating better AI, or when it gains the option to permanently escape human oversight or disempower humanity. We have no guarantees of the degree of generalization from our intent at those thresholds. They are opaque walls to us.
The continuity argument fails at all of these steps, not just new architectures. But if AI does an increasing amount of the AI research, we’ll also have no way of preventing the emergence of new architectures!
The emergent behavior of agent swarms was a relatively transparent threshold, and even that showed that we are just blundering forward with no idea what we are doing.
I continue to insist that we have a threshold problem, not just a speed problem. We currently have no way to constrain what will happen when AI becomes strictly superhuman at creating better AI, or when it gains the option to permanently escape human oversight or disempower humanity. We have no guarantees of the degree of generalization from our intent at those thresholds. They are opaque walls to us.
The continuity argument fails at all of these steps, not just new architectures. But if AI does an increasing amount of the AI research, we’ll also have no way of preventing the emergence of new architectures!
The emergent behavior of agent swarms was a relatively transparent threshold, and even that showed that we are just blundering forward with no idea what we are doing.