I want to talk about why automation is likely more dangerous and more useful than cyborgization, and the reason is Amdahl’s law.
In other words, the slowest process controls the outcome, and at very high levels, the human is likely to be the biggest bottleneck, since we aren’t special here.
Furthermore, I think that most interesting problems are in the NP complexity class assuming no deceptive alignment has happened. If that’s true, then goodhart that is non-adversarial is not a severe problem even with extreme capabilities, because while getting a solution might be super hard, it’s likely but not proven that p doesn’t equal np, and if that’s true than you can verify whether the solution actually works once you have it easily, even if coming up with solutions are harder.
I want to talk about why automation is likely more dangerous and more useful than cyborgization, and the reason is Amdahl’s law.
In other words, the slowest process controls the outcome, and at very high levels, the human is likely to be the biggest bottleneck, since we aren’t special here.
Furthermore, I think that most interesting problems are in the NP complexity class assuming no deceptive alignment has happened. If that’s true, then goodhart that is non-adversarial is not a severe problem even with extreme capabilities, because while getting a solution might be super hard, it’s likely but not proven that p doesn’t equal np, and if that’s true than you can verify whether the solution actually works once you have it easily, even if coming up with solutions are harder.