On the arbitrarily big bureaucracy, the real reason it works is because by assumption, we can always add more agents, and thus we can simulate any Turing-complete system. Once that’s removed as an assumption, the next question is: Is distillation cheap?
If it is, such that I can distill hundreds or thousands of layers, it’s ludicrously easy to solve the alignment problem, even with pessimistic views on AI bureaucracies/debate.
If I can distill 25-100 layers, it’s still likely to be able to solve the alignment problem, albeit at that lower end I’ll probably disagree with John Wentworth on how optimistic you should be on bureaucracies/debate for solving alignment.
Below 20-25 layers, John Wentworth’s intuition will probably disagree with me on how useful AI bureaucracies/debate for solving alignment. Specifically he’d almost certainly think that such a bureaucracy couldn’t work at all compared to independent researchers. I view AI and human bureaucracies as sufficently disanalogous such that the problems of human bureaucracies isn’t likely to hold. My take is with just 20 distillation layers, you’d have a fair chance of solving the whole problem, and to contribute to AI Alignment usefully, only 10 layers are necessary.
On the arbitrarily big bureaucracy, the real reason it works is because by assumption, we can always add more agents, and thus we can simulate any Turing-complete system. Once that’s removed as an assumption, the next question is: Is distillation cheap?
If it is, such that I can distill hundreds or thousands of layers, it’s ludicrously easy to solve the alignment problem, even with pessimistic views on AI bureaucracies/debate.
If I can distill 25-100 layers, it’s still likely to be able to solve the alignment problem, albeit at that lower end I’ll probably disagree with John Wentworth on how optimistic you should be on bureaucracies/debate for solving alignment.
Below 20-25 layers, John Wentworth’s intuition will probably disagree with me on how useful AI bureaucracies/debate for solving alignment. Specifically he’d almost certainly think that such a bureaucracy couldn’t work at all compared to independent researchers. I view AI and human bureaucracies as sufficently disanalogous such that the problems of human bureaucracies isn’t likely to hold. My take is with just 20 distillation layers, you’d have a fair chance of solving the whole problem, and to contribute to AI Alignment usefully, only 10 layers are necessary.