people often make an argument that being safe is just a special case of being capable, so therefore capabilities is good to work on. there is a boring sense in which this is obviously fallacious: going to the moon is a special case of getting far away from the earth, and building a ladder gets you further away from the earth, but this doesn’t mean you should work on ladder building.
but also, there is a stronger sense in which this is not only potentially fallacious but often contrary to reality. for example, making neural networks more adversarially robust generally makes them less capable. why is this? because optimizing the average case and the worst case are actually distinct problems that fight for the same resources, and so making one better generally hurts the other. in the specific case of neural networks, one intuition is that when you store more concepts in superposition, this increases average capability (you get more effective model capacity) but it also creates worse worst case failures (superposition gives you an opportunity to exploit the interference)
Isn’t the stronger counterargument orthogonality? Making models more capable of arbitrary goals seems highly unlikely to lead to safety unless you also steer them better, and if you believe that, all this hair-splitting is unnecessary.
the special case fallacy
people often make an argument that being safe is just a special case of being capable, so therefore capabilities is good to work on. there is a boring sense in which this is obviously fallacious: going to the moon is a special case of getting far away from the earth, and building a ladder gets you further away from the earth, but this doesn’t mean you should work on ladder building.
but also, there is a stronger sense in which this is not only potentially fallacious but often contrary to reality. for example, making neural networks more adversarially robust generally makes them less capable. why is this? because optimizing the average case and the worst case are actually distinct problems that fight for the same resources, and so making one better generally hurts the other. in the specific case of neural networks, one intuition is that when you store more concepts in superposition, this increases average capability (you get more effective model capacity) but it also creates worse worst case failures (superposition gives you an opportunity to exploit the interference)
Isn’t the stronger counterargument orthogonality? Making models more capable of arbitrary goals seems highly unlikely to lead to safety unless you also steer them better, and if you believe that, all this hair-splitting is unnecessary.
I get what you’re saying but nonetheless I think that at current margins, alignment research should look WAY more like capabilities research than it does right now. Though obviously my view is premised on a very unpopular perspective.