This hypothesis seems pretty compelling. I wonder what the strongest counterarguments are; i.e reasons that impeding prosaic alignment is a bad strategy for ensuring a good future?
I can think of the following:
Labs can easily transfer talent from capabilities to safety work, so we end up with the same pace of progress but instead it’s driven by less safety-minded / x-risk pilled people.
Impeding prosaic alignment successfully slows down AI progress overall, derailing anticipated productivity gains and leading to an eventual replacement of the current paradigm with something less safe.
I doubt (1) because I intuitively assume that the talent pool is sufficiently limited for this kind of transfer to be hard—though I could be wrong. I think (2) implies a successful slowdown which I think would be net-positive, though I’m a bit unsure about the downstream effects of stagnating industry revenue. Overall, I think these arguments are weak rationalizations.
This hypothesis seems pretty compelling. I wonder what the strongest counterarguments are; i.e reasons that impeding prosaic alignment is a bad strategy for ensuring a good future?
I can think of the following:
Labs can easily transfer talent from capabilities to safety work, so we end up with the same pace of progress but instead it’s driven by less safety-minded / x-risk pilled people.
Impeding prosaic alignment successfully slows down AI progress overall, derailing anticipated productivity gains and leading to an eventual replacement of the current paradigm with something less safe.
I doubt (1) because I intuitively assume that the talent pool is sufficiently limited for this kind of transfer to be hard—though I could be wrong. I think (2) implies a successful slowdown which I think would be net-positive, though I’m a bit unsure about the downstream effects of stagnating industry revenue. Overall, I think these arguments are weak rationalizations.
What am I missing?