well, yeah, AI developers will maybe succeed at shaping the AI along a bunch of specific dimensions. But they will not succeed at exhaustively shaping the AI along all dimensions that turn out to matter.
All dimensions that turn out to matter for what? Current AI is already implicitly optimizing people to use the world “delve” more often than they otherwise would, which is weird and unexpected, but not that bad in the grand scheme of things. Further arguments are needed to distinguish whether this ends in “humans dead, all value lost” or “transhuman utopia, but with some weird and unexpected features, which would also be true of the human-intelligence-augmentation trajectory.” (I’m not saying I believe in the utopia, but if we want that Pause treaty, we need to find the ironclad arguments that convince skeptical experts, not just appeal to intuition.)
Right, but I think a big part of how safety team earns its dignity points is by being as specific as possible about exactly how capabilities team is being suicidal, not just with metaphors and intuition pumps, but state-of-the-art knowledge: you want to be winning arguments with people who know the topic, not just policymakers and the public. My post on adversarial examples (currently up for 2024 Review voting) is an example of what I think this should look like. I’m not just saying “AI did something weird, therefore AI bad”, I’m reviewing the literature and trying to explain why the weird thing would go wrong.
I agree directionally and denotationally with this, but I feel the need to caution that “winning arguments” is itself a very dangerous epistemic frame to inhabit for long.
Also...
I think a big part of how safety team earns its dignity points is by being as specific as possible about exactly how capabilities team is being suicidal
well, yeah, AI developers will maybe succeed at shaping the AI along a bunch of specific dimensions. But they will not succeed at exhaustively shaping the AI along all dimensions that turn out to matter.
now what say you to this clever rejoinder?
All dimensions that turn out to matter for what? Current AI is already implicitly optimizing people to use the world “delve” more often than they otherwise would, which is weird and unexpected, but not that bad in the grand scheme of things. Further arguments are needed to distinguish whether this ends in “humans dead, all value lost” or “transhuman utopia, but with some weird and unexpected features, which would also be true of the human-intelligence-augmentation trajectory.” (I’m not saying I believe in the utopia, but if we want that Pause treaty, we need to find the ironclad arguments that convince skeptical experts, not just appeal to intuition.)
Those developing powerful technologies should treat exotic failure scenarios as major bugs.
Right, but I think a big part of how safety team earns its dignity points is by being as specific as possible about exactly how capabilities team is being suicidal, not just with metaphors and intuition pumps, but state-of-the-art knowledge: you want to be winning arguments with people who know the topic, not just policymakers and the public. My post on adversarial examples (currently up for 2024 Review voting) is an example of what I think this should look like. I’m not just saying “AI did something weird, therefore AI bad”, I’m reviewing the literature and trying to explain why the weird thing would go wrong.
I agree directionally and denotationally with this, but I feel the need to caution that “winning arguments” is itself a very dangerous epistemic frame to inhabit for long.
Also...
We do that too! There’s a lot of ground to cover.