in my view, there doesn’t seem to be such a thing as bad publicity for AGI, and I still don’t know for sure why that’s happening. And that seems like where most of the value is in figuring out this discussion, to me, at least.
It’s an incentive problem.
There is no way to discuss something being dangerous that does not also render it valuable. People are incentivized to seek out value; our entire economy is based on it. It works beautifully, but it is terrible at mitigating externalities. We only dial back from dangerous or bad things after the disaster; so long as doing things is profitable, rational economic actors seek out high-risk activities as far as permitted, because they alone get the profit and the majority of the risk is to other people.
In my view Yudkowsky’s body of work has had two main effects, which run in opposite directions:
Convincing many people that AI is extremely valuable, which is a large part of why we currently are where we are.
Convincing many people that AI is dangerous, which shows no signs of paying off yet but which may be crucially important at some future juncture. I am willing to pronounce it a complete failure at actually causing any regulatory regime whatsoever to come into existence thus far.
It’s an incentive problem.
There is no way to discuss something being dangerous that does not also render it valuable. People are incentivized to seek out value; our entire economy is based on it. It works beautifully, but it is terrible at mitigating externalities. We only dial back from dangerous or bad things after the disaster; so long as doing things is profitable, rational economic actors seek out high-risk activities as far as permitted, because they alone get the profit and the majority of the risk is to other people.
In my view Yudkowsky’s body of work has had two main effects, which run in opposite directions:
Convincing many people that AI is extremely valuable, which is a large part of why we currently are where we are.
Convincing many people that AI is dangerous, which shows no signs of paying off yet but which may be crucially important at some future juncture. I am willing to pronounce it a complete failure at actually causing any regulatory regime whatsoever to come into existence thus far.