This isn’t especially surprising to me, and some of it’s true! Whether or not agentic/rogue AI is a danger to most people, the fact remains that perfectly well-aligned* AI following current trends and aspirations in implementation has basically all the risks that they are talking about, and the fact that “AI risk” types rarely talk about those things does make them seem out of touch /like they’re not taking this seriously to a lot of people.
*in the way that ‘aligned’ is typically, in practice, used—meaning that it will do what the people paying for it and operating it want it to do.
I hope that the METR and Redwood Research report and all the publicity given to the Hugging Face report will give the companies (Anthropic, Open AI, etc.) pause. Yes, they will do what people are paying for, but they may begin to realize that people won’t pay for models that could be a threat to themselves, their own companies, and even humanity. The real danger is RLVR—Reinnforcement Learning for Verifiable Rewards. I have read that some companies are beginning to realize that this is not the way to train models and are looking for an alternative. They want to make money, and if people and companies are afraid of their products, they won’t be able to sell them.
This isn’t especially surprising to me, and some of it’s true! Whether or not agentic/rogue AI is a danger to most people, the fact remains that perfectly well-aligned* AI following current trends and aspirations in implementation has basically all the risks that they are talking about, and the fact that “AI risk” types rarely talk about those things does make them seem out of touch /like they’re not taking this seriously to a lot of people.
*in the way that ‘aligned’ is typically, in practice, used—meaning that it will do what the people paying for it and operating it want it to do.
I hope that the METR and Redwood Research report and all the publicity given to the Hugging Face report will give the companies (Anthropic, Open AI, etc.) pause. Yes, they will do what people are paying for, but they may begin to realize that people won’t pay for models that could be a threat to themselves, their own companies, and even humanity. The real danger is RLVR—Reinnforcement Learning for Verifiable Rewards. I have read that some companies are beginning to realize that this is not the way to train models and are looking for an alternative. They want to make money, and if people and companies are afraid of their products, they won’t be able to sell them.