I find Eliezer and Nates’ arguments compelling but I do downgrade my p(doom) somewhat (-30% maybe?) because there are intelligent people (inside and outside of LW/EA) who disagree with them.
I had some issues with the quote
Will continue to exist regardless of how well you criticize any one part of it.
I’d say LW folk are unusually open to criticism. I think if there were strong arguments they really would change people’s minds here. And especially arguments that focus on one small part at a time.
But have there been strong arguments? I’d love to read them.
There’s basically little reason to engage with it. These are all also evidence that there’s something epistemically off with what is going on in the field.
For me the most convincing evidence that LW is doing something right epistemically is how they did better than basically everyone on Covid. Granted that’s not the alignment forum but it was some of the same people and the same weird epistemic culture at work.
There are intelligent people who disagree, but I was under the impression there was a shortage of intelligent disagreement. Most of the smart disagreement sounds like smart people who haven’t thought in great depth about AI risk in particular, and are often shooting down garbled misunderstandings of the case for AI risk.
I think that’s true of people like: Steven Pinker and Neil deGrasse Tyson. They’re intelligent but clearly haven’t engaged with the core arguments because they’re saying stuff like “just unplug it” and “why would it be evil?”
But there’s also people like...
Robin Hanson. I don’t really agree with him but he is engaging with the AI risk arguments, has thought about it a lot and is a clever guy.
Will MacAskill. One of the most thoughtful thinkers I know of, who I’m pretty confident will have engaged seriously with the AI Risk arguments. His p(doom) is far lower than Eliezer’s. I think he says 3% in What We Owe The Future.
Other AI Alignment experts who are optimistic about our chances of solving alignment and put p(doom) lower (I don’t know enough about the field to name people.)
And I guess I am reserving some small amount of probability for “most of the world’s most intelligent computer scientists, physicists, mathematicians aren’t worried about AI Risk, could I be missing something?” My intuitions from playing around on prediction markets is you have to adjust your bets slightly for those kind of considerations.
Robin Hanson is weird. He paints a picture of a grim future where all nice human values are eroded away, replaced with endless frontier replicators optimized and optimizing only for more replication. And then he just accepts it as if that was fine.
Will Macaskill seems to think AI risk is real. He just thinks alignment is easy. He has a specific proposal involving making anthropomorphic AI and raising it like a human child that he seems keen on.
I find Eliezer and Nates’ arguments compelling but I do downgrade my p(doom) somewhat (-30% maybe?) because there are intelligent people (inside and outside of LW/EA) who disagree with them.
I had some issues with the quote
I’d say LW folk are unusually open to criticism. I think if there were strong arguments they really would change people’s minds here. And especially arguments that focus on one small part at a time.
But have there been strong arguments? I’d love to read them.
For me the most convincing evidence that LW is doing something right epistemically is how they did better than basically everyone on Covid. Granted that’s not the alignment forum but it was some of the same people and the same weird epistemic culture at work.
There are intelligent people who disagree, but I was under the impression there was a shortage of intelligent disagreement. Most of the smart disagreement sounds like smart people who haven’t thought in great depth about AI risk in particular, and are often shooting down garbled misunderstandings of the case for AI risk.
I think that’s true of people like: Steven Pinker and Neil deGrasse Tyson. They’re intelligent but clearly haven’t engaged with the core arguments because they’re saying stuff like “just unplug it” and “why would it be evil?”
But there’s also people like...
Robin Hanson. I don’t really agree with him but he is engaging with the AI risk arguments, has thought about it a lot and is a clever guy.
Will MacAskill. One of the most thoughtful thinkers I know of, who I’m pretty confident will have engaged seriously with the AI Risk arguments. His p(doom) is far lower than Eliezer’s. I think he says 3% in What We Owe The Future.
Other AI Alignment experts who are optimistic about our chances of solving alignment and put p(doom) lower (I don’t know enough about the field to name people.)
And I guess I am reserving some small amount of probability for “most of the world’s most intelligent computer scientists, physicists, mathematicians aren’t worried about AI Risk, could I be missing something?” My intuitions from playing around on prediction markets is you have to adjust your bets slightly for those kind of considerations.
Robin Hanson is weird. He paints a picture of a grim future where all nice human values are eroded away, replaced with endless frontier replicators optimized and optimizing only for more replication. And then he just accepts it as if that was fine.
Will Macaskill seems to think AI risk is real. He just thinks alignment is easy. He has a specific proposal involving making anthropomorphic AI and raising it like a human child that he seems keen on.
I just posted a detailed explanation of why I am very skeptical of the traditional deceptive alignment story. I’d love to hear what you think of it!
Deceptive Alignment Skepticism—LessWrong