Seems like a red flag. How can there not be a more up-to-date one? Is advocacy and recruitment not a goal of AI-risk people? Are they instrumentally irrational? What is preventing you from writing such a post right now?
Most importantly, could it be that people struggle to write a good case for AI-risk, because the case for it is actually pretty weak, when you think about it?
People have made tons of slightly-different things all tackling this sort of goal (for example: https://stampy.ai/ ), they just didn’t happen to fill this exact niche.
I do think maybe it’d actually just be good for @Scott Alexander to write an up-to-date one.
A lot of why I like this one is Scott’s prose, which I feel awkward completely copying and making changes to, and writing a new thing from scratch is a pretty high skill operation.
Stampy seems pretty shallow, even more so than this FAQ. Is that what you meant by it not filling “this exact niche”?
By the way, I come from AGI safety from first principles, where I found your comment linking to this. Notably, that sequence says “My underlying argument is that agency is not just an emergent property of highly intelligent systems, but rather a set of capabilities which need to be developed during training, and which won’t arise without selection for it.” which is reasonable and seems an order of magnitude more conservative than this FAQ, which doesn’t really touch the question of agency at all.
could it be that people struggle to write a good case for AI-risk, because the case for it is actually pretty weak, when you think about it?
The case for AI risk, is the same as the case for computers beating humans at chess. If the fate of the world depended on unaided humans being able to beat the best chess computers, we would have fought and lost about 25 years ago. Computers long ago achieved supremacy in the little domain of chess. They are now going to achieve supremacy in the larger domain of everyday life. If our relationship to them is adversarial, we will lose as surely as even the world champion of human chess loses to a moderately strong chess program.
If this FAQ is out of date, it might be because everyone is busy keeping up with current events.
If our relationship to them is adversarial, we will lose. But you also need to argue that this relationship will (likely) be adversarial.
Also, I’m not asking you to make the case here, I’m asking why the case is not being made on front page of LW and on every other platform. Would that not help with advocacy and recruitment? No idea what “keeping up with current events” means.
you also need to argue that this relationship will (likely) be adversarial
The world is full of adversarial relationships, from rivalry among humans, to machines that resist doing what we want them to do. There are many ways in which AIs and humans might end up clashing.
Superintelligent AI is of particular concern because you probably don’t get a second chance. If your goals clash with the goals of a superintelligent AI, your goals lose. So we have a particular incentive to get the goals of superintelligent AI correct in advance.
I’m not asking you to make the case here, I’m asking why the case is not being made on front page of LW and on every other platform… No idea what “keeping up with current events” means.
Less Wrong was set up to be a forum for discussion of rationality, not a hub of AI activism specifically. Eliezer’s views on AI form just a tiny part of his “Sequences” here. People wanting to work on AI safety could go to the MIRI website or the “AI Alignment” forum.
Certainly Less Wrong now overflows with AI news and discussion. It wasn’t always like that! Even as recently as 2020, I think there was more posting about Covid than there was about AI. A turning point was April last year, when the site founder announced that he thought humanity was on track to fail at the challenge of AI safety. Then came ChatGPT, and ecstasy and dread about AI became mainstream. If the site is now all AI, all the time, that simply reflects the state of the world.
“The world is full of adversarial relationships” is pretty much the weakest possible argument and is not going to convince anyone.
Are you saying that MIRI website has convincing introductory explanation of AI risk, the kind that Raemon wishes he had? Surely he would have found them already? If there aren’t, then, again, why not?
Let me first clarify something. Are you asking because you want to understand MIRI’s specific model of AI risk; or do you just want a simple argument that AI risk is real, and it doesn’t matter who makes the argument?
You’re writing as if the reality of AI risk depends on whether or not there’s an up-to-date FAQ about it, on this website. But you know that Less Wrong does not have a monopoly on AI doom, right? Everyone from the founders of deep learning to the officials of the deep state are worried about AI now, because it has become so powerful. This issue is somewhere in the media every day now; and it’s just common sense, given the way of the world, that entities which are not human and smarter than human potentially pose a risk to the human race.
I want neither. I observe that Raemon cannot find an up to date introduction that he’s happy with, and I point out that this is really weird. What I want is an explanation to this bizarre situation.
Is your position that Raemon is blind, and good, convincing explanations are actually abundant? If so, I’d like to see them, it doesn’t matter where from.
Expositions of AI risk are certainly abundant. there have been numerous books and papers. Or just go to Youtube and type in “AI risk”. As for whether any given exposition is convincing, I am no connoisseur. For a long time, I have taken it for granted that AI can be both smarter than humans and dangerous to humans. I’m more interested in details, like risk taxonomies and alignment theories.
But whether a given exposition is convincing, depends on the audience as well as on the author. Some people have highly specific objections. In our discussion, you questioned whether adversarial relations between AI and human are likely to occur, and with Raemon you bring up the topic of agency, so maybe you specifically need an argument that AIs would ever end up acting against human interests?
As for Raemon, I suspect he would like a superintelligence FAQ that that acknowledges the way things are in 2023 - e.g. the rise of a particular AI paradigm to dominate discussion (deep learning and large language models), and the existence of a public debate about AI safety, all the way up to the UN Security Council.
I don’t know if you know, but after being focused for 20 years on rather theoretical issues of AI, MIRI has just announced it will be changing focus to “broad public communication”. If you look back at their website, in the 2000s their introductory materials were mostly aimed at arguing that smarter-than-human AI is possible and important. Then in the 2010s (which is the era of Less Wrong), the MIRI homepage was more about their technical papers and workshops and so on, and didn’t try to be accessible to a general audience. Now in the mid-2020s, they really will be aiming at a broader audience.
OK, there are many people writing explanations, but if all of them are rehashing the same points from Superintelligence book, then there is not much value in that (and I’m tired of reading the same things over and over). Of course you don’t need new arguments or new evidence, but it’s still strange if there aren’t any.
Anyone who has read this FAQ and others, but isn’t a believer yet, will have some specific objections. But I don’t think everyone’s objections are unique, a better FAQ should be able to cover them, if their refutations exist to begin with.
Also, are you yourself working on AI risk? If not, why not? Is this not the most important problem of our time? Would EY not say that you should work on it? Could it be that you and him actually have wildly different estimates of P(AI doom), despite agreeing on the arguments?
As for Raemon, you’re right, I probably misunderstood why he’s unhappy with newer explanations.
are you yourself working on AI risk? If not, why not?
etc.
I presume you have no idea how enraging these questions are, because you know less than nothing about my life.
I will leave it to you to decide whether this “Average Redditor” style of behavior (look it up, it’s a Youtube character) is something you should avoid in future.
If you actually do want to work on AI risk, but something is preventing you, you can just say “personal reasons”, I’m not going to ask for details.
I understand that my style is annoying to some. Unfortunately, I have not observed polite and friendly people getting interesting answers, so I’ll have to remain like that.
Your questions opened multiple wounds, but I’ll get over it.
I “work on” AI risk, in the sense that I think about it when I can. Under better circumstances, I suspect I could make important contributions. I have not yet found a path to better circumstances.
In 2023 is this still your go-to?
It’s still my go-to for laymen, but as I looked at it yesterday I did sure wish there was a more up-to-date one.
It’s been 10 months now, is there a better one yet?
Seems like a red flag. How can there not be a more up-to-date one? Is advocacy and recruitment not a goal of AI-risk people? Are they instrumentally irrational? What is preventing you from writing such a post right now?
Most importantly, could it be that people struggle to write a good case for AI-risk, because the case for it is actually pretty weak, when you think about it?
People have made tons of slightly-different things all tackling this sort of goal (for example: https://stampy.ai/ ), they just didn’t happen to fill this exact niche.
I do think maybe it’d actually just be good for @Scott Alexander to write an up-to-date one.
A lot of why I like this one is Scott’s prose, which I feel awkward completely copying and making changes to, and writing a new thing from scratch is a pretty high skill operation.
Stampy seems pretty shallow, even more so than this FAQ. Is that what you meant by it not filling “this exact niche”?
By the way, I come from AGI safety from first principles, where I found your comment linking to this. Notably, that sequence says “My underlying argument is that agency is not just an emergent property of highly intelligent systems, but rather a set of capabilities which need to be developed during training, and which won’t arise without selection for it.” which is reasonable and seems an order of magnitude more conservative than this FAQ, which doesn’t really touch the question of agency at all.
The case for AI risk, is the same as the case for computers beating humans at chess. If the fate of the world depended on unaided humans being able to beat the best chess computers, we would have fought and lost about 25 years ago. Computers long ago achieved supremacy in the little domain of chess. They are now going to achieve supremacy in the larger domain of everyday life. If our relationship to them is adversarial, we will lose as surely as even the world champion of human chess loses to a moderately strong chess program.
If this FAQ is out of date, it might be because everyone is busy keeping up with current events.
If our relationship to them is adversarial, we will lose. But you also need to argue that this relationship will (likely) be adversarial.
Also, I’m not asking you to make the case here, I’m asking why the case is not being made on front page of LW and on every other platform. Would that not help with advocacy and recruitment? No idea what “keeping up with current events” means.
The world is full of adversarial relationships, from rivalry among humans, to machines that resist doing what we want them to do. There are many ways in which AIs and humans might end up clashing.
Superintelligent AI is of particular concern because you probably don’t get a second chance. If your goals clash with the goals of a superintelligent AI, your goals lose. So we have a particular incentive to get the goals of superintelligent AI correct in advance.
Less Wrong was set up to be a forum for discussion of rationality, not a hub of AI activism specifically. Eliezer’s views on AI form just a tiny part of his “Sequences” here. People wanting to work on AI safety could go to the MIRI website or the “AI Alignment” forum.
Certainly Less Wrong now overflows with AI news and discussion. It wasn’t always like that! Even as recently as 2020, I think there was more posting about Covid than there was about AI. A turning point was April last year, when the site founder announced that he thought humanity was on track to fail at the challenge of AI safety. Then came ChatGPT, and ecstasy and dread about AI became mainstream. If the site is now all AI, all the time, that simply reflects the state of the world.
It’s also full of co-ooeration.
That’s the wrong way round. EY primarily wanted to explain AI threat , but found that people weren’t rational enough to understand what he was saying.
“The world is full of adversarial relationships” is pretty much the weakest possible argument and is not going to convince anyone.
Are you saying that MIRI website has convincing introductory explanation of AI risk, the kind that Raemon wishes he had? Surely he would have found them already? If there aren’t, then, again, why not?
Let me first clarify something. Are you asking because you want to understand MIRI’s specific model of AI risk; or do you just want a simple argument that AI risk is real, and it doesn’t matter who makes the argument?
You’re writing as if the reality of AI risk depends on whether or not there’s an up-to-date FAQ about it, on this website. But you know that Less Wrong does not have a monopoly on AI doom, right? Everyone from the founders of deep learning to the officials of the deep state are worried about AI now, because it has become so powerful. This issue is somewhere in the media every day now; and it’s just common sense, given the way of the world, that entities which are not human and smarter than human potentially pose a risk to the human race.
I want neither. I observe that Raemon cannot find an up to date introduction that he’s happy with, and I point out that this is really weird. What I want is an explanation to this bizarre situation.
Is your position that Raemon is blind, and good, convincing explanations are actually abundant? If so, I’d like to see them, it doesn’t matter where from.
Expositions of AI risk are certainly abundant. there have been numerous books and papers. Or just go to Youtube and type in “AI risk”. As for whether any given exposition is convincing, I am no connoisseur. For a long time, I have taken it for granted that AI can be both smarter than humans and dangerous to humans. I’m more interested in details, like risk taxonomies and alignment theories.
But whether a given exposition is convincing, depends on the audience as well as on the author. Some people have highly specific objections. In our discussion, you questioned whether adversarial relations between AI and human are likely to occur, and with Raemon you bring up the topic of agency, so maybe you specifically need an argument that AIs would ever end up acting against human interests?
As for Raemon, I suspect he would like a superintelligence FAQ that that acknowledges the way things are in 2023 - e.g. the rise of a particular AI paradigm to dominate discussion (deep learning and large language models), and the existence of a public debate about AI safety, all the way up to the UN Security Council.
I don’t know if you know, but after being focused for 20 years on rather theoretical issues of AI, MIRI has just announced it will be changing focus to “broad public communication”. If you look back at their website, in the 2000s their introductory materials were mostly aimed at arguing that smarter-than-human AI is possible and important. Then in the 2010s (which is the era of Less Wrong), the MIRI homepage was more about their technical papers and workshops and so on, and didn’t try to be accessible to a general audience. Now in the mid-2020s, they really will be aiming at a broader audience.
OK, there are many people writing explanations, but if all of them are rehashing the same points from Superintelligence book, then there is not much value in that (and I’m tired of reading the same things over and over). Of course you don’t need new arguments or new evidence, but it’s still strange if there aren’t any.
Anyone who has read this FAQ and others, but isn’t a believer yet, will have some specific objections. But I don’t think everyone’s objections are unique, a better FAQ should be able to cover them, if their refutations exist to begin with.
Also, are you yourself working on AI risk? If not, why not? Is this not the most important problem of our time? Would EY not say that you should work on it? Could it be that you and him actually have wildly different estimates of P(AI doom), despite agreeing on the arguments?
As for Raemon, you’re right, I probably misunderstood why he’s unhappy with newer explanations.
etc.
I presume you have no idea how enraging these questions are, because you know less than nothing about my life.
I will leave it to you to decide whether this “Average Redditor” style of behavior (look it up, it’s a Youtube character) is something you should avoid in future.
If you actually do want to work on AI risk, but something is preventing you, you can just say “personal reasons”, I’m not going to ask for details.
I understand that my style is annoying to some. Unfortunately, I have not observed polite and friendly people getting interesting answers, so I’ll have to remain like that.
Your questions opened multiple wounds, but I’ll get over it.
I “work on” AI risk, in the sense that I think about it when I can. Under better circumstances, I suspect I could make important contributions. I have not yet found a path to better circumstances.
There’s issues about motivation as well as capability.