Like if you want the “remarkably strong reception” to have strong [retracting: any] evidential value it needs to be sent to skeptics.
Otherwise you’re just like, “Yeah, we sent it to a bunch of famous people using our connections and managed to get endorsements from some.” It’s basically just filtered evidence and attempting to persuade someone with it is attempting to persuade them with means disconnected from the truth.
We have lots of things in life that provide evidence while still being filtered. It’s trivially false that in order for it to have “any evidential value” it needs to be sent to skeptics.
I agree it would be good to do an unbiased survey, but there are lots and lots of institutions in society that rely on filtered evidence and still manage to provide a lot of signal. Calling references from someone’s job application actually helps, even if they of course chose the references to be ones that would give a positive testimonial.
Calling references from someone’s job application actually helps, even if they of course chose the references to be ones that would give a positive testimonial.
That’s quite different though!
References are generally people contacted at work; i.e., those who can be verified to have had a relationship with the person giving the reference. We think these people, by reason of having worked with them, are in a position to be knowledgeable about them, because it’s hard to fake qualities over long period of time. Furthermore, there’s an extremely finite number of such people; someone giving a list of references will probably give a reference they think would give a favorable review, but they have to select from a short list of such possible references. That short list is why it’s evidential. So it’s filtered, but it’s filtered from a small list of people well-positioned to evaluate your character.
Book reviews are different! You can hire a PR firm, contact friends, to try to send it to as many people for a favorable blurb as you possibly can. Rather than having a short list of potential references, you have a very long one. And of course, these people are in many cases not positioned to be knowledgeable about whether the book is correct; they’re certainly not as well-positioned as people you’ve worked with are to know about your character. So you’re filtering from a large list—potentially very large list—of people who are not well-positioned to evaluate the work.
You’re right it provides some evidence. I was wrong about the claim about none; if you could find no people to review a book positively, surely that’s some evidence about it.
But for the reasons above it’s very weak evidence, probably to the degree where you wonder about about the careful use of evidence (“evidence” is anything that might help you distinguish between worlds you’re in, regardless of strength -- 0.001 bits of evidence is still evidence) or the more colloquial (“evidence” is what helps you distinguish between worlds you’re in reasonably strongly).
And like, this is why it’s normal epistemics to ignore the blurbs on the backs of books when evaluating their quality, no matter how prestigious the list of blurbers! Like that’s what I’ve always done, that’s what I imagine you’ve always done, and that’s what we’d of course be doing if this wasn’t a MIRI-published book.
I think you are underestimating the difficulty of getting endorsements like this. Like, I have seen many people in the AI Safety space try to get endorsements like this over the years, for many of their projects, and failed.
Now, how much is that evidence about the correctness of the book? Extremely little! But I also think that’s not what Malo is excited about here. He is excited about the shift in the Overton window it might reflect, and I think that’s pretty real, given the historical failure of people to get endorsements like this for many other projects.
Like, IDK, I am into a more prominent “filtered evidence disclaimer” somewhere in this post, just so that people don’t make wrong updates, but even with the filtered evidence, I think for many people these endorsements are substantial updates.
Now, how much is that evidence about the correctness of the book? Extremely little!
It might not be much evidence for LWers, who are already steeped in arguments and evidence about AI risk. It should be a lot of evidence for people newer to this topic who start with a skeptical prior. Most books making extreme-sounding (conditional) claims about the future don’t have endorsements from Nobel-winning economists, former White House officials, retired generals, computer security experts, etc. on the back cover.
And like, this is why it’s normal epistemics to ignore the blurbs on the backs of books when evaluating their quality, no matter how prestigious the list of blurbers! Like that’s what I’ve always done, that’s what I imagine you’ve always done, and that’s what we’d of course be doing if this wasn’t a MIRI-published book.
If I see a book and I can’t figure out how seriously I should take it, I will look at the blurbs.
Good blurbs from serious, discerning, recognizable people are not on every book, even books from big publishers with strong sales. I realize this is N=2, so update (or not) accordingly, but the first book I could think of that I knew had good sales, but isn’t actually good is The Population Bomb. I didn’t find blurbs for that (I didn’t look all that hard, though, and the book is pretty old, so maybe not a good check for today’s publishing ecosystem anyway). The second book that came to mind was The Body Keeps the Score. The blurbs for that seem to be from a couple respectable-looking psychiatrists I’ve never heard of.
Yeah, I think people usually ignore blurbs, but sometimes blurbs are helpful. I think strong blurbs are unusually likely to be helpful when your book has a title like If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.
I second this. “If Anyone Builds It, Everyone Dies” triggers the “find out if this is insane crank horseshit” subroutine. And one of the quickest/strongest ways to negatively resolve that question is credible endorsements from well-known non-cranks.
Yep. And equally, the blurbs would be a lot less effective if the title were more timid and less stark.
Hearing that a wide range of respected figures endorse a book called If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All is a potential “holy shit” moment. If the same figures were endorsing a book with a vaguely inoffensive title like Smarter Than Us or The AI Crucible, it would spark a lot less interest (and concern).
I’d agree that this is to some extent playing the respectability game, but personally I’d be very happy for Eliezer and people to risk doing this too much rather than too little for once.
Schneier is also quite skeptical of the risk of extinction from AI. Here’s a table o3 generated just now when I asked it for some examples.
Date
Where he said it
What he said
Take-away
1 June 2023
Blog post “On the Catastrophic Risk of AI” (written two days after he signed the CAIS one-sentence “extinction risk” statement)
“I actually don’t think that AI poses a risk to human extinction. I think it poses a similar risk to pandemics and nuclear war — a risk worth taking seriously, but not something to panic over.” (schneier.com)
Explicitly rejects the “extinction” scenario, placing AI in the same (still-serious) bucket as pandemics or nukes.
1 June 2023
Same post, quoting his 2018 book Click Here to Kill Everybody
“I am less worried about AI; I regard fear of AI more as a mirror of our own society than as a harbinger of the future.” (schneier.com)
Long-standing view: most dangers come from how humans use technology we already have.
9 Oct 2023
Essay “AI Risks” (New York Times, reposted on his blog)
Warns against “doomsayers” who promote “Hollywood nightmare scenarios” and urges that we “not let apocalyptic prognostications overwhelm us.” (schneier.com)
Skeptical of the extinction narrative; argues policy attention should stay on present-day harms and power imbalances.
Agreed. As a long-time reader of Schneier’s blog, I was quite surprised by Schneier’s endorsement, and I would have cited exactly those two essays. He’s written a bunch of times about bad things that humans might intentionally use AI to do, talking about things like AI propaganda, AI-powered legal hacks, and AI spam clogging requests for public comments, but I would have described him as scornful of concerns about x-risk or alignment.
Aside from the usual suspects (people like Tegmark), we mostly sent the book to people following the heuristic “would an endorsement from this person be helpful?”, much more so than “do we know that this person would like the book?”. If you’d asked me individually about Church, Schneier, Bernanke, Shanahan, or Spaulding in advance, I’d have put most of my probability on “this person won’t be persuaded by the book (if they read it at all) and will come away strongly disagreeing and not wanting to endorse”. They seemed worth sharing the book with anyway, and then they ended up liking it (at least enough to blurb it) and some very excited MIRI slack messages ensued.
(I’d have expected Eddy to agree with the book, though I wouldn’t have expected him to give a blurb; and I didn’t know Wolfsthal well enough to have an opinion.)
Nate has a blog post coming out in the next few days that will say a bit more about “How filtered is this evidence?” (along with other topics), but my short answer is that we haven’t sent the book to that many people, we’ve mostly sent it to people whose AI opinions we didn’t know much about (and who we’d guess on priors would be skeptical to some degree), and we haven’t gotten many negative reactions at all. (Though we’ve gotten people who just didn’t answer our inquiries, and some of those might have read the book and disliked it enough to not reply.)
As far as I understand MIRI strategy, the point of the book is communication, it’s written from assumption that many people would be really worried if they knew the content of the book, and the main reason why they are not worried now is because before book this content exists mostly in Very Nerdy corners of the internet and is written in very idiosyncratic style. My intended purpose of Pope endorsement is not to singal that book is true, it’s to move for 2 billions of Catholics topic of AI x-risk from spot “something weird discussed in Very Nerdy corners of the internet” to spot “literal head of my church is interested in this stuff, maybe I should become interested too”.
I’d be pretty surprised if this book was persuasive to almost any skeptic who had expressed opinions on the topic before; I think you should just take that as a given. Eliezer and Nate are just not very good (and seemingly uninterested) at writing in ways that are persuasive to thoughtful skeptics. (Additionally, many AI x-risk skeptics are pretty dishonest/unreasonable IMO and thus hard to persuade.)
I think I somewhat disagree with this. My view is more like:
The recent writings of Eliezer (and probably Nate?) are not very good at persuading thoughtful skeptics, seemingly in part due to not really trying to do this / being uninterested (see e.g. Eliezer’s writing on X/twitter).
Eliezer and Nate tried much harder to make this book persuasive to moderately thoughtful people who are more skeptical by default (or at least not actively off-putting) including via mechanisms like having a bunch of test readers etc. I bet they didn’t try hard to engage with the arguments of people who were skeptical and already had detailed views on the topic. So, I expect “skeptics who had somewhat detailed views” will feel like the book doesn’t even responds to their disagreements, let alone persuades them.
In practice, I also expect minimal movement from “thoughtful skeptics”, though maybe a bit more than Buck seems to articulate.
However, this isn’t really the point of the book I’d guess, the point (I think) is to be reasonably persuasive (and make initial arguments) to people who haven’t really thought about the topic (including people who would be skeptical by default).
I expect the book is much less off-putting to the target audience than stuff like Eliezer’s writing on X/twitter which will make it much more effective at its aims.
I don’t expect that this book will (e.g.) engage with my disagreements with Eliezer or why I’m much more optimistic.
[your comment leads me to believe you may not see why MIRI/LC-clustered folks disagreed with your comment but thumbs-upped Ryan, so it might be worthwhile for me to point out why I think that is]
The delta I see between the comments is:
Buck: almost any skeptic who had expressed opinions on the topic before
vs
Ryan: skeptics who had somewhat detailed views
‘Almost any skeptic who has expressed opinions on the topic before’ includes people like Francis Fukayama, who is just a random public intellectual with no AI understanding and got kinda pressed to express a view on x-risk in an interview, so came out against it as a serious concern. Then he thought harder, and voila [more gradual disempowerment flavored, but still!]. I think the vast majority of people, both in gen pop and in powerful positions, are more like Francis than they are like Alex Turner.
So four hypotheses:
You just agree with Ryan’s narrower frame uncomplicatedly and your initial comment was a little strong.
You think the rat/EA in-fights are the important thing to address, and you’re annoyed the book won’t do this.
You think the arguments of the Very Optimistic are the important thing to address.
Not necessarily responding to the rest of your comment, but on the “four hypotheses” part:
You think the rat/EA in-fights are the important thing to address, and you’re annoyed the book won’t do this.
You think the arguments of the Very Optimistic are the important thing to address.
I’m not sure that I buy the “rat/EA infights” and “Very Optimistic” are the relevant categories. I think there are a broad group of people who already have some views on AI risk, who are at-least-plausibly at-least-somewhat open minded, and who could be useful allies if they either changed their mind or took their stated views more seriously.
Let me list some people / groups of people: Noam Brown, Matt Clifford, Boaz Barak, Jared Kaplan, Dario Amodei, Noam Shazeer, [a large group of AI/ML academics], Sayash Kapoor, Nat Friedman, Jake Sullivan, Dean Ball, [various people associated with progress studies], Bill Gates.
I’m not claiming people who didn’t have any considered view on AI don’t matter. But I think that in practice when trying to change the minds and actions of key people (in a way that will actually lead to productive actions etc), it’s often important to either convince some more skeptical people who already have views and objections or to at least have solid arguments against their best objections. At a more basic level, to actually achieve frontier intellectual progress (e.g., settle action relevant disagreements about the level and nature of the risk between people who are already doing things), this is required. Maybe this book isn’t the place to do this, but that isn’t a crux for the point being made in this thread.
Yup, this is a good clarification and I see now that omitting this was an error. Thank you!
I think Jack Shanahan is a member of the reference class you’re pointing at here, and I can say that there were others in this reference class who found the book compelling but not wholly convincing, who did not want to say so publicly (hopefully they start feeling comfortable talking more openly about the topic — that’s the goal!).
There are also other resources we’re currently developing, to release in tandem with the book, that should help with this leg of the conversation.
We are at least attempting to be attentive to this population as part of the overall book campaign, even if it seemed like too much to chew / not exactly the right target for the contents of the book itself.
Can you get Pope to read the book? He seems to have opinions on AI and getting his endorsement would be very useful from book promotion perspective.
I’ve suggested a pathway or two for this; if you have independent pathways, please try them / coordinate with Rob Bensinger about trying them.
I was thinking about something like this because Pope is also a math major.
Like if you want the “remarkably strong reception” to have strong [retracting: any] evidential value it needs to be sent to skeptics.
Otherwise you’re just like, “Yeah, we sent it to a bunch of famous people using our connections and managed to get endorsements from some.” It’s basically just filtered evidence and attempting to persuade someone with it is attempting to persuade them with means disconnected from the truth.
We have lots of things in life that provide evidence while still being filtered. It’s trivially false that in order for it to have “any evidential value” it needs to be sent to skeptics.
I agree it would be good to do an unbiased survey, but there are lots and lots of institutions in society that rely on filtered evidence and still manage to provide a lot of signal. Calling references from someone’s job application actually helps, even if they of course chose the references to be ones that would give a positive testimonial.
That’s quite different though!
References are generally people contacted at work; i.e., those who can be verified to have had a relationship with the person giving the reference. We think these people, by reason of having worked with them, are in a position to be knowledgeable about them, because it’s hard to fake qualities over long period of time. Furthermore, there’s an extremely finite number of such people; someone giving a list of references will probably give a reference they think would give a favorable review, but they have to select from a short list of such possible references. That short list is why it’s evidential. So it’s filtered, but it’s filtered from a small list of people well-positioned to evaluate your character.
Book reviews are different! You can hire a PR firm, contact friends, to try to send it to as many people for a favorable blurb as you possibly can. Rather than having a short list of potential references, you have a very long one. And of course, these people are in many cases not positioned to be knowledgeable about whether the book is correct; they’re certainly not as well-positioned as people you’ve worked with are to know about your character. So you’re filtering from a large list—potentially very large list—of people who are not well-positioned to evaluate the work.
You’re right it provides some evidence. I was wrong about the claim about none; if you could find no people to review a book positively, surely that’s some evidence about it.
But for the reasons above it’s very weak evidence, probably to the degree where you wonder about about the careful use of evidence (“evidence” is anything that might help you distinguish between worlds you’re in, regardless of strength -- 0.001 bits of evidence is still evidence) or the more colloquial (“evidence” is what helps you distinguish between worlds you’re in reasonably strongly).
And like, this is why it’s normal epistemics to ignore the blurbs on the backs of books when evaluating their quality, no matter how prestigious the list of blurbers! Like that’s what I’ve always done, that’s what I imagine you’ve always done, and that’s what we’d of course be doing if this wasn’t a MIRI-published book.
I think you are underestimating the difficulty of getting endorsements like this. Like, I have seen many people in the AI Safety space try to get endorsements like this over the years, for many of their projects, and failed.
Now, how much is that evidence about the correctness of the book? Extremely little! But I also think that’s not what Malo is excited about here. He is excited about the shift in the Overton window it might reflect, and I think that’s pretty real, given the historical failure of people to get endorsements like this for many other projects.
Like, IDK, I am into a more prominent “filtered evidence disclaimer” somewhere in this post, just so that people don’t make wrong updates, but even with the filtered evidence, I think for many people these endorsements are substantial updates.
It might not be much evidence for LWers, who are already steeped in arguments and evidence about AI risk. It should be a lot of evidence for people newer to this topic who start with a skeptical prior. Most books making extreme-sounding (conditional) claims about the future don’t have endorsements from Nobel-winning economists, former White House officials, retired generals, computer security experts, etc. on the back cover.
If I see a book and I can’t figure out how seriously I should take it, I will look at the blurbs.
Good blurbs from serious, discerning, recognizable people are not on every book, even books from big publishers with strong sales. I realize this is N=2, so update (or not) accordingly, but the first book I could think of that I knew had good sales, but isn’t actually good is The Population Bomb. I didn’t find blurbs for that (I didn’t look all that hard, though, and the book is pretty old, so maybe not a good check for today’s publishing ecosystem anyway). The second book that came to mind was The Body Keeps the Score. The blurbs for that seem to be from a couple respectable-looking psychiatrists I’ve never heard of.
Yeah, I think people usually ignore blurbs, but sometimes blurbs are helpful. I think strong blurbs are unusually likely to be helpful when your book has a title like If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.
I second this. “If Anyone Builds It, Everyone Dies” triggers the “find out if this is insane crank horseshit” subroutine. And one of the quickest/strongest ways to negatively resolve that question is credible endorsements from well-known non-cranks.
Yep. And equally, the blurbs would be a lot less effective if the title were more timid and less stark.
Hearing that a wide range of respected figures endorse a book called If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All is a potential “holy shit” moment. If the same figures were endorsing a book with a vaguely inoffensive title like Smarter Than Us or The AI Crucible, it would spark a lot less interest (and concern).
I’d agree that this is to some extent playing the respectability game, but personally I’d be very happy for Eliezer and people to risk doing this too much rather than too little for once.
FWIW, I think Jack Shanahan definitely counts as a skeptic.
Schneier is also quite skeptical of the risk of extinction from AI. Here’s a table o3 generated just now when I asked it for some examples.
Agreed. As a long-time reader of Schneier’s blog, I was quite surprised by Schneier’s endorsement, and I would have cited exactly those two essays. He’s written a bunch of times about bad things that humans might intentionally use AI to do, talking about things like AI propaganda, AI-powered legal hacks, and AI spam clogging requests for public comments, but I would have described him as scornful of concerns about x-risk or alignment.
Aside from the usual suspects (people like Tegmark), we mostly sent the book to people following the heuristic “would an endorsement from this person be helpful?”, much more so than “do we know that this person would like the book?”. If you’d asked me individually about Church, Schneier, Bernanke, Shanahan, or Spaulding in advance, I’d have put most of my probability on “this person won’t be persuaded by the book (if they read it at all) and will come away strongly disagreeing and not wanting to endorse”. They seemed worth sharing the book with anyway, and then they ended up liking it (at least enough to blurb it) and some very excited MIRI slack messages ensued.
(I’d have expected Eddy to agree with the book, though I wouldn’t have expected him to give a blurb; and I didn’t know Wolfsthal well enough to have an opinion.)
Nate has a blog post coming out in the next few days that will say a bit more about “How filtered is this evidence?” (along with other topics), but my short answer is that we haven’t sent the book to that many people, we’ve mostly sent it to people whose AI opinions we didn’t know much about (and who we’d guess on priors would be skeptical to some degree), and we haven’t gotten many negative reactions at all. (Though we’ve gotten people who just didn’t answer our inquiries, and some of those might have read the book and disliked it enough to not reply.)
As far as I understand MIRI strategy, the point of the book is communication, it’s written from assumption that many people would be really worried if they knew the content of the book, and the main reason why they are not worried now is because before book this content exists mostly in Very Nerdy corners of the internet and is written in very idiosyncratic style. My intended purpose of Pope endorsement is not to singal that book is true, it’s to move for 2 billions of Catholics topic of AI x-risk from spot “something weird discussed in Very Nerdy corners of the internet” to spot “literal head of my church is interested in this stuff, maybe I should become interested too”.
I’d be pretty surprised if this book was persuasive to almost any skeptic who had expressed opinions on the topic before; I think you should just take that as a given. Eliezer and Nate are just not very good (and seemingly uninterested) at writing in ways that are persuasive to thoughtful skeptics. (Additionally, many AI x-risk skeptics are pretty dishonest/unreasonable IMO and thus hard to persuade.)
I think I somewhat disagree with this. My view is more like:
The recent writings of Eliezer (and probably Nate?) are not very good at persuading thoughtful skeptics, seemingly in part due to not really trying to do this / being uninterested (see e.g. Eliezer’s writing on X/twitter).
Eliezer and Nate tried much harder to make this book persuasive to moderately thoughtful people who are more skeptical by default (or at least not actively off-putting) including via mechanisms like having a bunch of test readers etc. I bet they didn’t try hard to engage with the arguments of people who were skeptical and already had detailed views on the topic. So, I expect “skeptics who had somewhat detailed views” will feel like the book doesn’t even responds to their disagreements, let alone persuades them.
In practice, I also expect minimal movement from “thoughtful skeptics”, though maybe a bit more than Buck seems to articulate.
However, this isn’t really the point of the book I’d guess, the point (I think) is to be reasonably persuasive (and make initial arguments) to people who haven’t really thought about the topic (including people who would be skeptical by default).
I expect the book is much less off-putting to the target audience than stuff like Eliezer’s writing on X/twitter which will make it much more effective at its aims.
I don’t expect that this book will (e.g.) engage with my disagreements with Eliezer or why I’m much more optimistic.
I don’t think I disagree with any of this, it doesn’t seem to conflict much with what I said.
[your comment leads me to believe you may not see why MIRI/LC-clustered folks disagreed with your comment but thumbs-upped Ryan, so it might be worthwhile for me to point out why I think that is]
The delta I see between the comments is:
vs
‘Almost any skeptic who has expressed opinions on the topic before’ includes people like Francis Fukayama, who is just a random public intellectual with no AI understanding and got kinda pressed to express a view on x-risk in an interview, so came out against it as a serious concern. Then he thought harder, and voila [more gradual disempowerment flavored, but still!]. I think the vast majority of people, both in gen pop and in powerful positions, are more like Francis than they are like Alex Turner.
So four hypotheses:
You just agree with Ryan’s narrower frame uncomplicatedly and your initial comment was a little strong.
You think the rat/EA in-fights are the important thing to address, and you’re annoyed the book won’t do this.
You think the arguments of the Very Optimistic are the important thing to address.
William is just confused.
Not necessarily responding to the rest of your comment, but on the “four hypotheses” part:
I’m not sure that I buy the “rat/EA infights” and “Very Optimistic” are the relevant categories. I think there are a broad group of people who already have some views on AI risk, who are at-least-plausibly at-least-somewhat open minded, and who could be useful allies if they either changed their mind or took their stated views more seriously.
Let me list some people / groups of people: Noam Brown, Matt Clifford, Boaz Barak, Jared Kaplan, Dario Amodei, Noam Shazeer, [a large group of AI/ML academics], Sayash Kapoor, Nat Friedman, Jake Sullivan, Dean Ball, [various people associated with progress studies], Bill Gates.
I’m not claiming people who didn’t have any considered view on AI don’t matter. But I think that in practice when trying to change the minds and actions of key people (in a way that will actually lead to productive actions etc), it’s often important to either convince some more skeptical people who already have views and objections or to at least have solid arguments against their best objections. At a more basic level, to actually achieve frontier intellectual progress (e.g., settle action relevant disagreements about the level and nature of the risk between people who are already doing things), this is required. Maybe this book isn’t the place to do this, but that isn’t a crux for the point being made in this thread.
Yup, this is a good clarification and I see now that omitting this was an error. Thank you!
I think Jack Shanahan is a member of the reference class you’re pointing at here, and I can say that there were others in this reference class who found the book compelling but not wholly convincing, who did not want to say so publicly (hopefully they start feeling comfortable talking more openly about the topic — that’s the goal!).
There are also other resources we’re currently developing, to release in tandem with the book, that should help with this leg of the conversation.
We are at least attempting to be attentive to this population as part of the overall book campaign, even if it seemed like too much to chew / not exactly the right target for the contents of the book itself.