Hi Luke! Thanks for replying. Quick counterpoints:
Probably most importantly, what do you view as the purpose of SIAI’s publishing papers? Or, if there are multiple purposes, which do you see as the most important?
If in-person conversations (despite all their limitations) are still the much preferred way to discuss things, instead of papers, that’s evidence in favor of papers being bad. (It’s also evidence of SIAI being effective, which is great, but that isn’t the point under discussion.) If papers were a good discussion forum, there’d be fewer conversations and more papers.
If, as you say, the main audience for papers written by SIAI is through SIAI’s website and not through the journals themselves, why spend the time and expense and hassle to write them up in journal form? Why not just publish them directly on the site, in (probably) a much more readable format?
The problem with conformity in academia isn’t that it’s impossible to find someplace to publish. You can always find somewhere, given enough effort. The problem is that a) it restricts the sorts of things you can say, b) restricts you, in many cases, to an awkward way of wording things (which I believe you’ve written about at http://lesswrong.com/lw/4r1/how_siai_could_publish_in_mainstream_cognitive/), and c) it makes academia a less fertile ground for recruiting people. Those are probably in addition to other problems.
I agree that we care more about prestige within academia than we do about prestige in almost all similarly sized groups. However, it seems fairly strongly that we aren’t going to have that much prestige in academia anyway, given that the main prestige mechanism is elite university affiliations, and most of us don’t have those.
Which people have come through Eliezer and Bostrom’s papers? (That isn’t a rhetorical question; given how large our community is compared to Dunbar’s number, it’s likely there is someone and it’s also likely I’ve missed them, and they might be really cool people to know.)
Using my own personal experiences is generalizing from a single dataset, and that’s indeed biased in some ways. However, it’s very far from generalizing from a single example; it’s generalizing from the many thousands of arguments that I’ve read and accepted at some point in the past. It’s still obviously better to use multiple datasets, if you can get them.… but in this case they’re difficult to get, because it’s hard to know where your friends got all their beliefs.
Sure, it’s easier to get people to read a single paper than all of the Sequences. But that’s a totally unfair comparison: the Sequences are much, much longer, and it’s always easier to read something shorter than something longer. How hard would it be to get someone to read a paper, vs. a single Sequence post of equal length, or a bunch of Sequence posts that sum to an equal length?
If all new areas of research are developed through in-person conversations and mailing lists, that doesn’t imply that papers are a good way to do FAI research; it implies that papers are a bad way to do all those other kinds of research. If what you say is true, then my argument equally well applies to those fields too.
Of course, there are some instances of academic moderation being net good rather than net bad. However, to quote of your earlier arguments, “don’t generalize from one example”. I’m sure that there are some well-moderated journals, just as I’m sure there are Mafia bosses who are really nice helpful guys. However, that doesn’t imply that hanging out with Mafia bosses is a good idea.
what do you view as the purpose of SIAI’s publishing papers?
Grab the interest of smart people who won’t be grabbed by cheaper methods. This has worked before. Also: Many smart and productive people are extremely busy, and they use “Did they bother to pass peer review?” as a filter for what they choose to read. In addition, many smart people prefer to read papers over blog posts because papers are generally better organized, are more clearly written, helpfully cite related work, etc.
Reduce communication overhead. We don’t have time to have a personal conversation with every interested smart person, and blog posts are often too disorganized and ambiguous to help. Though for this, a scholarly AI risk wiki would probably be even better. Luckily, as I say in that post, there isn’t much additional cost involved in turning parts of papers into wiki articles, or combining wiki articles into papers.
Grab some prestige and credibility, because this matters to lots of the people we care about.
Show that we’re capable of doing serious research. “Eliezer did some work with Marcello that we can never tell you about” and “We wrote some blog posts this month” don’t quite show to most people that we can do research.
Be kinda-forced into writing more clearly, and in a way that is more thoroughly connected to the relevant empirical literatures, than we might otherwise be tempted to write.
Why not just publish them directly on the site, in (probably) a much more readable format?
As I said before, many people find papers more readable than ambiguous blog posts barely connected to the relevant literatures. Eliezer’s papers aren’t written in a different style than his blog posts, anyway. Also, peer review often improves the final product.
The problem with conformity in academia… is that a) it restricts the sorts of things you can say, b) restricts you, in many cases, to an awkward way of wording things, and c) it makes academia a less fertile ground for recruiting people.
Agree with (a) and somewhat with (b), but we’re only writing certain things in paper form. Like I said, the vast majority of FAI work and discussion happens outside papers. I don’t know what you mean by (c).
it seems… that we aren’t going to have that much prestige in academia anyway, given that the main prestige mechanism is elite university affiliations, and most of us don’t have those.
I don’t care about something like “average prestige in academia.” What I care about is some particular people thinking we have enough credibility to bother reading and engaging with. Many of the people I care about won’t bother to check whether the author of an article has elite university affiliation, but will care if we bothered to write up our ideas clearly and with references to related work. The Singularity and Machine Ethics looks much less crankish than Creating Friendly AI, even though none of the authors have elite university affiliation.
Which people have come through Eliezer and Bostrom’s papers?
Still gathering data, and I haven’t gathered permission to share it. I think two people who wouldn’t mind you knowing they came to x-risk through “Astronomical Waste” are Nick Beckstead and Jason Gaverick Matheny.
Using my own personal experiences is… very far from generalizing from a single example
Point taken.
How hard would it be to get someone to read a paper, vs. a single Sequence post of equal length, or a bunch of Sequence posts that sum to an equal length?
My intended point was that sometimes a paper has summed up the main points from something that Eliezer took 30 blog posts to write when he wrote The Sequences. But obviously you don’t have to write a paper to do this, so I drop the point.
If all new areas of research are developed through in-person conversations and mailing lists, that doesn’t imply that papers are a good way to do FAI research; it implies that papers are a bad way to do all those other kinds of research.
Remember: almost all FAI research is not done via papers. In my above list of reasons why SI publishes papers, I didn’t even think to mention “to produce original research” (and I won’t go back and add it now), though that sometimeshappens.
there are some instances of academic moderation being net good rather than net bad. However, to quote of your earlier arguments, “don’t generalize from one example”. I’m sure that there are some well-moderated journals, just as I’m sure there are Mafia bosses who are really nice helpful guys. However, that doesn’t imply that hanging out with Mafia bosses is a good idea.
If one journal is poorly moderated, then you jump to another one. Unlike Mafia bosses, a “problem” with journal moderators means “I wasted a few hours communicating with them and making revisions,” not “They decided to cut off my thumbs.”
For people who “are extremely busy, and they use “Did they bother to pass peer review?” as a filter for what they choose to read”, which specific examples are you thinking of, and how much any of them become nontrivial members of our community, or helped us out in nontrivial ways?
I’m sure there are people who a) are very smart, b) look impressive on paper, who c) we’ve contacted about FAI research, and d) have said “I’m not going to pay attention, since this isn’t peer reviewed” (or some equivalent). However, I think that for most of those people, that isn’t their true rejection (http://lesswrong.com/lw/wj/is_that_your_true_rejection/), and they aren’t going to take us seriously anyway. But I could be wrong—what evidence do you have in mind?
A lot of your points are criticisms of blog posts, like “a lot of them don’t have citations”, or “a lot of them are poorly organized”. These are true in many cases. However, if SIAI is considering whether to publish some given idea in paper or blog post form, they could simply spend the (fairly small) effort to write a blog post which was well organized and had citations, thereby making these problems moot.
Journal editors obviously aren’t perfectly analogous to mob bosses. However, I’ve heard many stories from academics of authors spending huge amounts of time and effort trying to get stuff published. In the most recent case, which I discussed with a grad student just a few hours ago, it took hundreds of hours, over a full year. If it’s usually easy to get around that sort of thing, by just publishing in a different journal, why don’t more academics do so?
Your first two questions ask about evidence that I already said I’m not in a position to share yet. I know that’s unsatisfying, but… are your priors on my claims being true really very low? Famous scientists, especially, are barraged with a few purported unifications of quantum theory and relativity every month, and “Did they bother to pass peer review?” is a pretty useful heuristic for them. When you visualize a busy academic receiving CFAI from one person, and The Singularity and Machine Ethics from somebody else, which one do you think they’re more likely to read and take seriously, and why? (Feel free to take this as a rhetorical question.)
A lot of your points are criticisms of blog posts, like “a lot of them don’t have citations”, or “a lot of them are poorly organized”. These are true in many cases. However, if SIAI is considering whether to publish some given idea in paper or blog post form, they could simply spend the (fairly small) effort to write a blog post which was well organized and had citations, thereby making these problems moot.
The effort required may be much larger than you think. Eliezer finds it very difficult to do that kind of work, for example. (Which is why his papers still read like long blog posts, and include very few citations. CEV even contains zero citations, despite re-treading ground that has been discussed by philosophers for centuries, as “The Singularity and Machine Ethics” shows.)
And if you’ve done all that work, then why not also tweak it for use in a scholarly AI risk wiki, and then combine it with a couple other wiki articles into a paper?
I’ve heard many stories from academics of authors spending huge amounts of time and effort trying to get stuff published. In the most recent case, which I discussed with a grad student just a few hours ago, it took hundreds of hours, over a full year. If it’s usually easy to get around that sort of thing, by just publishing in a different journal, why don’t more academics do so?
Because their career depends on satisfying their advisors, or on getting published in particular journals. SI researchers’ careers don’t depend on investing hundreds of hours making revisions. If publishing in a certain journal is going to require 30 hours of revisions that don’t actually improve the paper in our eyes, then we aren’t going to bother publishing in that journal.
The effort required may be much larger than you think. Eliezer finds it very difficult to do that kind of work, for example. (Which is why his papers still read like long blog posts, and include very few citations. CEV even contains zero citations, despite re-treading ground that has been discussed by philosophers for centuries, as “The Singularity and Machine Ethics” shows.)
If this is the case, then a significant benefit to Eliezer of trying to get papers published would be that it would be excellent discipline for Eliezer, and would make him an even better scholar.
A benefit that would follow on is that it would establish by example that nobody is above showing their work, acknowledging their debts and being current on the relevant literature. Conceivably Eliezer is such a talented guy that it is of no benefit to him to do these things, but if everyone who thought they were that talented were excused from showing their work and keeping current then progress would slow significantly.
It also avoids reinventing the wheel. No matter how smart Eliezer is, it’s always conceivable that someone else thought of something first and expressed it in rigorous detail with proper citations. A proper literature review avoids this waste of valuable research time.
It also avoids reinventing the wheel. No matter how smart Eliezer is, it’s always conceivable that someone else thought of something first and expressed it in rigorous detail with proper citations. A proper literature review avoids this waste of valuable research time.
Luke (and his remote research assistants) have this angle covered.
A lot of your points are criticisms of blog posts, like “a lot of them don’t have citations”, or “a lot of them are poorly organized”. These are true in many cases. However, if SIAI is considering whether to publish some given idea in paper or blog post form, they could simply spend the (fairly small) effort to write a blog post which was well organized and had citations, thereby making these problems moot.
… and if you’re going through the effort of writing a blog post that’s journal-quality anyway, you might as well go ahead and publish it as a full paper while you’re at it.
… and if you’re going through the effort of writing a blog post that’s journal-quality anyway, you might as well go ahead and publish it as a full paper while you’re at it.
If it’s usually easy to get around that sort of thing, by just publishing in a different journal, why don’t more academics do so?
Clearly the grad student (or more likely, their advisor) thought that getting published in journal X was worth enough status to spend over a year working on it. Fake utility functions and all that. Not even academics are perfectly rational.
If, as you say, the main audience for papers written by SIAI is through SIAI’s website and not through the journals themselves, why spend the time and expense and hassle to write them up in journal form? Why not just publish them directly on the site, in (probably) a much more readable format?
I imagine that some academics would be more willing to read a paper from the website if it is also published in a journal, especially if they are writing their own journal articles in which they would prefer to cite a journal article to a paper published on a website.
they would prefer to cite a journal article to a paper published on a website.
This. Once our ideas enter the academic memespace, they may continue to live independently there. The difficult part is crossing the boundary, but it only has to be done once.
Which people have come through Eliezer and Bostrom’s papers?
While it would be incorrect to say that I originally came to these issues only via Bostrom’s papers, they certainly made me a lot more interested in the field. Partly because of the prestige of being actually peer-reviewed, but mostly (I think—it was a long time ago) because they were clear, rigorous and self-contained to an extent that few other materials were.
Probably most importantly, what do you view as the purpose of SIAI’s publishing papers? Or, if there are multiple purposes, which do you see as the most important?
In order to think of some things I do that only have one important purpose, it was necessary to perform the ritual of closing my eyes and thinking about nothing else for a few minutes by the clock.
I plan on assuming things have multiple important purposes and asking for several, e.g. “what do you view as the purposes of X.”
There was nothing wrong with what you said, but it is strange how easily the (my?) mind stops questioning after coming up with just one purpose for something someone is doing. In contrast, when justifying one’s own behavior, it is easy to think of multiple justifications.
It makes some sense in a story about motivated cognition and tribal arguments. It might be that to criticize, we look mostly for something someone does that has no justification, and invest less in attacking someone along a road that has some defenses. A person being criticized invests in defending against those attacks they know are coming, and does not try and think of all possible weaknesses in their position. There is some advantage in being genuinely blind to one’s weaknesses so one can, without lying, be confident in one’s own position.
Maybe it is ultimately unimportant to ask what the “purposes” of someone doing something is, since they will be motivated to justify themselves as much as possible. In this case, asking what the “purpose” is would force them concentrate on their most persuasive and potentially best argument, even if it will rarely actually be the case that one purpose is a large supermajority of their motivation.
Hi Luke! Thanks for replying. Quick counterpoints:
Probably most importantly, what do you view as the purpose of SIAI’s publishing papers? Or, if there are multiple purposes, which do you see as the most important?
If in-person conversations (despite all their limitations) are still the much preferred way to discuss things, instead of papers, that’s evidence in favor of papers being bad. (It’s also evidence of SIAI being effective, which is great, but that isn’t the point under discussion.) If papers were a good discussion forum, there’d be fewer conversations and more papers.
If, as you say, the main audience for papers written by SIAI is through SIAI’s website and not through the journals themselves, why spend the time and expense and hassle to write them up in journal form? Why not just publish them directly on the site, in (probably) a much more readable format?
The problem with conformity in academia isn’t that it’s impossible to find someplace to publish. You can always find somewhere, given enough effort. The problem is that a) it restricts the sorts of things you can say, b) restricts you, in many cases, to an awkward way of wording things (which I believe you’ve written about at http://lesswrong.com/lw/4r1/how_siai_could_publish_in_mainstream_cognitive/), and c) it makes academia a less fertile ground for recruiting people. Those are probably in addition to other problems.
I agree that we care more about prestige within academia than we do about prestige in almost all similarly sized groups. However, it seems fairly strongly that we aren’t going to have that much prestige in academia anyway, given that the main prestige mechanism is elite university affiliations, and most of us don’t have those.
Which people have come through Eliezer and Bostrom’s papers? (That isn’t a rhetorical question; given how large our community is compared to Dunbar’s number, it’s likely there is someone and it’s also likely I’ve missed them, and they might be really cool people to know.)
Using my own personal experiences is generalizing from a single dataset, and that’s indeed biased in some ways. However, it’s very far from generalizing from a single example; it’s generalizing from the many thousands of arguments that I’ve read and accepted at some point in the past. It’s still obviously better to use multiple datasets, if you can get them.… but in this case they’re difficult to get, because it’s hard to know where your friends got all their beliefs.
Sure, it’s easier to get people to read a single paper than all of the Sequences. But that’s a totally unfair comparison: the Sequences are much, much longer, and it’s always easier to read something shorter than something longer. How hard would it be to get someone to read a paper, vs. a single Sequence post of equal length, or a bunch of Sequence posts that sum to an equal length?
If all new areas of research are developed through in-person conversations and mailing lists, that doesn’t imply that papers are a good way to do FAI research; it implies that papers are a bad way to do all those other kinds of research. If what you say is true, then my argument equally well applies to those fields too.
Of course, there are some instances of academic moderation being net good rather than net bad. However, to quote of your earlier arguments, “don’t generalize from one example”. I’m sure that there are some well-moderated journals, just as I’m sure there are Mafia bosses who are really nice helpful guys. However, that doesn’t imply that hanging out with Mafia bosses is a good idea.
Grab the interest of smart people who won’t be grabbed by cheaper methods. This has worked before. Also: Many smart and productive people are extremely busy, and they use “Did they bother to pass peer review?” as a filter for what they choose to read. In addition, many smart people prefer to read papers over blog posts because papers are generally better organized, are more clearly written, helpfully cite related work, etc.
Reduce communication overhead. We don’t have time to have a personal conversation with every interested smart person, and blog posts are often too disorganized and ambiguous to help. Though for this, a scholarly AI risk wiki would probably be even better. Luckily, as I say in that post, there isn’t much additional cost involved in turning parts of papers into wiki articles, or combining wiki articles into papers.
Grab some prestige and credibility, because this matters to lots of the people we care about.
Show that we’re capable of doing serious research. “Eliezer did some work with Marcello that we can never tell you about” and “We wrote some blog posts this month” don’t quite show to most people that we can do research.
Be kinda-forced into writing more clearly, and in a way that is more thoroughly connected to the relevant empirical literatures, than we might otherwise be tempted to write.
As I said before, many people find papers more readable than ambiguous blog posts barely connected to the relevant literatures. Eliezer’s papers aren’t written in a different style than his blog posts, anyway. Also, peer review often improves the final product.
Agree with (a) and somewhat with (b), but we’re only writing certain things in paper form. Like I said, the vast majority of FAI work and discussion happens outside papers. I don’t know what you mean by (c).
I don’t care about something like “average prestige in academia.” What I care about is some particular people thinking we have enough credibility to bother reading and engaging with. Many of the people I care about won’t bother to check whether the author of an article has elite university affiliation, but will care if we bothered to write up our ideas clearly and with references to related work. The Singularity and Machine Ethics looks much less crankish than Creating Friendly AI, even though none of the authors have elite university affiliation.
Still gathering data, and I haven’t gathered permission to share it. I think two people who wouldn’t mind you knowing they came to x-risk through “Astronomical Waste” are Nick Beckstead and Jason Gaverick Matheny.
Point taken.
My intended point was that sometimes a paper has summed up the main points from something that Eliezer took 30 blog posts to write when he wrote The Sequences. But obviously you don’t have to write a paper to do this, so I drop the point.
Remember: almost all FAI research is not done via papers. In my above list of reasons why SI publishes papers, I didn’t even think to mention “to produce original research” (and I won’t go back and add it now), though that sometimes happens.
If one journal is poorly moderated, then you jump to another one. Unlike Mafia bosses, a “problem” with journal moderators means “I wasted a few hours communicating with them and making revisions,” not “They decided to cut off my thumbs.”
This comment and your others in this thread have greatly improved my confidence in SI.
Re-replying:
For people who “are extremely busy, and they use “Did they bother to pass peer review?” as a filter for what they choose to read”, which specific examples are you thinking of, and how much any of them become nontrivial members of our community, or helped us out in nontrivial ways?
I’m sure there are people who a) are very smart, b) look impressive on paper, who c) we’ve contacted about FAI research, and d) have said “I’m not going to pay attention, since this isn’t peer reviewed” (or some equivalent). However, I think that for most of those people, that isn’t their true rejection (http://lesswrong.com/lw/wj/is_that_your_true_rejection/), and they aren’t going to take us seriously anyway. But I could be wrong—what evidence do you have in mind?
A lot of your points are criticisms of blog posts, like “a lot of them don’t have citations”, or “a lot of them are poorly organized”. These are true in many cases. However, if SIAI is considering whether to publish some given idea in paper or blog post form, they could simply spend the (fairly small) effort to write a blog post which was well organized and had citations, thereby making these problems moot.
Journal editors obviously aren’t perfectly analogous to mob bosses. However, I’ve heard many stories from academics of authors spending huge amounts of time and effort trying to get stuff published. In the most recent case, which I discussed with a grad student just a few hours ago, it took hundreds of hours, over a full year. If it’s usually easy to get around that sort of thing, by just publishing in a different journal, why don’t more academics do so?
Your first two questions ask about evidence that I already said I’m not in a position to share yet. I know that’s unsatisfying, but… are your priors on my claims being true really very low? Famous scientists, especially, are barraged with a few purported unifications of quantum theory and relativity every month, and “Did they bother to pass peer review?” is a pretty useful heuristic for them. When you visualize a busy academic receiving CFAI from one person, and The Singularity and Machine Ethics from somebody else, which one do you think they’re more likely to read and take seriously, and why? (Feel free to take this as a rhetorical question.)
The effort required may be much larger than you think. Eliezer finds it very difficult to do that kind of work, for example. (Which is why his papers still read like long blog posts, and include very few citations. CEV even contains zero citations, despite re-treading ground that has been discussed by philosophers for centuries, as “The Singularity and Machine Ethics” shows.)
And if you’ve done all that work, then why not also tweak it for use in a scholarly AI risk wiki, and then combine it with a couple other wiki articles into a paper?
Because their career depends on satisfying their advisors, or on getting published in particular journals. SI researchers’ careers don’t depend on investing hundreds of hours making revisions. If publishing in a certain journal is going to require 30 hours of revisions that don’t actually improve the paper in our eyes, then we aren’t going to bother publishing in that journal.
Both links go to the same place.
Fixed, thanks.
If this is the case, then a significant benefit to Eliezer of trying to get papers published would be that it would be excellent discipline for Eliezer, and would make him an even better scholar.
A benefit that would follow on is that it would establish by example that nobody is above showing their work, acknowledging their debts and being current on the relevant literature. Conceivably Eliezer is such a talented guy that it is of no benefit to him to do these things, but if everyone who thought they were that talented were excused from showing their work and keeping current then progress would slow significantly.
It also avoids reinventing the wheel. No matter how smart Eliezer is, it’s always conceivable that someone else thought of something first and expressed it in rigorous detail with proper citations. A proper literature review avoids this waste of valuable research time.
Luke (and his remote research assistants) have this angle covered.
… and if you’re going through the effort of writing a blog post that’s journal-quality anyway, you might as well go ahead and publish it as a full paper while you’re at it.
Research assistants!
Clearly the grad student (or more likely, their advisor) thought that getting published in journal X was worth enough status to spend over a year working on it. Fake utility functions and all that. Not even academics are perfectly rational.
I imagine that some academics would be more willing to read a paper from the website if it is also published in a journal, especially if they are writing their own journal articles in which they would prefer to cite a journal article to a paper published on a website.
This. Once our ideas enter the academic memespace, they may continue to live independently there. The difficult part is crossing the boundary, but it only has to be done once.
While it would be incorrect to say that I originally came to these issues only via Bostrom’s papers, they certainly made me a lot more interested in the field. Partly because of the prestige of being actually peer-reviewed, but mostly (I think—it was a long time ago) because they were clear, rigorous and self-contained to an extent that few other materials were.
In order to think of some things I do that only have one important purpose, it was necessary to perform the ritual of closing my eyes and thinking about nothing else for a few minutes by the clock.
I plan on assuming things have multiple important purposes and asking for several, e.g. “what do you view as the purposes of X.”
There was nothing wrong with what you said, but it is strange how easily the (my?) mind stops questioning after coming up with just one purpose for something someone is doing. In contrast, when justifying one’s own behavior, it is easy to think of multiple justifications.
It makes some sense in a story about motivated cognition and tribal arguments. It might be that to criticize, we look mostly for something someone does that has no justification, and invest less in attacking someone along a road that has some defenses. A person being criticized invests in defending against those attacks they know are coming, and does not try and think of all possible weaknesses in their position. There is some advantage in being genuinely blind to one’s weaknesses so one can, without lying, be confident in one’s own position.
Maybe it is ultimately unimportant to ask what the “purposes” of someone doing something is, since they will be motivated to justify themselves as much as possible. In this case, asking what the “purpose” is would force them concentrate on their most persuasive and potentially best argument, even if it will rarely actually be the case that one purpose is a large supermajority of their motivation.