If one had doubts about conferences and junkets, one can imagine asking a random doctor or academic whether conferences really were all that great, and the answer my imagination spits out is that they would probably say ‘yes, conferences are great! I learned about X, Y, and Z there, it was totally useful’. And one would not put much weight on it because of the obvious conflict of interest and because they could easily have heard of X Y and Z a week later when they ran into an article or a colleague brought it up or a relevant paper surfaced in their email alerts or he could have cold-emailed in the first place, and it’s not clear to the disinterested outsider that conferences really are worth the money—you already know that the insiders will say it’s worthwhile because how else are they going to get the money and grants, and why would they be operating or attending if they didn’t think so?
So what have we learned by this exercise of asking an academic with the name CellBioGuy and him saying ‘yes, conferences are great! I learned about microbiological things X, Y, and Z there, it was totally useful’, exactly?
(But hey, at least this means it’s easy to investigate all biases and methods: just ask someone involved… ‘Hey, psychiatrist consulting for big pharmacorp: are you sure it’s OK to be taking thousands of dollars in consulting fees and junkets from them?’ ‘Absolutely! Thanks to the consulting, I keep up with all the cutting-edge techniques and new drugs which has revolutionized my research, and it’s much easier to get access to private datasets!’ ‘Thanks! It’s important to hear from people who, you know, actually consult for big pharmacorps!’)
What have we learned from a random LW person opining about things (s)he likely has no experience with?
(With apologies to the OP).
See also: peer review is terrible, frequentist statistics is terrible, academic career structure is terrible, etc. etc.
And yes, it is important to actually have some first-hand experience regarding stuff you are criticizing, or at the very least ask people who do have it. See, e.g. Scott’s critique of sneaky pharma practices. I would think this is obvious enough to not even need saying (but I wonder sometimes...)
I don’t think academics are incentivized to hide flaws in academia enough that they are just silent about them. Plenty of academics criticize plenty of aspects of academia openly. For example, I recently talked to a very senior person who really hates peer reviewed conference publications (not conferences themselves though).
What have we learned from a random LW person opining about things (s)he likely has no experience with?
Not much either, but OP does say true factual things which can be easily checked and which do in fact undermine the claimed benefits of conferences: particularly how lectures are horrible forms of communication.
See, e.g. Scott’s critique of sneaky pharma practices. I would think this is obvious enough to not even need saying (but I wonder sometimes...)
Yvain is a self-selected critic. In the mean time, there are countless psychiatrists who engage in the practices and see nothing wrong with it and predictably justify it in the same terms CellBioGuy does, and you would learn as much asking them. What distinguishes them? It feels the same way from the inside.
See also: peer review is terrible, frequentist statistics is terrible, academic career structure is terrible, etc. etc.
Dem’s fighting words. Peer review is terrible, frequentist statistics as applied usually are terrible, etc. And this is especially clear when you are on the outside looking in, with no need to pretend to politeness or worry about antagonizing peer reviewers (as one n-back researcher told me) and you can smell the rank careerism and watch the desperate evasions of researchers sensationalizing their findings and have the leisure to watch the replications unfold. (I commonly find that the more I learn about a topic, the worse peer-review papers are and the more important replication is; I’ve commented often about dual n-back and how Jaeggi has earned tenure & a lab for a non-result, but to give a more recent example that still rankles, on the topic of black-markets, even the quantitative peer-reviewed papers typically range from junk to 100% steaming bullshit like Dolliver’s*, and I know this because I have access to multiple independent datasets I can replicate claimed results on.)
* fun tidbit: Dolliver says she can’t possibly share her scrape of Silk Road because she signed an NDA.
I don’t think academics are incentivized to hide flaws in academia enough that they are just silent about them.
This strikes me as laughably naive: but that’s exactly what they do! What I hear in private is remarkably different from what I read in published papers, and if academics didn’t remain silent routinely, based on the surveys about how often they manipulate or engage in other questionable research practices, we would be seeing thousands of whistles blowing every day. Which we don’t. This is the norm in every profession: you don’t air your dirty laundry in public.
Plenty of academics criticize plenty of aspects of academia openly.
Most important, in Broockman’s opinion, his experience highlights a failure on the part of political science to nurture and assist young researchers who have suspicions about other scientists’ data, but who can’t, or can’t yet, prove any sort of malfeasance. In fact, throughout the entire process, until the very last moment when multiple “smoking guns” finally appeared, Broockman was consistently told by friends and advisers to keep quiet about his concerns lest he earn a reputation as a troublemaker, or — perhaps worse — someone who merely replicates and investigates others’ research rather than plant a flag of his own.
which do in fact undermine the claimed benefits of conferences
But that’s incredibly weak. By that logic, 4chan being terrible is (a) true, and (b) is evidence that we should shut LW down, because online forums are known for being terrible.
Yvain is a self-selected critic.
Yes. My point about Scott is he actually knows what he is talking about. Also, while he’s brave for speaking up, he is not exactly getting ran out of Detroit by Big Pharma. The problem with outsiders is while they have no incentives to keep quiet, they also don’t know what they are talking about, unless they did a lot of homework.
“Not knowing what you are talking about, but talking anyways” is a chronic lesswrong disease.
I agree that people will do evil things, and keep quiet about evil things—anywhere. For example, if there is a politically powerful department person involved, etc. But academia is not Stalinist Russia, you are not going to get disappeared for loudly discussing flaws. And in fact, we have periodic academic scandals. Here is Broockman’s paper, btw. It is super nifty.
This is what “not keeping silent about flaws” looks like.
Getting back to conferences: my concrete claims are:
(a) There is no conspiracy of silence about conferences. There doesn’t even seem to be a conspiracy of silence about data fraud.
(b) In fact, conferences are quite useful. For example, one big useful function of conferences is solving the coordination problem of scheduling collaboration time for lots of busy people. Without conferences, some collaborators will never get in the same room to get work done.
(c) Meta question: what does your process for figuring out if conferences are a waste of time look like? Does it involve dealing with any data about actual conferences at any point? Do you think the recent Cambridge conference MIRI helped finance (on decision theory) was a waste of time?
Contrary to what some posters may suspect about evil rent-seeker academics going to tropical paradises on taxpayer dime, academics are super busy, and constant travel is kind of a pain in the ass. I personally wish I could do less conference travel.
Peer review is terrible
Nah, peer review is pretty great. I am glad we had this productive chat about it. Do you do any peer review, gwern, or get any peer review?
Really now?
If one had doubts about conferences and junkets, one can imagine asking a random doctor or academic whether conferences really were all that great, and the answer my imagination spits out is that they would probably say ‘yes, conferences are great! I learned about X, Y, and Z there, it was totally useful’. And one would not put much weight on it because of the obvious conflict of interest and because they could easily have heard of X Y and Z a week later when they ran into an article or a colleague brought it up or a relevant paper surfaced in their email alerts or he could have cold-emailed in the first place, and it’s not clear to the disinterested outsider that conferences really are worth the money—you already know that the insiders will say it’s worthwhile because how else are they going to get the money and grants, and why would they be operating or attending if they didn’t think so?
So what have we learned by this exercise of asking an academic with the name CellBioGuy and him saying ‘yes, conferences are great! I learned about microbiological things X, Y, and Z there, it was totally useful’, exactly?
(But hey, at least this means it’s easy to investigate all biases and methods: just ask someone involved… ‘Hey, psychiatrist consulting for big pharmacorp: are you sure it’s OK to be taking thousands of dollars in consulting fees and junkets from them?’ ‘Absolutely! Thanks to the consulting, I keep up with all the cutting-edge techniques and new drugs which has revolutionized my research, and it’s much easier to get access to private datasets!’ ‘Thanks! It’s important to hear from people who, you know, actually consult for big pharmacorps!’)
What have we learned from a random LW person opining about things (s)he likely has no experience with?
(With apologies to the OP).
See also: peer review is terrible, frequentist statistics is terrible, academic career structure is terrible, etc. etc.
And yes, it is important to actually have some first-hand experience regarding stuff you are criticizing, or at the very least ask people who do have it. See, e.g. Scott’s critique of sneaky pharma practices. I would think this is obvious enough to not even need saying (but I wonder sometimes...)
I don’t think academics are incentivized to hide flaws in academia enough that they are just silent about them. Plenty of academics criticize plenty of aspects of academia openly. For example, I recently talked to a very senior person who really hates peer reviewed conference publications (not conferences themselves though).
Not much either, but OP does say true factual things which can be easily checked and which do in fact undermine the claimed benefits of conferences: particularly how lectures are horrible forms of communication.
Yvain is a self-selected critic. In the mean time, there are countless psychiatrists who engage in the practices and see nothing wrong with it and predictably justify it in the same terms CellBioGuy does, and you would learn as much asking them. What distinguishes them? It feels the same way from the inside.
Dem’s fighting words. Peer review is terrible, frequentist statistics as applied usually are terrible, etc. And this is especially clear when you are on the outside looking in, with no need to pretend to politeness or worry about antagonizing peer reviewers (as one n-back researcher told me) and you can smell the rank careerism and watch the desperate evasions of researchers sensationalizing their findings and have the leisure to watch the replications unfold. (I commonly find that the more I learn about a topic, the worse peer-review papers are and the more important replication is; I’ve commented often about dual n-back and how Jaeggi has earned tenure & a lab for a non-result, but to give a more recent example that still rankles, on the topic of black-markets, even the quantitative peer-reviewed papers typically range from junk to 100% steaming bullshit like Dolliver’s*, and I know this because I have access to multiple independent datasets I can replicate claimed results on.)
* fun tidbit: Dolliver says she can’t possibly share her scrape of Silk Road because she signed an NDA.
This strikes me as laughably naive: but that’s exactly what they do! What I hear in private is remarkably different from what I read in published papers, and if academics didn’t remain silent routinely, based on the surveys about how often they manipulate or engage in other questionable research practices, we would be seeing thousands of whistles blowing every day. Which we don’t. This is the norm in every profession: you don’t air your dirty laundry in public.
What can be criticized is very limited, critics are few, and in practice little ever changes. There’s a lot of pressure to conform and be quiet. No profession likes whistleblowers. Oh look, another example of the pressures to not criticize or rock the boat from last week: http://nymag.com/scienceofus/2015/05/how-a-grad-student-uncovered-a-huge-fraud.html
But that’s incredibly weak. By that logic, 4chan being terrible is (a) true, and (b) is evidence that we should shut LW down, because online forums are known for being terrible.
Yes. My point about Scott is he actually knows what he is talking about. Also, while he’s brave for speaking up, he is not exactly getting ran out of Detroit by Big Pharma. The problem with outsiders is while they have no incentives to keep quiet, they also don’t know what they are talking about, unless they did a lot of homework.
“Not knowing what you are talking about, but talking anyways” is a chronic lesswrong disease.
I agree that people will do evil things, and keep quiet about evil things—anywhere. For example, if there is a politically powerful department person involved, etc. But academia is not Stalinist Russia, you are not going to get disappeared for loudly discussing flaws. And in fact, we have periodic academic scandals. Here is Broockman’s paper, btw. It is super nifty.
http://stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf
This is what “not keeping silent about flaws” looks like.
Getting back to conferences: my concrete claims are:
(a) There is no conspiracy of silence about conferences. There doesn’t even seem to be a conspiracy of silence about data fraud.
(b) In fact, conferences are quite useful. For example, one big useful function of conferences is solving the coordination problem of scheduling collaboration time for lots of busy people. Without conferences, some collaborators will never get in the same room to get work done.
(c) Meta question: what does your process for figuring out if conferences are a waste of time look like? Does it involve dealing with any data about actual conferences at any point? Do you think the recent Cambridge conference MIRI helped finance (on decision theory) was a waste of time?
http://www.phil.cam.ac.uk/events/decision-theory-conf
Contrary to what some posters may suspect about evil rent-seeker academics going to tropical paradises on taxpayer dime, academics are super busy, and constant travel is kind of a pain in the ass. I personally wish I could do less conference travel.
Nah, peer review is pretty great. I am glad we had this productive chat about it. Do you do any peer review, gwern, or get any peer review?
That’s like cards against humanity for anything.