Are conferences an inefficient/terrible discussion forum (in addition to academic papers)?
I’m asking this as a follow-up to http://lesswrong.com/lw/d5y/why_academic_papers_are_a_terrible_discussion/, which was written a few years ago, and which I find very interesting.
Many of the arguments advanced in http://lesswrong.com/lw/d5y/why_academic_papers_are_a_terrible_discussion/ (especially inaccessibility) could just as well apply to conferences, too.
I’d also wonder—would you consider conferences to also be a terrible discussion forum? What do you think would be some good alternatives?
The audience for conferences is limited, and people seem to remember only a tiny tiny fraction of everything they’ve encountered in a conference. The ideas in conferences don’t seem to do much for building up platforms of public discussions around the new subjects that are often announced in conferences (rather than, say, on online platforms).
I suppose one could advance the argument that ideas often get brought up/discussed at conferences that wouldn’t be conveniently discussed in any other medium (for now..). But is this mostly because people are too comfortable with what they’re been brought up with?
A conference is a way to get a lot of experts together at one place. Face to face contact is useful for building relationships. A good conference leaves plenty of time for participants to get to know each other.
A conference I recently attended was extremely helpful to my research. I met several scientific grandaddies of my research topic who started the field out years ago that collectively raised important questions that I had not been considering and subsequently addressed in my work, and drew my attention to longstanding sticking points in the field that I altered my approach towards in order to incontrovertibly put to rest. Meeting these people and others also has given me a number of new contacts working on similar topics that will not ignore cold emails and allowed me to see some of their raw data and validate my techniques.
It gave me a great idea about the state of the art in cell biological techniques and ways our lab could improve.
It also introduced me to a number of widely-flung ideas that I would not have considered on my own which I have been following A—out of interest for my own research, B—for purposes of thinking about changing tracks after I graduate, and C—out of personal fascination for the topics.
Thanks. It is important to hear from people who, you know, actually go to conferences.
Really now?
If one had doubts about conferences and junkets, one can imagine asking a random doctor or academic whether conferences really were all that great, and the answer my imagination spits out is that they would probably say ‘yes, conferences are great! I learned about X, Y, and Z there, it was totally useful’. And one would not put much weight on it because of the obvious conflict of interest and because they could easily have heard of X Y and Z a week later when they ran into an article or a colleague brought it up or a relevant paper surfaced in their email alerts or he could have cold-emailed in the first place, and it’s not clear to the disinterested outsider that conferences really are worth the money—you already know that the insiders will say it’s worthwhile because how else are they going to get the money and grants, and why would they be operating or attending if they didn’t think so?
So what have we learned by this exercise of asking an academic with the name CellBioGuy and him saying ‘yes, conferences are great! I learned about microbiological things X, Y, and Z there, it was totally useful’, exactly?
(But hey, at least this means it’s easy to investigate all biases and methods: just ask someone involved… ‘Hey, psychiatrist consulting for big pharmacorp: are you sure it’s OK to be taking thousands of dollars in consulting fees and junkets from them?’ ‘Absolutely! Thanks to the consulting, I keep up with all the cutting-edge techniques and new drugs which has revolutionized my research, and it’s much easier to get access to private datasets!’ ‘Thanks! It’s important to hear from people who, you know, actually consult for big pharmacorps!’)
What have we learned from a random LW person opining about things (s)he likely has no experience with?
(With apologies to the OP).
See also: peer review is terrible, frequentist statistics is terrible, academic career structure is terrible, etc. etc.
And yes, it is important to actually have some first-hand experience regarding stuff you are criticizing, or at the very least ask people who do have it. See, e.g. Scott’s critique of sneaky pharma practices. I would think this is obvious enough to not even need saying (but I wonder sometimes...)
I don’t think academics are incentivized to hide flaws in academia enough that they are just silent about them. Plenty of academics criticize plenty of aspects of academia openly. For example, I recently talked to a very senior person who really hates peer reviewed conference publications (not conferences themselves though).
Not much either, but OP does say true factual things which can be easily checked and which do in fact undermine the claimed benefits of conferences: particularly how lectures are horrible forms of communication.
Yvain is a self-selected critic. In the mean time, there are countless psychiatrists who engage in the practices and see nothing wrong with it and predictably justify it in the same terms CellBioGuy does, and you would learn as much asking them. What distinguishes them? It feels the same way from the inside.
Dem’s fighting words. Peer review is terrible, frequentist statistics as applied usually are terrible, etc. And this is especially clear when you are on the outside looking in, with no need to pretend to politeness or worry about antagonizing peer reviewers (as one n-back researcher told me) and you can smell the rank careerism and watch the desperate evasions of researchers sensationalizing their findings and have the leisure to watch the replications unfold. (I commonly find that the more I learn about a topic, the worse peer-review papers are and the more important replication is; I’ve commented often about dual n-back and how Jaeggi has earned tenure & a lab for a non-result, but to give a more recent example that still rankles, on the topic of black-markets, even the quantitative peer-reviewed papers typically range from junk to 100% steaming bullshit like Dolliver’s*, and I know this because I have access to multiple independent datasets I can replicate claimed results on.)
* fun tidbit: Dolliver says she can’t possibly share her scrape of Silk Road because she signed an NDA.
This strikes me as laughably naive: but that’s exactly what they do! What I hear in private is remarkably different from what I read in published papers, and if academics didn’t remain silent routinely, based on the surveys about how often they manipulate or engage in other questionable research practices, we would be seeing thousands of whistles blowing every day. Which we don’t. This is the norm in every profession: you don’t air your dirty laundry in public.
What can be criticized is very limited, critics are few, and in practice little ever changes. There’s a lot of pressure to conform and be quiet. No profession likes whistleblowers. Oh look, another example of the pressures to not criticize or rock the boat from last week: http://nymag.com/scienceofus/2015/05/how-a-grad-student-uncovered-a-huge-fraud.html
But that’s incredibly weak. By that logic, 4chan being terrible is (a) true, and (b) is evidence that we should shut LW down, because online forums are known for being terrible.
Yes. My point about Scott is he actually knows what he is talking about. Also, while he’s brave for speaking up, he is not exactly getting ran out of Detroit by Big Pharma. The problem with outsiders is while they have no incentives to keep quiet, they also don’t know what they are talking about, unless they did a lot of homework.
“Not knowing what you are talking about, but talking anyways” is a chronic lesswrong disease.
I agree that people will do evil things, and keep quiet about evil things—anywhere. For example, if there is a politically powerful department person involved, etc. But academia is not Stalinist Russia, you are not going to get disappeared for loudly discussing flaws. And in fact, we have periodic academic scandals. Here is Broockman’s paper, btw. It is super nifty.
http://stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf
This is what “not keeping silent about flaws” looks like.
Getting back to conferences: my concrete claims are:
(a) There is no conspiracy of silence about conferences. There doesn’t even seem to be a conspiracy of silence about data fraud.
(b) In fact, conferences are quite useful. For example, one big useful function of conferences is solving the coordination problem of scheduling collaboration time for lots of busy people. Without conferences, some collaborators will never get in the same room to get work done.
(c) Meta question: what does your process for figuring out if conferences are a waste of time look like? Does it involve dealing with any data about actual conferences at any point? Do you think the recent Cambridge conference MIRI helped finance (on decision theory) was a waste of time?
http://www.phil.cam.ac.uk/events/decision-theory-conf
Contrary to what some posters may suspect about evil rent-seeker academics going to tropical paradises on taxpayer dime, academics are super busy, and constant travel is kind of a pain in the ass. I personally wish I could do less conference travel.
Nah, peer review is pretty great. I am glad we had this productive chat about it. Do you do any peer review, gwern, or get any peer review?
That’s like cards against humanity for anything.
Do you attribute the main information exchange to have happened in formal discussion (things on the agenda) or to chatting more informally?
Both were important. Formal discussions and things on the agenda were great for seeing the general state of the art and how people were doing things and how our lab could improve, as well as going through the list of sessions and saying ‘oh wow that looks fascinating’ and getting exposed to new ideas. But my own research got the most direct benefit from unstructured talking with other researchers and people seeing my name and research on the list of posters and coming by to talk, and vice versa.
Having recently attended my first conference, I can say that I gained a LOT more tacit knowledge than explicit knowledge. I got a feel for the types of people doing all the work behind the scenes to make things actually happen, and a generally much broader perspective.
Reading and researching independently and interacting with others with a similarly narrow focus doesn’t lend itself to comprehensive understandings. I try very hard to always get a feel for the big picture before diving into specifics, but individuals or small groups can only ever develop a few ways of looking at things and attacking problems.
In short, you learn the answers to all sorts of questions that you never even thought to wonder about. You can’t google unknown unknowns.
My understanding is that the real benefit to conferences is getting all the experts together in one place and giving them time to socialize and catch up on what the others are doing.
1) Writing and Speaking by Paul Graham is a relevant read.
2) As others have said, X isn’t about Y. Conferences aren’t about discussion (personally, I don’t really know that that’s true; I’m basing this assumption on my hearing lots of other people saying it).
However, I still think there are important questions left to ask:
3) If the goal of conferences isn’t discussion, what is it? I suspect that conference organizers aren’t really asking themselves this question, and that they’re suffering from Lost Purposes. How do stronger (social) relationships lead to more scientific discovery? Is it trust? Informal discussions? I think that science is really really important, so even if the marginal improvement to conferences is small, I sense that it’s multiplied by a big enough number such that the marginal improvement itself is also quite important.
4) Even if the goal of conferences isn’t discussion, discussion is still something that’s really important. “How can discussion be optimized?” is a really important question (similar logic to 3). As to your question of how discussion could be improved, here are some initial thoughts/brainstorming (nothing novel or too useful though):
In person discussion might not be best. Well, I think there’s a place for it, but I think it often takes a lot of time to think before you respond. There are a lot of benefits to communication via writing over communicating in person. My impression is that it should be the main way to have real discussions about difficult things (I’m not in academia, but I’d guess that it already is).
But if you’re going to have in person communication...
I sense that groups of 2-5 are probably best. In my experience, there’s too much friction in bigger groups. I wonder if there’s any research on this?
I think it’s really important to diagram things out. In my experience, people either a) don’t think to do that in a conversation, or b) are too lazy to do it. Yes, it’s not always appropriate because it slows down the flow, but I think it’s appropriate to do a lot more than it’s currently done.
I think there’s probably some interesting uses of social pressure and “pushing people’s buttons” to motivate them (think basketball coaches like Phil Jackson).
Maybe use some sort of technique to establish group cohesion. Working together to solve a common (short-term/pressing) problem? Fight a common enemy?
My experience with conferences (in machine learning):
1) It seems almost universally true that the first few conferences one attends (especially the first one) are incredibly high marginal value, for the reasons that Mars_Colony notes.
2) After that, the marginal value seems to decrease somewhat. I attend 1-2 conferences per year and still get value out of them, but perhaps not enough to justify going except that it’s a great venue in which to spread one’s ideas to a larger audience. On the other hand, if I only went once per year (or per 2 years) then I would probably get more value per conference due to networking (in the good way) with colleagues who I had not seen in a long time.
3) Workshops at conferences can also be very valuable; they often act as a venue for a budding subfield to try to figure out its directions/priorities, or for a budding not-yet-subfield to coalesce into existence.
4) It is not my experience that conferences are a paid vacation. The last 4 ICML locations were: Edinburgh, Atlanta, Beijing, Paris. This seems consistent with what you would choose if you wanted to rotate through Europe, North America, and Asia while picking centralized locations near major airports. The last 4 NIPS locations were: Lake Tahoe, Lake Tahoe, Montreal, Montreal. This is admittedly at least partially optimized for skiing, but I also think it’s pretty hard to call these places boondoggles. Yes, I certainly enjoy being able to go to Beijing for free, though I don’t really understand how this is a strong criticism (note that many companies also pay for their employees to travel places as well).
5) I agree with some others in this thread that it’s annoying for people to randomly opine on how terrible academia is without being informed. (For some reason, it seems in vogue on LessWrong to be anti-academic; I don’t really understand what causes this?) I agree that academia has tons of problems (much like every other human endeavor) but I don’t think that uninformed speculation will do much to solve them.
I think it’s founder effect.
There’s a lot of that here.
Let’s add here, that most of the scientists treat conferences as a form of vacation funded by academia or grant money. So there is a strong bias to find reasons for their necessity and/or benefits.
Hi, you have no possible way to know this.
That has not been my experience. Visiting foreign parts is icing on the cake, but no more than that. There may be a few exceptions, such as a large conference that a colleague of mine once went to in Hawaii. I heard that the lecture theatres were thinly attended.
Within academia you can find parasitic plagiarists like this one:
https://andreasplagiarism.wordpress.com/2020/12/02/andreas-theodorou-committed-plagiarism-in-his-phd-thesis/
(Yes, going to a conference can be inefficient/terrible way of spending your time. It doesn’t prove anything.) There are ways in which conferences fail. There’s too little time to talk to other people, especially if they only came to make their report and then leave; you can’t count on them accommodating you. (And if you want them to show you how they do things, it is even worse, so just plan it ahead.) There are sometimes lectures which you would not learn anything useful from, and you skip them without remorse, but you don’t go to a conference just to skip stuff. There are people who will be less useful to you than you are to them, so what? It is good to be actually needed and not just cited. There are people who strongly disagree with your lab’s approach, and they might not give you coherent feedback—that just means that you should write to them later.
I mean, yeah, it’s not ideal, so what?
Yes, conferences are terrible as a discussion forum, but that’s not really what they are for. They are more-or-less a chance for the people working in the same field to see each other in person and talk to each other informally. The actual presentations are almost beside the point, an excuse for getting together. They can be interesting, but a lot of the stuff people come for happens elsewhere.
What would a world without conferences look like?
What would a world with conferences being seen as natural Shelling points punctuating online discussions look like, with pre-registered forums dedicated to specific questions and at least two weeks allocated for every participant to familiarize herself with current literature? Like, I was chosen to participate, and I spend 14 days just reading up on the topic (and then give a free-to-attend lecture summarizing my understanding.) Possibly on weekends, if my institute does not agree to pay for that fortnight and I want to go badly enough.
And when I come back, I give another lecture on what was discussed.