How does this work in practice when someone has an idea they want to present in a post? Are you suggesting that they go towards some kind of immediate peer review as a Formative Evaluation?
Uh… no. “Peer review” is something that happens to a work after it’s been published (EDIT: Not quite true; see comments below). Formative evaluations are… the opposite of that. I mean, what are you asking, exactly? How do discussions of an idea work? In the usual way: you write a post, people write comments where they discuss the contents of the post. Comments like “what do you mean by that word?”, or “what are some examples of that?”, or “could you clarify what this part means?”, or “if this thing you say in this here part is true, then it seems like X follows—do you agree, and how does this affect your idea?”, or “interesting points; here are some further thoughts on this”, or “here is some related work—what are your thoughts on these things?”, or “does this apply to X?”, or… etc., etc. You know… discussion.
(If you’re not familiar with evaluation in usability engineering, it might also help clarify things if you clicked the links I include and read about how it’s done there; analogies to discussion of posts on a forum should present themselves pretty clearly, I think.)
before an idea is absorbed into the local culture, before it becomes the foundation of a dozen more posts that build on it as an assumption
How many examples are there of this where the consensus was once treating it as an assumption, and now it isn’t?
I don’t understand the connection between your question and what you quoted. I never said anything about anything that “the consensus was once treating it as an assumption, and now it isn’t”, so I’m not sure why I would have any examples of this. Please clarify what you’re asking here?
But you’re saying Formative Evaluations must happen before an idea is absorbed into the local culture. Isn’t when it’s posted too late for that? Since you’re impressing the importance of instantaneous evaluation. I’m just getting very confused how this process looks like in a Forum.
it might also help clarify things if you clicked the links I include and read about
There are a lot of links, which one would you prioritize?
, so I’m not sure why I would have any examples of this. Please clarify what you’re asking here?
Which specific ideas in the past have you seen been absorbed in to the local culture and become the formation of a dozen or more posts prematurely or that needed more Formative evaluation? Those examples.
True, that was definitely a misstatement on my part.
However, what is still the case is that peer review happens when the work is complete. It doesn’t inform the study design, it doesn’t effect serious changes in the direction of the work, etc.
But you’re saying Formative Evaluations must happen before an idea is absorbed into the local culture. Isn’t when it’s posted too late for that?
No. Of course not. Why would that be the case?
Since you’re impressing the importance of instantaneous evaluation.
Instantaneous…? No…
I… genuinely don’t see what you’re saying here. Why would when it’s posted be too late? This doesn’t make any sense to me.
I’m just getting very confused how this process looks like in a Forum.
I mean… I described it. It works by… discussing a post. In the comments. This is… really straightforward and ordinary. I’m really not suggesting anything weird here.
it might also help clarify things if you clicked the links I include and read about
There are a lot of links, which one would you prioritize?
, so I’m not sure why I would have any examples of this. Please clarify what you’re asking here?
Which specific ideas in the past have you seen been absorbed in to the local culture and become the formation of a dozen or more posts prematurely or that needed more Formative evaluation? Those examples.
Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design. You can read the peer review back-and-forth in many journals, they are often pretty interesting. Machine learning and computer science are different because they usually publish in conference proceedings. That means there are very tight deadlines, so it’s more common to rebut the reviewer’s comments outside of very minor changes. In my opinion it’s why peer review is seen so poorly in ML, because there’s not much paper-improvement going on as a result of the process.
Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design.
Ok… how does this work exactly? You submit your paper for peer review, they say “you should’ve done this differently from the start”, and you go back and start over…?
Isn’t that… basically what I said? You submit a basically finished product, which might get rejected and you have to start over. But you’re not submitting a paper for peer review midway through the study, right…?
You submit a finished product, yes, and it can be accepted without revisions, but I have never heard of that happening actually and nobody I know has had that happen to them, I believe. Or, it might get rejected (but if so, no, you don’t have to start over. If it was sent for review, you will receive feedback you can use to improve the study, and you may be invited to resubmit after making those changes, or you might submit the same paper to a different journal). Hopefully, it is accepted with major or minor revisions, so you go away and make the requested changes over a few more months, and then the reviewers take another look. And these changes can, but not always, be significant alterations to the study design.
Examples from my recent experience: I submitted a paper recently that developed a new data analysis method and then evaluated it on two different synthetic datasets. I was then asked by the editor for revisions: obtaining and using observational data as well as synthetic data. That’s not changing the original study design, but it is a new chunk of research, a lot of work, and the results have to be interpreted differently. Another paper that I co-authored has been asked for major revisions which, if implemented, would be a massive change in the setup, data used, analysis methodology and narrative of the paper. The lead author is still deciding if they want to do that or instead to withdraw and resubmit somewhere else. On the other hand, often I have only been asked for minor text changes to explain things more clearly.
In Nature, the peer review files are openly available for each article, and they are pretty interesting to read, because papers there often go through quite significant changes before publication. That’s a good way to get an idea of the ways papers and studies can evolve as they go through the peer review process. But, yeah, I assure you, in my experience as an author and reviewer, it is a collaborative process that can really reshape the study design in some cases.
I mean… I described it. It works by… discussing a post. In the comments. This is… really straightforward and ordinary. I’m really not suggesting anything weird here.
Ah, you’re just suggesting is that people ask more questions like “what does this important word mean?” “Can you give more examples of that”—the onus is falling on the commenters not on some kind of micro-peer-review panel before an author publishes a post?
That the overall process doesn’t need to change, just people ask more of these kinds of questions and to do right after the post is published. Am I oversimplifying it?
At risk of sounding like a broken record: All you’re imploring people to do is to ask more of these types of questions in the comments immediately?
the onus is falling on the commenters not on some kind of micro-peer-review panel before an author publishes a post?
Yes, of course. Just regular commenting on a post. (Of course, the “draft sharing” feature of LW that lets an author share a post draft with some small set of users, for them to comment on it prior to publication, is sort of like a “micro-peer-review panel”, and that’s fine too. Although I wouldn’t call it anything like that; it’s just… commenting on a draft.)
That the overall process doesn’t need to change, just people ask more of these kinds of questions and to do right after the post is published. Am I oversimplifying it?
Right, for people to ask more of these kinds of questions, and for authors to invite more of these kinds of questions, and for authors and other commenters to take these kinds of questions as invitations for discussion.
At risk of sounding like a broken record: All you’re imploring people to do is to ask more of these types of questions in the comments immediately?
And for the moderation system to not interfere in this process. (This has been the biggest obstacle by far to anything like what I suggest working properly on Less Wrong.)
Uh… no. “Peer review” is something that happens to a work
after it’s been published(EDIT: Not quite true; see comments below). Formative evaluations are… the opposite of that. I mean, what are you asking, exactly? How do discussions of an idea work? In the usual way: you write a post, people write comments where they discuss the contents of the post. Comments like “what do you mean by that word?”, or “what are some examples of that?”, or “could you clarify what this part means?”, or “if this thing you say in this here part is true, then it seems like X follows—do you agree, and how does this affect your idea?”, or “interesting points; here are some further thoughts on this”, or “here is some related work—what are your thoughts on these things?”, or “does this apply to X?”, or… etc., etc. You know… discussion.(If you’re not familiar with evaluation in usability engineering, it might also help clarify things if you clicked the links I include and read about how it’s done there; analogies to discussion of posts on a forum should present themselves pretty clearly, I think.)
I don’t understand the connection between your question and what you quoted. I never said anything about anything that “the consensus was once treating it as an assumption, and now it isn’t”, so I’m not sure why I would have any examples of this. Please clarify what you’re asking here?
Incorrect, a peer review is reviewing a draft before the work is published.
But you’re saying Formative Evaluations must happen before an idea is absorbed into the local culture. Isn’t when it’s posted too late for that? Since you’re impressing the importance of instantaneous evaluation. I’m just getting very confused how this process looks like in a Forum.
There are a lot of links, which one would you prioritize?
Which specific ideas in the past have you seen been absorbed in to the local culture and become the formation of a dozen or more posts prematurely or that needed more Formative evaluation? Those examples.
True, that was definitely a misstatement on my part.
However, what is still the case is that peer review happens when the work is complete. It doesn’t inform the study design, it doesn’t effect serious changes in the direction of the work, etc.
No. Of course not. Why would that be the case?
Instantaneous…? No…
I… genuinely don’t see what you’re saying here. Why would when it’s posted be too late? This doesn’t make any sense to me.
I mean… I described it. It works by… discussing a post. In the comments. This is… really straightforward and ordinary. I’m really not suggesting anything weird here.
Huh? No, there’s just the one link: “Formative vs. Summative Evaluations”, at nngroup.com.
Oh, I see. Sure, here’s one: “frame control”.
Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design. You can read the peer review back-and-forth in many journals, they are often pretty interesting. Machine learning and computer science are different because they usually publish in conference proceedings. That means there are very tight deadlines, so it’s more common to rebut the reviewer’s comments outside of very minor changes. In my opinion it’s why peer review is seen so poorly in ML, because there’s not much paper-improvement going on as a result of the process.
The study design… of the paper being reviewed?
Yes.
Ok… how does this work exactly? You submit your paper for peer review, they say “you should’ve done this differently from the start”, and you go back and start over…?
Isn’t that… basically what I said? You submit a basically finished product, which might get rejected and you have to start over. But you’re not submitting a paper for peer review midway through the study, right…?
You submit a finished product, yes, and it can be accepted without revisions, but I have never heard of that happening actually and nobody I know has had that happen to them, I believe. Or, it might get rejected (but if so, no, you don’t have to start over. If it was sent for review, you will receive feedback you can use to improve the study, and you may be invited to resubmit after making those changes, or you might submit the same paper to a different journal). Hopefully, it is accepted with major or minor revisions, so you go away and make the requested changes over a few more months, and then the reviewers take another look. And these changes can, but not always, be significant alterations to the study design.
Examples from my recent experience: I submitted a paper recently that developed a new data analysis method and then evaluated it on two different synthetic datasets. I was then asked by the editor for revisions: obtaining and using observational data as well as synthetic data. That’s not changing the original study design, but it is a new chunk of research, a lot of work, and the results have to be interpreted differently. Another paper that I co-authored has been asked for major revisions which, if implemented, would be a massive change in the setup, data used, analysis methodology and narrative of the paper. The lead author is still deciding if they want to do that or instead to withdraw and resubmit somewhere else. On the other hand, often I have only been asked for minor text changes to explain things more clearly.
In Nature, the peer review files are openly available for each article, and they are pretty interesting to read, because papers there often go through quite significant changes before publication. That’s a good way to get an idea of the ways papers and studies can evolve as they go through the peer review process. But, yeah, I assure you, in my experience as an author and reviewer, it is a collaborative process that can really reshape the study design in some cases.
Ah, you’re just suggesting is that people ask more questions like “what does this important word mean?” “Can you give more examples of that”—the onus is falling on the commenters not on some kind of micro-peer-review panel before an author publishes a post?
That the overall process doesn’t need to change, just people ask more of these kinds of questions and to do right after the post is published. Am I oversimplifying it?
At risk of sounding like a broken record: All you’re imploring people to do is to ask more of these types of questions in the comments immediately?
Yes, of course. Just regular commenting on a post. (Of course, the “draft sharing” feature of LW that lets an author share a post draft with some small set of users, for them to comment on it prior to publication, is sort of like a “micro-peer-review panel”, and that’s fine too. Although I wouldn’t call it anything like that; it’s just… commenting on a draft.)
Right, for people to ask more of these kinds of questions, and for authors to invite more of these kinds of questions, and for authors and other commenters to take these kinds of questions as invitations for discussion.
And for the moderation system to not interfere in this process. (This has been the biggest obstacle by far to anything like what I suggest working properly on Less Wrong.)