I probably put in an extra 20-60 hours, so the total is probably closer to 150 - which surprises me. I will add that a lot of the conversion time was dealing with writing more, LaTeX figures and citations, which were all, I think, substantive valuable additions. (Changing to a more scholarly style was not substantively valuable, nor was struggling with latex margins and TikZ for the diagrams, and both took some part of the time.)
Davidmanheim
Karma: 3,639
Thanks, agreed. And as an aside, I don’t think it’s entirely coincidental that neither of the people who agree with you are in the Bay.
I think that the costs usually are worth it far more often than it occurs, from an outside view—which was David’s point, and what I was trying to respond to. I think that it’s more valuable than one expects to actually just jump through the hoops. And especially for people who haven’t yet ever had any outputs actually published, they really should do that at least once.
(Also, sorry for the zombie reply.)
I think this ignores how decisions actually get made, but I think we’re operating at too high a level of abstraction to actually disagree productively.
You’re very unusually proactive, and I think the median member of the community would be far better served if they were more engaged the way you are. Doing that without traditional peer reviewed work is fine, but unusual, and in many ways is more difficult than peer-reviewed publication. And for early career researchers, I think it’s hard to be taken seriously without some more legible record—you have a PhD, but many others don’t.
To respond briefly, I think that people underinvest in (D), and write sub-par forum posts rather than aim for the degree of clarity that would allow them to do (E) at far less marginal cost. I agree that people overinvest in (B)[1], but also think that it’s very easy to tell yourself your work is “actual progress” when you’re doing work that, if submitted to peer-reviewed outlets, would be quickly demolished as duplicative of work you’re unaware of, or incompletely thought-out in other ways.
I also worry that many people have never written a peer reviewed paper, and aren’t thinking through the tradeoff, they just never develop the necessary skills, and can’t ever move to more academic outlets[2]. I say all of this as someone who routinely writes for both peer-reviewed outlets and for the various forums—my thinking needs to be clearer for reviewed work, and I agree that the extraneous costs are high, but I think that the tradeoff in terms of getting feedback and providing something for others to build on, especially others outside of the narrow EA-motivated community, is often worthwhile.Edit to add: But yes, I unambiguously endorse starting with writing Arxiv papers, as they get a lot of the benefit without needing to deal with the costs of review. They do fail to get as much feedback, which is a downside. (It’s also relatively easy to put something on Arxiv and submit to a journal for feedback, and decide whether to finish the process after review.)
- ^
Though much of that work—reviews, restatements, etc. can be valuable despite that.
- ^
To be fair, I may be underestimating the costs of learning the skills for those who haven’t done this—but I do think there’s tons of peer mentorship within EA which can work to greatly reduce those costs, if people are willing to use those resources.
- ^
That seems right.
It’s a reasonable model. One problem with this as a predictive model, however, is that log-rolling happens across issues; a politician might give up on their budget-cutting to kill an anti-business provision, or give up an environmental rule to increase healthcare spending. So the gradients aren’t actually single valued, there’s a complex correlation / tradeoff matrix between them.
they don’t judge those costs to be worth it
Worth it to whom? And if they did work that’s valuable, how much of that value is lost if others who could benefit don’t see it, because it’s written up only informally or not shared widely?
There have also been plenty of other adapatations, ones which were not low-effort. I worked on 2, the Goodhart’s law paper and a paper with Issa Rice on HRAD. Both were very significantly rewritten and expanded into “real” preprints, but I think it was clearly worthwhile.
I mostly agree with this—deep ideas should get relatively less focus, but not stop getting funding / attention. See my EA forum post from last year, Interesting vs. Important Work—A Place EA is Prioritizing Poorly, which makes a related point.
And I think the post here is saying that you should jump through those effort and editing hoops far more often than currently occurs.
If someone says the opportunity cost is not worth it for them, I see that as a claim that a priori might be true or false. Your post seems to imply that almost everyone is making an error in the same direction, and therefore funders should put their thumb on the scale. That’s at least not obvious to me.
I do think this is the wrong calculation, and the error caused by it is widely shared and pushes in the same direction.Publication is a public good, where most of the benefit accrues to others / the public. Obviously costs to individuals are higher than the benefits to them in far more cases than where costs to individuals are higher than the summed benefits to others. And evaluating good accrued to the researchers is the wrong thing to check—if our goal is aligned AI, the question should be the benefit to the field.
Unless I’m missing something, this seems correct, but unhelpful. It doesn’t point towards how we should do anything substantive to understand, much less control AI, it just gives us a way to do better at tasks that help explain current models. Is there something you’re pointing to that would make this more useful than just for prompt engineering or ad-hoc / post-hoc explanation of models we don’t understand?
Mostly correct, but because passing isn’t allowed, it is not necessarily the case that black doesn’t have a forced win.
There’s a different principle that’s important here, which is that the space of bad ways to do things is almost always larger than the set of good ways to do it, and appealing to what has been sufficient so far is at least a great way to ensure you don’t do far worse. I’m not going to try to make the argument fully right here, but in general, doing things differently means you’re risking new failure modes—and the fact that it was once done this way doesn’t avoid the problem, because the situation now is different. (On the other hand, this is a fully generalized argument against trying anything, which is bad if overused. It does function as a reason to exercise significant additional caution.)
These seem like arguments that it should be possible to be very, very cautious, and to create an agent that doesn’t immediately crash and burn due to Russell’s claim, not that they are unlikely, nor that even these agents don’t fail slightly later.
I don’t really see an argument here against the central claim you say you disagree with.
I no longer believe this to be obviously true.
This is based on a straightforward claim from optimization theory, and you don’t address it, nor do you explain your model, other than to vaguely gesture at uncertainties and caution, without looking at whether VoI itself would lead to extremization, nor why caution would be optimal for an agent.
It’s also mostly “conditional on acceptance, homeschooled students do better”—and given the selection bias in the conditional sample, that would reflect a bias against them in admissions, rather than being a fact about homeschooling.
Thanks, reading closely I see how you said that, but it wasn’t clear initially. (There’s an illusion of disagreement, which I’ll christen the “twitter fight fallacy,” where unless the opposite is said clearly, people automatically assume replies are disagreements.)