There’s no official, endorsed CFAR handbook that’s publicly available for download. The CFAR handbook from summer 2016, which I found on libgen, warns
While you may be tempted to read ahead, be forewarned—we’ve often found that participants have a harder time grasping a given technique if they’ve already anchored themselves on an incomplete understanding. Many of the explanations here are intentionally approximate or incomplete, because we believe this content is best transmitted in person. It helps to think of this handbook as a companion to the workshop, rather than as a standalone resource.
which I think is still their view on the matter.
I have heard that they would be more comfortable with people learning rationality techniques in-person from a friend, so if you know any CFAR alumni you could ask them (they’d probably also have a better answer to your question).
This is, however, a position which has always made me extremely suspicious of CFAR’s output. I wish they would, at the very least, acknowledge what a huge red flag this sort of thing is.
Noting that I do agree with this particular claim.
I see the situation as:
There are, in fact, good reasons that its hard to communicate and demonstrate some things, and that hyperfocus on “what can be made legible to a third party” results in a lot of looking under street lamps, rather than where the value necessarily is. I have very different priors than Said on how suspicious CFAR’s actions are, as well as different firsthand experience that leads me to believe there’s a lot of value in CFAR’s work that Said presumably dismisses.
[this is not “zero suspicion”, but I bet my suspicion takes a very different shape than Said’s]
But, it’s still important for group rationality to have sound game theory re: what sort of ideas gain what sort of momentum. An important meta-agreement/policy is for people and organizations to be clear about the epistemic status of their ideas and positions.
I think it takes effort to maintain the right epistemic state, as groups and as individuals. So I think it would have been better if CFAR explicitly stated in their handbook, or in a public blogpost*, that “yes, this limits how much people should trust us, and right now we think it’s more important for us to focus on developing a good product than trying to make our current ideas legibly trustworthy.”
As habryka goes into here, I think there are some benefits for researchers to focus internally for a while. The benefits are high bandwidth communication, and being able to push ideas farther and faster than they would if they were documenting every possibly-dead-end-approach at every step of the way. But, after a few years of this, it’s important to write up your findings in a more public/legible way, both so that others can critique it and so that others can build on it.
CFAR seems overdue for this. But, also, CFAR has had lots of staff turnover by now and it’s not that useful to think in terms of “what CFAR ought to do” vs “what people who are invested in the CFAR paradigm should do.” (The Multi-Agent Models sequence is a good step here. I think good next steps would be someone writing up several other aspects of the CFAR paradigm with a similar degree of clarity, and good steps after that would be to think about what good critique/evaluation would look like)
I see this as needing a something of a two-way contract, where:
Private researchers credibly commit to doing more public writeups (even though it’s a lot of work that often won’t result in immediate benefit), and at the very least writing up quick, clear epistemic statuses of how people should relate to the research in the meanwhile.
Third party skeptics develop a better understanding of what sort of standards are reasonable, and “cutting researchers exactly the right amount of slack.” I think there’s good reason at this point to be like “geez, CFAR, can you actually write up your stuff and put reasonable disclaimers on things and not ride a wave of vague illegible endorsement?” But my impression is that even if CFAR did all the right things and checked all the right boxes, people would still be frustrated, because the domain CFAR is trying to excel at is in fact very difficult, and a rush towards demonstrability wouldn’t be useful. And I think good criticism needs to understand that.
*I’m not actually sure they haven’t made a public blogpost about this.
“It is very difficult to find a black cat in a dark room—especially when the cat is not there.”
I’ve quoted this saying before, more than once, but I think it’s very applicable to CFAR, and I think it is long past time we acknowledged this.
The point is this: yes, it is possible that what CFAR is trying to excel at is in fact very difficult. It could, indeed, be that CFAR’s techniques are genuinely difficult to teach—and forget about trying to teach them via (non-personally-customized!) text. It could be, yes, that CFAR’s progress in toward the problem they are attacking is, while quite real, nonetheless fiendishly difficult to demonstrate, in any sort of legible way. All of these things are possible.
But there is another possibility. It could be that what CFAR is trying to excel at is not very difficult, but impossible. And because of this, it could be that CFAR’s techniques are not hard to teach, but rather there is nothing there to be taught. It could be that CFAR’s progress toward the problem they are attacking, is not hard to demonstrate, but rather nonexistent.
What you say is not, as far as it goes, wrong. But it seems to be predicated on the notion that there is a cat in the room—hidden quite well, perhaps, but nonetheless findable; and also, secondarily, on the notion that what CFAR is doing to look for said cat is even approximately the sort of thing which has any chance at all of finding it.
But while it may have been sensible to start (fully 10 years ago, now!) with the assumption of an existing and definitely present cat, that was then; we’ve since had 10 years of searching, and no apparent progress. Now, it is a very large room, and the illumination is not at all bright, and we can expect no help from the cat, in our efforts to find it. So perhaps this is the sort of thing that takes 10 years to produce results, or 50, or 500—who knows! But the more time passes with no results, the more we should re-examine our initial assumption. It is even worth considering the possibility of calling off the search. Maybe the cat is just too well-hidden (at least, for now).
Yet CFAR does claim that they’ve found something, don’t they? Not the whole cat, perhaps, but its tail, at least. But they can’t show it to us, because, apparently—contrary to what we thought we were looking for, and expecting to find—it turns out that our ideas about what sort of thing a cat is, and what it even means to find a cat, and how ones knows when one has found it, were mistaken!
Well, I’ve worn out this analogy enough, so here’s my conclusion:
As far as I can tell, CFAR has found nothing (or as near to nothing as hardly matters). The sorts of things they seem to be claiming to have found (to the extent that they’re being at all clear) don’t really seem anything like the sorts of things they set out to look for. And that extent is not very large, because they’re being far, far more secretive than seems to me to be remotely warranted. And CFAR’s behavior in the past has not exactly predisposed me to believe that they are, or will be, honest in reporting their progress (or lack thereof).
If CFAR has found nothing, because the goal is ambitious and progress is difficult, they should say so. “We worked on figuring out how to make humans more rational for nearly a decade but so far, all we have is a catalog of a bunch of things that don’t really work”—that is understandable. CFAR’s actual behavior, not so much—or not, at least, without assuming dishonesty or other deficits of virtue.
On the “is there something worth teaching there” front, I think you’re just wrong, and obviously so from my perspective (since I have, in fact, learned things. Sunset at Noon is probably the best writeup of what CFAR-descended things I’ve learned and why they’re valuable to me).
This doesn’t mean you’re obligated to believe me. I put moderate probability on “There is variation on what techniques are useful for what people, and Said’s mind is shaped such that the CFAR paradigm isn’t useful, and it will never be legible to Said that the CFAR paradigm is useful.” But, enough words have been spent trying to demonstrate things to you that seem obvious to me that it doesn’t seem worth further time on it.
The Multi-Agent Model of Mind is the best current writeup of (one of) the important elements of what I think of as the CFAR paradigm. I think it’d be more useful for you to critique that than to continue this conversation.
I have read your post Sunset at Noon; I do not recall finding much to comment on at all, there (I didn’t really get the impression that it was meant to be a “writeup of … CFAR-descended things”!), but I will re-read it and get back to you.
As for the multi-agent model of mind, I have already critiqued it, though, admittedly, in a haphazard way—a comment here, a comment there… I have not bothered to critique more in-depth, or in a more targeted way, because… well, to be frank, because of the many times when I’ve attempted to really critique anything, on the new Less Wrong, and been rebuffed for being insufficiently “prosocial”, insufficiently “nice”, etc., etc.
Should I understand your suggestion to mean that if I post critiquing comments on some of the posts in the sequence you linked, they will not spawn lengthy threads about niceness, will not be met with moderator comments about whether I am inserting enough caveats about how of course I respect the OP very much, etc.?
I’m sorry if this seems blunt or harsh; I really don’t mean to be antagonistic toward you in particular (or anyone, really)! But if you reply to my critical comments by saying “but we have put forth our best effort; the ball’s in your court now” (an entirely fair response, if true!), then before I run with said ball, I need to know it’s not a waste of my time.
And, to be clear, if you respond with “no, all the politeness rules still apply, you have to stick to them if you want to critique these writeups [on Less Wrong]”, then—fair enough! But I’d like to know it in advance. (In such a case, I of course will not post any such critiques here; I may post them elsewhere, or not at all, as I find convenient.)
I think that your past criticisms have been useful, and I’ve explicitly tried to take them into account in the sequence. E.g. the way I defined subagents in the first post of the sequence, was IIRC in part copy-pasted from an earlier response to you, and it was your previous comment that helped/forced me to clarify what exactly I meant. I’d in fact been hoping to see more comments from you on the posts, and expect them to be useful regardless of the tone.
I think I should actually punt this question to Kaj_Sotala, since they are his posts, and the meta rule is that authors get to set the norms on this posts. But:
a) if I had written the posts, I would see them as “yes, now these are actually at the stage where the sort of critique Said does is more relevant.” I still think it’d be most useful if you came at it from the frame of “What product is Kaj trying to build, and if I think that product isn’t useful, are there different products that would better solve the problem that Kaj’s product is trying to solve?”
b) relatedly, if you have criticism of the Sunset at Noon content I’d be interested in that. (this is not a general rule about whether I want critiques of that sort. Most of my work is downstream of CFAR paradigm stuff, and I don’t want most of my work to turn into a debate about CFAR. But it does seem interesting to revisit SaN through the “how content that Raemon attributes to CFAR holds up to Said” lens)
c) Even if Kaj prefers you not to engage with them (or to engage only in particular ways), it would be fine under the meta-rules for you to start a separate post and/or discussion thread for the purpose of critiquing. I actually think the most useful thing you might do is write a more extensive post that critiques the sequence as a whole.
I think I should actually punt this question to Kaj_Sotala, since they are his posts, and the meta rule is that authors get to set the norms on this posts.
Sure.
I still think it’d be most useful if you came at it from the frame of “What product is Kaj trying to build, and if I think that product isn’t useful, are there different products that would better solve the problem that Kaj’s product is trying to solve?”
Sure, but what if (as seems likely enough) I think there aren’t any different products that better solve the problem…?
I actually think the most useful thing you might do is write a more extensive post that critiques the sequence as a whole.
So, just as a general point (and this is related to the previous paragraph)…
The problem with the norm of writing critiques as separate posts, is that it biases (or, if you like, nudges) critiques toward the sort that constitute points or theses in their own right.
In other words, if you write a post, and I comment to say that your post is dumb and you are dumb for thinking and writing this and the whole thing is wrong and bad (except, you know, in a tactful way), well, that is, at least in some sense, appropriate (or we might say, it is relevant, in the Gricean sense); you wrote a post, I posted a comment about that post. Fine.
But if you write a post, and I write a post of my own the entirety of whose content and message is “post X which Raemon just wrote is wrong and bad etc.”, well, what is that? Who writes a whole post just to say that someone else is wrong? It seems… odd; and also, antagonistic, somehow. “What was the point of this post?”, commenters may inquire; “Surely you didn’t write a whole post just to say that another post is wrong? What’s your take, then? What Raemon said is wrong, but then what’s right?”—and what do I answer? “I have no idea what’s right, but that is wrong, and… that’s all I wanted to say, really.” As I said, this simply looks odd (socially speaking). (And certainly one is much less likely to get any traction or even engagement—except the dubious sort of engagement; the kind which is all meta, no substance.)
And the thing is, many of my critiques (of CFAR stuff, yes, and of many other things that are discussed in rationalist spaces) boil down to just “what you are saying is wrong”. If you ask me what I think the right answer is, in such cases, I will have nothing to offer you. I don’t know what the right answer is. I don’t think you know what the right answer is, either; I don’t think anyone has the right answer. Beyond saying that (hypothetical) you are wrong, I often really don’t have much to add.
But such criticisms are extremely important! Refraining from falsely believing ourselves to have the right answer, or even a good answer, or even “the best answer so far”, when what we actually have is simply wrong—this is extremely important! It is very tempting to think that we’ve found an answer, when we have not. Avoiding this trap is what allows us to keep looking, and eventually (one hopes!) find the actual right answer.
I understand that you are coming at this from a view in which an idea that someone proposes, a “framework”, etc., has value, and we take that idea and we build on it; or perhaps we say “but what about this instead”, and we offer our own idea or framework, and maybe we synthesize them, and together, cooperatively, we work toward the answer. Under that view, what you say makes sense.
My commentary (not quite an objection, really) is just that it’s crucial to instead be able to say “no, actually, that is simply wrong [because reasons X Y Z]”, and have that be the end of (that branch of) the conversation. You had an idea, that idea was wrong, end of story, back to the drawing board.
That having been said, I do find your response entirely reasonable and satisfactory, as far as this specific case goes; thank you. I will reread both your post and Kaj’s sequence, and comment on both (the latter, contingent on Kaj’s approval).
And the thing is, many of my critiques (of CFAR stuff, yes, and of many other things that are discussed in rationalist spaces) boil down to just “what you are saying is wrong”. If you ask me what I think the right answer is, in such cases, I will have nothing to offer you. I don’t know what the right answer is. I don’t think you know what the right answer is, either; I don’t think anyone has the right answer. Beyond saying that (hypothetical) you are wrong, I often really don’t have much to add.
If all you have to say is “this seems wrong”, that… basically just seems fine. [edit to clarify: I mean making a comment, not a post].
I don’t expect most LessWrong users would get annoyed at that. The specific complaint we’ve gotten about you has more to do with the way you Socratic-ly draw people into lengthy conversations that don’t acknowledge the difference in frame, and leave people feeling like it was a waste of time. (This has more to do with implicitly demanding asymmetric effort between you and the author, than about criticism).
I’m not quite sure what you’re saying. Yes, no doubt, no one’s complained about me doing the thing I described—because, obviously, I haven’t ever done it! You say that it “basically seems just fine”, but… I don’t expect that it would actually seem “just fine” if I (or anyone else) were to actually do it.
Of course, I could be wrong. What are three examples of posts that others have written, that boil down simply to “other post X, written by person Y, is wrong”, and which have gotten a good reception? Perhaps if we did a case study or three, we’d gain some more insight into this thing.
(As for the “specific complaint”—there I just don’t know what you mean. Feel free to elaborate, if you like.)
Slight clarification – I think I worded the previous comment confusingly. I meant to say, if the typical LessWrong user wrote a single comment in reply to a post saying “this seems wrong”, I would expect that to basically be fine.
I only recommend the “create a whole new post” thing when an author specifically asks you to stop commenting.
(In some cases I think creating a whole new post would actually be just fine, based on how I’ve seen, say, Eliezer, Robin Hanson, Zvi, Ben Hoffman and Sarah Constantin respond to each other in longform on occasion. In other cases creating a whole new post might go over less well, and/or might be a bit of an experiment rather than a tried-and-true-solution, but I think it’s the correct experiment to try)
Also want to be a clear—if authors are banning or asking lots of users to avoid criticism, I do think the author should take something of a social hit as “a person who can’t accept any criticism”. But I nonetheless think it’s still a better metanorm for established authors to have control over their post’s discussion area.
[The LessWrong team is currently trying to develop a much clearer understanding of what good moderation policies are, which might result in some of my opinions changing over the next few weeks, this is just a quick summary of what I currently believe]
Also want to be a clear—if authors are banning or asking lots of users to avoid criticism, I do think the author should take something of a social hit as “a person who can’t accept any criticism”. But I nonetheless think it’s still a better metanorm for established authors to have control over their post’s discussion area.
Quite. A suggestion, then, if I may: display “how many people has this person banned from their posts” (with, upon a click or mouseover or some such, the full list of users available, who have been thus banned) prominently, when viewing a person’s post (somewhere near the post’s author line, perhaps). This way, if I open a post by one Carol, say, I can see at once that she’s banned 12 people from her posts; I take note of this (as that is unusually many); I then click/mouseover/etc., and see either that all the banned accounts are known trolls and curmudgeons (and conclude that Carol is a sensible person with a low tolerance for low-grade nonsense), or that all the banned accounts are people I judge to be reasonable and polite (and conclude that Carol is a prima donna with a low tolerance for having her ideas challenged).
Something in that space seems basically reasonable. Note that I haven’t prioritized cleaning up (and then improving visibility for) the moderation log in part because the list of users who have ever banned users is actually just extremely short, and meanwhile there’s a lot of other site features that seem higher priority.
I have been revisiting it recently and think it’d be a good thing to include in the nearish future (esp. if I am prioritizing other features that’d make archipelago-norms more likely to actually get used), but for the immediate future I actually think just saying to the few people who’ve expressed concerns ‘yo, when you look at the moderation log almost nobody has used it’ is the right call given limited dev time.
I meant to say, if the typical LessWrong user wrote a single comment in reply to a post saying “this seems wrong”, I would expect that to basically be fine.
Ah, I see. Well, yes. But then, that’s also what I was saying: this sort of thing is generally fine as a comment, but as a post…
I only recommend the “create a whole new post” thing when an author specifically asks you to stop commenting.
I entirely understand your intention here, but consider: this would be even worse, “optics”-wise! “So,” thinks the reader, “this guy was so annoying, with his contrarian objections, that the victim of his nitpicking actually asked him to stop commenting; but he can’t let it go, so he wrote a whole post about it?!” And of course this is an uncharitable perspective, and one which isn’t consistent with “good truth-seeking norms”, etc. But… do you doubt that this is the sort of impression that will, if involuntarily, be formed in the minds of the commentariat?
3. Author says “this is annoying enough that I’d prefer you not to comment on my posts anymore.” [Hopefully, although not necessarily, the author does this knowing that they are basically opting into you now being encouraged by LessWrong moderators to post your criticism elsewhere if you think it’s important. This might not currently be communicated that well but I think it should be]
4. Then you go and write a post titled ‘My thoughts on X’ or ‘Alternative Conversation about X’ or whatever, that says ‘the author seems wrong / bad.’
By that point, sure it might be annoying, but it’s presumably an improvement from the author’s take. (I know that if I wanted to write a post about some high level Weird Introspection Stuff that took a bunch of Weird Introspection Paradigm stuff for granted, I’d personally probably be annoyed if you made the discussion about whether the Weird Introspection Paradigm was even any good, and much less annoyed if you wrote another post saying so.
I might be typical minding, but two important bits from my perspective are ‘getting to have the conversation that I actually wanted to have’, and ‘not being forced to provide my own platform for someone else who I don’t think is arguing in good faith’
Addenda: my Strategies of Personal Growth post is also particularly downstream of CFAR. (I realize that much of it is something you can elsewhere. My perspective is that the main product CFAR provides is a culture that makes it easier to orient this sort of thing, and stick with it. CFAR iterates on “what combination of techniques can you present to a person in 4 days that best help jump-start them into that culture?”, and they chose that feedback-loop-cycle after exploring others and finding them less effective)
One salient thing from the Strategies of Personal Growth perspective (which I attribute to exploration by CFAR researchers) is that many of the biggest improvements you can gain come from healing and removing psychological blockers.
“There is variation on what techniques are useful for what people, and Said’s mind is shaped such that the CFAR paradigm isn’t useful, and it will never be legible to Said that the CFAR paradigm is useful.”
This happens to be phrased such that it could be literally true, but the implication—that the CFAR paradigm is in fact useful (to some people), and that it could potentially be useful (to some people) but the fact of its usefulness could be illegible to me—cannot be true. (Or, to be more precisely, it cannot be true simultaneously with the claim being true that “the CFAR paradigm” constitutes CFAR succeeding at finding [at least part of] what they were looking for. Is this claim being made? It seems like it is—if not, that should be made clear!)
The reason is simple: the kind of thing that CFAR (claimed to have) set out to look for, is the kind of thing that should be quite legible even to very skeptical third parties. “We found what we were looking for, but you just can’t tell that we did” is manifestly an evasion.
The reason is simple: the kind of thing that CFAR (claimed to have) set out to look for, is the kind of thing that should be quite legible even to very skeptical third parties.
What is your current model of what CFAR “claimed to have set out to look for”? I don’t actually know much of an explicit statement of what CFAR was trying to look for, besides the basic concepts of “applied rationality”.
But while it may have been sensible to start (fully 10 years ago, now!)
Correction: CFAR was started in 2012 (though I believe some of the founders ran rationality camps the previous summer, in 2011), so it’s been 7 (or 8) years, not 10.
Less Wrong, however, was launched in 2009, and that is what I was referring to (namely, Eliezer’s posts about the community and Bayesian dojos and so forth).
a harder time grasping a given technique if they’ve already anchored themselves on an incomplete understanding
This is certainly theoretically possible, but I’m very suspicious of it on reversal test grounds: if additional prior reading is bad, then why isn’t less prior reading even better? Should aspiring rationalists notread the Sequences for fear of an incomplete understanding spoiling themselves for some future $3,900 CfAR workshop? (And is it bad that I know about the reversal test without having attended a CfAR workshop?)
I feel the same way about schoolteachers who discourage their students from studying textbooks on their own (because they “should” be learning that material by enrolling in the appropriate school course). Yes, when trying to learn from a book, there is some risk of making mistakes that you wouldn’t make with the help of a sufficiently attentive personal tutor (which, realistically, you’re not going to get from attending lecture classes in school anyway). But given the alternative of placing my intellectual trajectory at the mercy of an institution that has no particular reason to care about my welfare, I think I’ll take my chances.
Note that I’m specifically reacting to the suggestion that people not read things for their own alleged benefit. If the handbook had just said, “Fair warning, this isn’t a substitute for the workshop because there’s a lot of stuff we don’t know how to teach in writing,” then fine; that seems probably true. What I’m skeptical of is hypothesized non-monotonicity whereby additional lower-quality study allegedly damages later higher-quality study. First, because I just don’t think it’s true on the merits: I falsifiably predict that, e.g., math students who read the course textbook on their own beforehand will do much better in the course than controls who haven’t. (Although the pre-readers might annoy teachers whose jobs are easier if everyone in the class is obedient and equally ignorant.) And second, because the general cognitive strategy of waiting for the designated teacher to spoonfeed you the “correct” version carries massive opportunity costs when iterated (even if spoonfeeding is generally higher-quality than autodidactism, and could be much higher-quality in some specific cases).
You use math as an example, but that’s highly focused on System 2 learning. That suggests that you have false assumptions about what CFAR is trying to teach.
There are many subjects where written instructions are much less valuable than instruction that includes direct practice: circling, karate, meditation, dancing, etc. Most of those analogies are fairly imperfect, and some have partially useful written instructions (in the case of meditation, the written version might have lagged in-person instruction by many centuries). Circling is the example that I’d consider most apt, but it won’t mean much to people who haven’t taken a good circling workshop.
A different analogy, which more emphasizes the costs of false assumptions: people often imagine that economics teaches something like how to run a good business or how to predict the stock market, because there isn’t any slot in their worldview for what a good economics course actually teaches. There are plenty of mediocre executive summaries of economics, which fail to convey to most people that economics requires a pervasive worldview shift (integrating utilitarianism, empiricism about preferences, and some counterintuitive empirical patterns).
The CFAR handbook is more like the syllabus for an economics course than it is like an economics textbook, and a syllabus is useless (possibly harmful) for teaching economics to people who have bad assumptions about what kind of questions economics answers. (This analogy is imperfect because economics textbooks have been written, unlike a CFAR textbook.)
Maybe CFAR is making a mistake, but it appears that the people who seem most confident about that usually seem to be confused about what it is that CFAR is trying to teach.
Reading the sequences, or reading about the reversal test, are unlikely to have much relevance to what CFAR teaches. Just be careful not to imagine that they’re good examples of what CFAR is about.
Sometimes, we don’t know how to teach a subject in writing because the subject matter is inherently about action (rather than concepts, analysis, explanation, prediction, numbers, words, etc.).
But sometimes, we don’t know how to teach a subject in writing because there is, in fact, nothing (or, at best, nothing much) to be taught. Sometimes, a subject is actually empty (or mostly empty) of content.
In the latter case, attempting to write it down reveals this (and opens the alleged “content” to criticism)—whereas in person, the charisma of the instructors, the social pressure of being in a group of others who are there to receive the instruction, possibly the various biases associated with having made some costly sacrifice (time, money, etc.) to be there, possibly the various biases associated with the status dynamics at play (e.g. if the instructors are respected, or at least if those around you act as if they are), all serve to mask the fundamental emptiness of what is being “taught”.
I leave it to the reader to discern which of the given examples fall into which category. I will only note that while the subjects found in the former category are often difficult to teach, nevertheless one’s mastery of them, and their effectiveness, is usually quite easy to verify—because action can be demonstrated.
Meditation is action, in some important sense, and mostly can’t be demonstrated.
It is hard to reliably distinguish between the results of peer pressure and actual learning. I think CFAR’s best reply to this has been it’s refund policy: last I knew they offered full refunds to anyone who requested it within one year (although I can’t find any online mention of their current policy).
Meditation is action, in some important sense, and mostly can’t be demonstrated.
Everything is “action” in “some sense”. (Whether that sense is “important”, in any given case, is a matter of perspective.)
As far as I am concerned—for the purposes of this topic—if it can’t be demonstrated, it ain’t action.
It is hard to reliably distinguish between the results of peer pressure and actual learning.
I submit to you that if this is true of any given case, then that is an excellent signal that no actual learning has taken place. (And the more true it is—the harder it is to distinguish between actual learning and the results of various biases, social pressure included—the stronger the signal is.)
There are many subjects where written instructions are much less valuable than instruction that includes direct practice: circling, karate, meditation, dancing, etc.
Yes, I agree: for these subjects, the “there’s a lot of stuff we don’t know how to teach in writing” disclaimer I suggested in the grandparent would be a big understatement.
a syllabus is useless (possibly harmful) for teaching economics to people who have bad assumptions about what kind of questions economics answers
Useless, I can believe. (The extreme limiting case of “there’s a lot of stuff we don’t know how to teach in this format” is “there is literally nothing we know how to teach in this format.”) But harmful? How? Won’t the unexpected syllabus section titles at least disabuse them of their bad assumptions?
Reading the sequences [...] are unlikely to have much relevance to what CFAR teaches.
Really? The tagline on the website says, “Developing clear thinking for the sake of humanity’s future.” I guess I’m having trouble imagining a developing-clear-thinking-for-the-sake-of-humanity’s-future curriculum for which the things we write about on this website would be irrelevant. The “comfort zone expansion” exercises I’ve heard about would qualify, but Sequences-knowledge seems totally relevant to something like, say, double crux.
(It’s actually pretty weird/surprising that I’ve never personally been to a CfAR workshop! I think I’ve been assuming that my entire social world has already been so anchored on the so-called “rationalist” community for so long, that the workshop proper would be superfluous.)
The idea that CFAR would be superfluous is fairly close to the kind of harm that CFAR worries about. (You might have been right to believe that it would have been superfluous in 2012, but CFAR has changed since then in ways that it hasn’t managed to make very legible.)
I think meditation provides the best example for illustrating the harm. It’s fairly easy to confuse simple meditation instructions (e.g. focus on your breath, sit still with a straight spine) with the most important features of meditation. It’s fairly easy to underestimate the additional goals of meditation, because they’re hard to observe and don’t fit well with more widely accepted worldviews.
My experience suggests that getting value out of meditation is heavily dependent on a feeling (mostly at a system 1 level) that I’m trying something new, and there were times when I wasn’t able to learn from meditation, because I mistakenly thought that focusing on my breath was a much more central part of meditation than it actually is.
The times when I got more value out of meditation were times when I tried new variations on the instructions, or new environments (e.g. on a meditation retreat). I can’t see any signs that the new instructions or new environment were inherently better at teaching meditation. It seems to have been mostly that any source of novelty about the meditation makes me more alert to learning from it.
My understanding is that CFAR is largely concerned that participants will mistakenly believe that they’ve already learned something that CFAR is teaching, and that will sometimes be half-true—participants may know it at a system 2 level, when CFAR is trying to teach other parts of their minds that still reject it.
I think I experienced that a bit, due to having experience with half-baked versions of early CFAR before I took a well-designed version of their workshop. E.g. different parts of my mind have different attitudes to acknowledging my actual motivations when they’re less virtuous than the motivations that my system 2 endorses. I understood that pretty well at some level before CFAR existed, yet there are still important parts of my mind that cling to self-deceptive beliefs about my motives.
CFAR likely can’t teach a class that’s explicitly aimed at that without having lots of participants feel defensive about their motives, in a way that makes them less open to learning. So they approach it via instruction that is partly focused on teaching other things that look more mundane and practical. Those other things often felt familiar enough to me that I reacted by saying: I’ll relax now and conserve my mental energy for some future part of the curriculum that’s more novel. That might have led me to do the equivalent of what I did when I was meditating the same way repeatedly without learning anything new. How can I tell whether that caused me to miss something important?
A key problem is moving from knowing about a technique to action. When you know 20 techniques and use no one of them it’s harder to get you to actually use the 21st technique that you are taught then if you start out with less techniques in your head and have no established pattern of not doing any of the exercises that you were taught.
There’s less that needs unlearning if you haven’t been exposed to material beforehand.
I would still err on the side of being more public with information, but I do understand that there is a tradeoff.
I can offer an explanation that might fit. Rationalists tend toward expertise mode thinking (expert from the torbert action logic framework). Behaviour like reading the book is in line with the expert behaviour.
Cfar techniques and related in-person methods are not always about being the expert, they are about doing the best thing. Being a better expert is not always the same as being the better munchkin, the better person or the person who can step out of their knowledge beliefs.
In theory, the expert thing is the best thing. In theory there’s no difference between theory and practice, in practice, there’s a big difference between theory and practice.
Having said that, I’ve never done cfar, and I teach workshops monthly in Sydney and I think they are wrong to discourage sharing of their resources. As the same time I accept the idea of intellectual property being protected even if that’s not the case they are claiming.
(I’m in the process of writing up my resources into a collection)
As the same time I accept the idea of intellectual property being protected even if that’s not the case they are claiming.
I suspect that this is the real reason. Although if the much vaster sequences by Yudkowsky are freely available I don’t see it as a good justification for not making the CFAR handbook available.
It’s pretty uncharitable of you to just accuse CfAR of lying like that! If the actual reason were “Many of the explanations here are intentionally approximate or incomplete because we predict that this handbook will be leaked and we don’t want to undercut our core product,” then the handbook would have just said that.
Wait, are you invoking the principle of charity as an epistemic axiom (“assume people don’t lie”)? Why would that be truth-aligned at all?
If you didn’t mean to invoke the principle of charity, why not just say it’s likely to be incorrect based on priors, CFAR’s reputation, etc, instead of using the word “uncharitable” as an insult?
You caught me—introspecting, I think the grandparent was written in a spirit of semi-deliberate irony. (“Semi” because it just felt like the “right” thing to say there; I don’t think I put a lot of effort into modeling how various readers would interpret it.)
Roland is speculating that the real reason for intentionally incomplete explanations in the handbook is different from the stated reason, and I offered a particularly blunt phrasing (“we don’t want to undercut our core product”) of the hypothesized true reason, and suggested that that’s what the handbook would have said in that case. I think I anticipated that a lot of readers would find my proposal intuitively preposterous: “everyone knows” that no one would matter-of-factly report such a self-interested rationale (especially when writing on behalf of an organization, rather than admitting a vice among friends). That’s why the earlierscenesinthe 2009 film The Invention of Lying, or your post “Act of Charity”, are (typically) experienced as absurdist comedy rather than an inspiring and heartwarming portrayal of a more truthful world.
But it shouldn’t be absurd for the stated reason and the real reason to be the same! Particularly for an organization like CfAR which is specifically about advancing the art of rationality. And, I don’t know—I think sometimes I talk in a way that makes me seem more politically naïve than I actually am, because I feel as if the “naïve” attitude is in some way normative? (“You really think someone would do that? Just go on the internet and tell lies?”) Arguably this is somewhat ironic (being deceptive about your ability to detect deception is probably not actually the same thing as honesty), but I haven’t heretofore analyzed this behavioral pattern of mine in enough detail to potentially decide to stop doing it??
I think another factor might be that I feel guilty about being “mean” to CfAR in the great-great-great grandparent comment? (CfAR isn’t a person and doesn’t have feelings, but my friend who works there is and does.) Such that maybe the emotional need to signal that I’m still fundamentally loyal to the “mainstream rationality” tribe (despite the underlying background situation where I’ve been collaborating with you and Ben and Michael to discuss what you see as fatal deficits of integrity in “the community” as presently organized) interacted with my preëxisting tendency towards semi-performative naiveté in a way that resulted in me writing a bad blog comment? It’s a good thing you were here to hold me to account for it!
I thought your comment was fine and the irony was obvious, but this kind of misunderstanding can be easily avoided by making the straightforward reading more boring, like so:
Given that CfAR is an organization which is specifically about seeking truth, one could safely assume that if the actual reason were “Many of the explanations here are intentionally approximate or incomplete because we predict that this handbook will be leaked and we don’t want to undercut our core product,” then the handbook would have just said that. To do otherwise would be to call the whole premise into question!
Yeah, I would have expected Jessica to get it, except that I suspect she’s also executing a strategy of habitual Socratic irony (but without my additional innovation of immediately backing down and unpacking the intent when challenged), which doesn’t work when both sides of a conversation are doing it.
I actually didn’t get it. I was confused but I didn’t consciously generate the hypothesis that it was ironic.
I think I don’t share the background assumption that it is overwhelmingly obvious that CFAR wouldn’t tell the truth about this in their handbook. I also reflectively endorse a policy of calling out things that could easily be mistaken for sincere (though not obvious sarcasm), in order to ensure common knowledge.
Quick note for your model of how people interpret various kinds of writing, my initial read of your comment was to put a 60% probability on “Zack is currently undergoing a pendulum swing in the direction away from calling people out on lying, and overcompensating.” (which was wrong and/or overconfident on my part)
In theory, the expert thing is the best thing. In theory there’s no difference between theory and practice, in practice, there’s a big difference between theory and practice.
We’ve all heard this sort of thing many times, of course. The best response is probably Schopenhauer’s:
“That’s all very well in theory, but it won’t do in practice.” In this sophism you admit the premisses but deny the conclusion, in contradiction with a well-known rule of logic. The assertion is based upon an impossibility: what is right in theory must work in practice; and if it does not, there is a mistake in the theory; something has been overlooked and not allowed for; and, consequently, what is wrong in practice is wrong in theory too.
the torbert action logic framework
You are, I assume, referring to the ideas of this person? He appears to be some variety of management consultant. Is there any reason to take this “action logic” of his seriously? It seems to be yet another among the many, many self-help / management consulting / etc. “frameworks” or “systems” etc. Do any of his ideas have any empirical verification, or… well, anything, really?
That’s up to you. I’ve got a lot of value from the structure he outlines. It’s a lot more reasoned than some of the other mysterious odd things I read.
If there is something wrong with the theory and the way it maps to the practice, is it better to read more theory or do more practice and make new theories? I would suggest it depends on the person and what they have found to work in the past. And also with an awareness to the loops of bad habits—“sharpen the saw” type problems. Sometimes it’s more valuable to stop sharpening the saw, and start cutting down the tree. (rationality frame of thinking loves to sharpen more and cut less)
There’s no official, endorsed CFAR handbook that’s publicly available for download. The CFAR handbook from summer 2016, which I found on libgen, warns
which I think is still their view on the matter.
I have heard that they would be more comfortable with people learning rationality techniques in-person from a friend, so if you know any CFAR alumni you could ask them (they’d probably also have a better answer to your question).
This is, however, a position which has always made me extremely suspicious of CFAR’s output. I wish they would, at the very least, acknowledge what a huge red flag this sort of thing is.
Noting that I do agree with this particular claim.
I see the situation as:
There are, in fact, good reasons that its hard to communicate and demonstrate some things, and that hyperfocus on “what can be made legible to a third party” results in a lot of looking under street lamps, rather than where the value necessarily is. I have very different priors than Said on how suspicious CFAR’s actions are, as well as different firsthand experience that leads me to believe there’s a lot of value in CFAR’s work that Said presumably dismisses.
[this is not “zero suspicion”, but I bet my suspicion takes a very different shape than Said’s]
But, it’s still important for group rationality to have sound game theory re: what sort of ideas gain what sort of momentum. An important meta-agreement/policy is for people and organizations to be clear about the epistemic status of their ideas and positions.
I think it takes effort to maintain the right epistemic state, as groups and as individuals. So I think it would have been better if CFAR explicitly stated in their handbook, or in a public blogpost*, that “yes, this limits how much people should trust us, and right now we think it’s more important for us to focus on developing a good product than trying to make our current ideas legibly trustworthy.”
As habryka goes into here, I think there are some benefits for researchers to focus internally for a while. The benefits are high bandwidth communication, and being able to push ideas farther and faster than they would if they were documenting every possibly-dead-end-approach at every step of the way. But, after a few years of this, it’s important to write up your findings in a more public/legible way, both so that others can critique it and so that others can build on it.
CFAR seems overdue for this. But, also, CFAR has had lots of staff turnover by now and it’s not that useful to think in terms of “what CFAR ought to do” vs “what people who are invested in the CFAR paradigm should do.” (The Multi-Agent Models sequence is a good step here. I think good next steps would be someone writing up several other aspects of the CFAR paradigm with a similar degree of clarity, and good steps after that would be to think about what good critique/evaluation would look like)
I see this as needing a something of a two-way contract, where:
Private researchers credibly commit to doing more public writeups (even though it’s a lot of work that often won’t result in immediate benefit), and at the very least writing up quick, clear epistemic statuses of how people should relate to the research in the meanwhile.
Third party skeptics develop a better understanding of what sort of standards are reasonable, and “cutting researchers exactly the right amount of slack.” I think there’s good reason at this point to be like “geez, CFAR, can you actually write up your stuff and put reasonable disclaimers on things and not ride a wave of vague illegible endorsement?” But my impression is that even if CFAR did all the right things and checked all the right boxes, people would still be frustrated, because the domain CFAR is trying to excel at is in fact very difficult, and a rush towards demonstrability wouldn’t be useful. And I think good criticism needs to understand that.
*I’m not actually sure they haven’t made a public blogpost about this.
“It is very difficult to find a black cat in a dark room—especially when the cat is not there.”
I’ve quoted this saying before, more than once, but I think it’s very applicable to CFAR, and I think it is long past time we acknowledged this.
The point is this: yes, it is possible that what CFAR is trying to excel at is in fact very difficult. It could, indeed, be that CFAR’s techniques are genuinely difficult to teach—and forget about trying to teach them via (non-personally-customized!) text. It could be, yes, that CFAR’s progress in toward the problem they are attacking is, while quite real, nonetheless fiendishly difficult to demonstrate, in any sort of legible way. All of these things are possible.
But there is another possibility. It could be that what CFAR is trying to excel at is not very difficult, but impossible. And because of this, it could be that CFAR’s techniques are not hard to teach, but rather there is nothing there to be taught. It could be that CFAR’s progress toward the problem they are attacking, is not hard to demonstrate, but rather nonexistent.
What you say is not, as far as it goes, wrong. But it seems to be predicated on the notion that there is a cat in the room—hidden quite well, perhaps, but nonetheless findable; and also, secondarily, on the notion that what CFAR is doing to look for said cat is even approximately the sort of thing which has any chance at all of finding it.
But while it may have been sensible to start (fully 10 years ago, now!) with the assumption of an existing and definitely present cat, that was then; we’ve since had 10 years of searching, and no apparent progress. Now, it is a very large room, and the illumination is not at all bright, and we can expect no help from the cat, in our efforts to find it. So perhaps this is the sort of thing that takes 10 years to produce results, or 50, or 500—who knows! But the more time passes with no results, the more we should re-examine our initial assumption. It is even worth considering the possibility of calling off the search. Maybe the cat is just too well-hidden (at least, for now).
Yet CFAR does claim that they’ve found something, don’t they? Not the whole cat, perhaps, but its tail, at least. But they can’t show it to us, because, apparently—contrary to what we thought we were looking for, and expecting to find—it turns out that our ideas about what sort of thing a cat is, and what it even means to find a cat, and how ones knows when one has found it, were mistaken!
Well, I’ve worn out this analogy enough, so here’s my conclusion:
As far as I can tell, CFAR has found nothing (or as near to nothing as hardly matters). The sorts of things they seem to be claiming to have found (to the extent that they’re being at all clear) don’t really seem anything like the sorts of things they set out to look for. And that extent is not very large, because they’re being far, far more secretive than seems to me to be remotely warranted. And CFAR’s behavior in the past has not exactly predisposed me to believe that they are, or will be, honest in reporting their progress (or lack thereof).
If CFAR has found nothing, because the goal is ambitious and progress is difficult, they should say so. “We worked on figuring out how to make humans more rational for nearly a decade but so far, all we have is a catalog of a bunch of things that don’t really work”—that is understandable. CFAR’s actual behavior, not so much—or not, at least, without assuming dishonesty or other deficits of virtue.
P.S.:
This is rather suspicious all on its own.
On the “is there something worth teaching there” front, I think you’re just wrong, and obviously so from my perspective (since I have, in fact, learned things. Sunset at Noon is probably the best writeup of what CFAR-descended things I’ve learned and why they’re valuable to me).
This doesn’t mean you’re obligated to believe me. I put moderate probability on “There is variation on what techniques are useful for what people, and Said’s mind is shaped such that the CFAR paradigm isn’t useful, and it will never be legible to Said that the CFAR paradigm is useful.” But, enough words have been spent trying to demonstrate things to you that seem obvious to me that it doesn’t seem worth further time on it.
The Multi-Agent Model of Mind is the best current writeup of (one of) the important elements of what I think of as the CFAR paradigm. I think it’d be more useful for you to critique that than to continue this conversation.
I have read your post Sunset at Noon; I do not recall finding much to comment on at all, there (I didn’t really get the impression that it was meant to be a “writeup of … CFAR-descended things”!), but I will re-read it and get back to you.
As for the multi-agent model of mind, I have already critiqued it, though, admittedly, in a haphazard way—a comment here, a comment there… I have not bothered to critique more in-depth, or in a more targeted way, because… well, to be frank, because of the many times when I’ve attempted to really critique anything, on the new Less Wrong, and been rebuffed for being insufficiently “prosocial”, insufficiently “nice”, etc., etc.
Should I understand your suggestion to mean that if I post critiquing comments on some of the posts in the sequence you linked, they will not spawn lengthy threads about niceness, will not be met with moderator comments about whether I am inserting enough caveats about how of course I respect the OP very much, etc.?
I’m sorry if this seems blunt or harsh; I really don’t mean to be antagonistic toward you in particular (or anyone, really)! But if you reply to my critical comments by saying “but we have put forth our best effort; the ball’s in your court now” (an entirely fair response, if true!), then before I run with said ball, I need to know it’s not a waste of my time.
And, to be clear, if you respond with “no, all the politeness rules still apply, you have to stick to them if you want to critique these writeups [on Less Wrong]”, then—fair enough! But I’d like to know it in advance. (In such a case, I of course will not post any such critiques here; I may post them elsewhere, or not at all, as I find convenient.)
I think that your past criticisms have been useful, and I’ve explicitly tried to take them into account in the sequence. E.g. the way I defined subagents in the first post of the sequence, was IIRC in part copy-pasted from an earlier response to you, and it was your previous comment that helped/forced me to clarify what exactly I meant. I’d in fact been hoping to see more comments from you on the posts, and expect them to be useful regardless of the tone.
I think I should actually punt this question to Kaj_Sotala, since they are his posts, and the meta rule is that authors get to set the norms on this posts. But:
a) if I had written the posts, I would see them as “yes, now these are actually at the stage where the sort of critique Said does is more relevant.” I still think it’d be most useful if you came at it from the frame of “What product is Kaj trying to build, and if I think that product isn’t useful, are there different products that would better solve the problem that Kaj’s product is trying to solve?”
b) relatedly, if you have criticism of the Sunset at Noon content I’d be interested in that. (this is not a general rule about whether I want critiques of that sort. Most of my work is downstream of CFAR paradigm stuff, and I don’t want most of my work to turn into a debate about CFAR. But it does seem interesting to revisit SaN through the “how content that Raemon attributes to CFAR holds up to Said” lens)
c) Even if Kaj prefers you not to engage with them (or to engage only in particular ways), it would be fine under the meta-rules for you to start a separate post and/or discussion thread for the purpose of critiquing. I actually think the most useful thing you might do is write a more extensive post that critiques the sequence as a whole.
Sure.
Sure, but what if (as seems likely enough) I think there aren’t any different products that better solve the problem…?
So, just as a general point (and this is related to the previous paragraph)…
The problem with the norm of writing critiques as separate posts, is that it biases (or, if you like, nudges) critiques toward the sort that constitute points or theses in their own right.
In other words, if you write a post, and I comment to say that your post is dumb and you are dumb for thinking and writing this and the whole thing is wrong and bad (except, you know, in a tactful way), well, that is, at least in some sense, appropriate (or we might say, it is relevant, in the Gricean sense); you wrote a post, I posted a comment about that post. Fine.
But if you write a post, and I write a post of my own the entirety of whose content and message is “post X which Raemon just wrote is wrong and bad etc.”, well, what is that? Who writes a whole post just to say that someone else is wrong? It seems… odd; and also, antagonistic, somehow. “What was the point of this post?”, commenters may inquire; “Surely you didn’t write a whole post just to say that another post is wrong? What’s your take, then? What Raemon said is wrong, but then what’s right?”—and what do I answer? “I have no idea what’s right, but that is wrong, and… that’s all I wanted to say, really.” As I said, this simply looks odd (socially speaking). (And certainly one is much less likely to get any traction or even engagement—except the dubious sort of engagement; the kind which is all meta, no substance.)
And the thing is, many of my critiques (of CFAR stuff, yes, and of many other things that are discussed in rationalist spaces) boil down to just “what you are saying is wrong”. If you ask me what I think the right answer is, in such cases, I will have nothing to offer you. I don’t know what the right answer is. I don’t think you know what the right answer is, either; I don’t think anyone has the right answer. Beyond saying that (hypothetical) you are wrong, I often really don’t have much to add.
But such criticisms are extremely important! Refraining from falsely believing ourselves to have the right answer, or even a good answer, or even “the best answer so far”, when what we actually have is simply wrong—this is extremely important! It is very tempting to think that we’ve found an answer, when we have not. Avoiding this trap is what allows us to keep looking, and eventually (one hopes!) find the actual right answer.
I understand that you are coming at this from a view in which an idea that someone proposes, a “framework”, etc., has value, and we take that idea and we build on it; or perhaps we say “but what about this instead”, and we offer our own idea or framework, and maybe we synthesize them, and together, cooperatively, we work toward the answer. Under that view, what you say makes sense.
My commentary (not quite an objection, really) is just that it’s crucial to instead be able to say “no, actually, that is simply wrong [because reasons X Y Z]”, and have that be the end of (that branch of) the conversation. You had an idea, that idea was wrong, end of story, back to the drawing board.
That having been said, I do find your response entirely reasonable and satisfactory, as far as this specific case goes; thank you. I will reread both your post and Kaj’s sequence, and comment on both (the latter, contingent on Kaj’s approval).
If all you have to say is “this seems wrong”, that… basically just seems fine. [edit to clarify: I mean making a comment, not a post].
I don’t expect most LessWrong users would get annoyed at that. The specific complaint we’ve gotten about you has more to do with the way you Socratic-ly draw people into lengthy conversations that don’t acknowledge the difference in frame, and leave people feeling like it was a waste of time. (This has more to do with implicitly demanding asymmetric effort between you and the author, than about criticism).
I’m not quite sure what you’re saying. Yes, no doubt, no one’s complained about me doing the thing I described—because, obviously, I haven’t ever done it! You say that it “basically seems just fine”, but… I don’t expect that it would actually seem “just fine” if I (or anyone else) were to actually do it.
Of course, I could be wrong. What are three examples of posts that others have written, that boil down simply to “other post X, written by person Y, is wrong”, and which have gotten a good reception? Perhaps if we did a case study or three, we’d gain some more insight into this thing.
(As for the “specific complaint”—there I just don’t know what you mean. Feel free to elaborate, if you like.)
Slight clarification – I think I worded the previous comment confusingly. I meant to say, if the typical LessWrong user wrote a single comment in reply to a post saying “this seems wrong”, I would expect that to basically be fine.
I only recommend the “create a whole new post” thing when an author specifically asks you to stop commenting.
(In some cases I think creating a whole new post would actually be just fine, based on how I’ve seen, say, Eliezer, Robin Hanson, Zvi, Ben Hoffman and Sarah Constantin respond to each other in longform on occasion. In other cases creating a whole new post might go over less well, and/or might be a bit of an experiment rather than a tried-and-true-solution, but I think it’s the correct experiment to try)
Also want to be a clear—if authors are banning or asking lots of users to avoid criticism, I do think the author should take something of a social hit as “a person who can’t accept any criticism”. But I nonetheless think it’s still a better metanorm for established authors to have control over their post’s discussion area.
[The LessWrong team is currently trying to develop a much clearer understanding of what good moderation policies are, which might result in some of my opinions changing over the next few weeks, this is just a quick summary of what I currently believe]
Quite. A suggestion, then, if I may: display “how many people has this person banned from their posts” (with, upon a click or mouseover or some such, the full list of users available, who have been thus banned) prominently, when viewing a person’s post (somewhere near the post’s author line, perhaps). This way, if I open a post by one Carol, say, I can see at once that she’s banned 12 people from her posts; I take note of this (as that is unusually many); I then click/mouseover/etc., and see either that all the banned accounts are known trolls and curmudgeons (and conclude that Carol is a sensible person with a low tolerance for low-grade nonsense), or that all the banned accounts are people I judge to be reasonable and polite (and conclude that Carol is a prima donna with a low tolerance for having her ideas challenged).
Something in that space seems basically reasonable. Note that I haven’t prioritized cleaning up (and then improving visibility for) the moderation log in part because the list of users who have ever banned users is actually just extremely short, and meanwhile there’s a lot of other site features that seem higher priority.
I have been revisiting it recently and think it’d be a good thing to include in the nearish future (esp. if I am prioritizing other features that’d make archipelago-norms more likely to actually get used), but for the immediate future I actually think just saying to the few people who’ve expressed concerns ‘yo, when you look at the moderation log almost nobody has used it’ is the right call given limited dev time.
Ah, I see. Well, yes. But then, that’s also what I was saying: this sort of thing is generally fine as a comment, but as a post…
I entirely understand your intention here, but consider: this would be even worse, “optics”-wise! “So,” thinks the reader, “this guy was so annoying, with his contrarian objections, that the victim of his nitpicking actually asked him to stop commenting; but he can’t let it go, so he wrote a whole post about it?!” And of course this is an uncharitable perspective, and one which isn’t consistent with “good truth-seeking norms”, etc. But… do you doubt that this is the sort of impression that will, if involuntarily, be formed in the minds of the commentariat?
I’m fairly uncertain here. But I don’t currently share the intuition.
Note that the order of events I’m suggesting is:
1. Author posts.
2. Commenter says “this seems wrong / bad”. Disagreement ensues
3. Author says “this is annoying enough that I’d prefer you not to comment on my posts anymore.” [Hopefully, although not necessarily, the author does this knowing that they are basically opting into you now being encouraged by LessWrong moderators to post your criticism elsewhere if you think it’s important. This might not currently be communicated that well but I think it should be]
4. Then you go and write a post titled ‘My thoughts on X’ or ‘Alternative Conversation about X’ or whatever, that says ‘the author seems wrong / bad.’
By that point, sure it might be annoying, but it’s presumably an improvement from the author’s take. (I know that if I wanted to write a post about some high level Weird Introspection Stuff that took a bunch of Weird Introspection Paradigm stuff for granted, I’d personally probably be annoyed if you made the discussion about whether the Weird Introspection Paradigm was even any good, and much less annoyed if you wrote another post saying so.
I might be typical minding, but two important bits from my perspective are ‘getting to have the conversation that I actually wanted to have’, and ‘not being forced to provide my own platform for someone else who I don’t think is arguing in good faith’
Addenda: my Strategies of Personal Growth post is also particularly downstream of CFAR. (I realize that much of it is something you can elsewhere. My perspective is that the main product CFAR provides is a culture that makes it easier to orient this sort of thing, and stick with it. CFAR iterates on “what combination of techniques can you present to a person in 4 days that best help jump-start them into that culture?”, and they chose that feedback-loop-cycle after exploring others and finding them less effective)
One salient thing from the Strategies of Personal Growth perspective (which I attribute to exploration by CFAR researchers) is that many of the biggest improvements you can gain come from healing and removing psychological blockers.
Separately from my other comment, I will say—
This happens to be phrased such that it could be literally true, but the implication—that the CFAR paradigm is in fact useful (to some people), and that it could potentially be useful (to some people) but the fact of its usefulness could be illegible to me—cannot be true. (Or, to be more precisely, it cannot be true simultaneously with the claim being true that “the CFAR paradigm” constitutes CFAR succeeding at finding [at least part of] what they were looking for. Is this claim being made? It seems like it is—if not, that should be made clear!)
The reason is simple: the kind of thing that CFAR (claimed to have) set out to look for, is the kind of thing that should be quite legible even to very skeptical third parties. “We found what we were looking for, but you just can’t tell that we did” is manifestly an evasion.
What is your current model of what CFAR “claimed to have set out to look for”? I don’t actually know much of an explicit statement of what CFAR was trying to look for, besides the basic concepts of “applied rationality”.
Correction: CFAR was started in 2012 (though I believe some of the founders ran rationality camps the previous summer, in 2011), so it’s been 7 (or 8) years, not 10.
Less Wrong, however, was launched in 2009, and that is what I was referring to (namely, Eliezer’s posts about the community and Bayesian dojos and so forth).
This is certainly theoretically possible, but I’m very suspicious of it on reversal test grounds: if additional prior reading is bad, then why isn’t less prior reading even better? Should aspiring rationalists not read the Sequences for fear of an incomplete understanding spoiling themselves for some future $3,900 CfAR workshop? (And is it bad that I know about the reversal test without having attended a CfAR workshop?)
I feel the same way about schoolteachers who discourage their students from studying textbooks on their own (because they “should” be learning that material by enrolling in the appropriate school course). Yes, when trying to learn from a book, there is some risk of making mistakes that you wouldn’t make with the help of a sufficiently attentive personal tutor (which, realistically, you’re not going to get from attending lecture classes in school anyway). But given the alternative of placing my intellectual trajectory at the mercy of an institution that has no particular reason to care about my welfare, I think I’ll take my chances.
Note that I’m specifically reacting to the suggestion that people not read things for their own alleged benefit. If the handbook had just said, “Fair warning, this isn’t a substitute for the workshop because there’s a lot of stuff we don’t know how to teach in writing,” then fine; that seems probably true. What I’m skeptical of is hypothesized non-monotonicity whereby additional lower-quality study allegedly damages later higher-quality study. First, because I just don’t think it’s true on the merits: I falsifiably predict that, e.g., math students who read the course textbook on their own beforehand will do much better in the course than controls who haven’t. (Although the pre-readers might annoy teachers whose jobs are easier if everyone in the class is obedient and equally ignorant.) And second, because the general cognitive strategy of waiting for the designated teacher to spoonfeed you the “correct” version carries massive opportunity costs when iterated (even if spoonfeeding is generally higher-quality than autodidactism, and could be much higher-quality in some specific cases).
You use math as an example, but that’s highly focused on System 2 learning. That suggests that you have false assumptions about what CFAR is trying to teach.
There are many subjects where written instructions are much less valuable than instruction that includes direct practice: circling, karate, meditation, dancing, etc. Most of those analogies are fairly imperfect, and some have partially useful written instructions (in the case of meditation, the written version might have lagged in-person instruction by many centuries). Circling is the example that I’d consider most apt, but it won’t mean much to people who haven’t taken a good circling workshop.
A different analogy, which more emphasizes the costs of false assumptions: people often imagine that economics teaches something like how to run a good business or how to predict the stock market, because there isn’t any slot in their worldview for what a good economics course actually teaches. There are plenty of mediocre executive summaries of economics, which fail to convey to most people that economics requires a pervasive worldview shift (integrating utilitarianism, empiricism about preferences, and some counterintuitive empirical patterns).
The CFAR handbook is more like the syllabus for an economics course than it is like an economics textbook, and a syllabus is useless (possibly harmful) for teaching economics to people who have bad assumptions about what kind of questions economics answers. (This analogy is imperfect because economics textbooks have been written, unlike a CFAR textbook.)
Maybe CFAR is making a mistake, but it appears that the people who seem most confident about that usually seem to be confused about what it is that CFAR is trying to teach.
Reading the sequences, or reading about the reversal test, are unlikely to have much relevance to what CFAR teaches. Just be careful not to imagine that they’re good examples of what CFAR is about.
Sometimes, we don’t know how to teach a subject in writing because the subject matter is inherently about action (rather than concepts, analysis, explanation, prediction, numbers, words, etc.).
But sometimes, we don’t know how to teach a subject in writing because there is, in fact, nothing (or, at best, nothing much) to be taught. Sometimes, a subject is actually empty (or mostly empty) of content.
In the latter case, attempting to write it down reveals this (and opens the alleged “content” to criticism)—whereas in person, the charisma of the instructors, the social pressure of being in a group of others who are there to receive the instruction, possibly the various biases associated with having made some costly sacrifice (time, money, etc.) to be there, possibly the various biases associated with the status dynamics at play (e.g. if the instructors are respected, or at least if those around you act as if they are), all serve to mask the fundamental emptiness of what is being “taught”.
I leave it to the reader to discern which of the given examples fall into which category. I will only note that while the subjects found in the former category are often difficult to teach, nevertheless one’s mastery of them, and their effectiveness, is usually quite easy to verify—because action can be demonstrated.
Meditation is action, in some important sense, and mostly can’t be demonstrated.
It is hard to reliably distinguish between the results of peer pressure and actual learning. I think CFAR’s best reply to this has been it’s refund policy: last I knew they offered full refunds to anyone who requested it within one year (although I can’t find any online mention of their current policy).
Everything is “action” in “some sense”. (Whether that sense is “important”, in any given case, is a matter of perspective.)
As far as I am concerned—for the purposes of this topic—if it can’t be demonstrated, it ain’t action.
I submit to you that if this is true of any given case, then that is an excellent signal that no actual learning has taken place. (And the more true it is—the harder it is to distinguish between actual learning and the results of various biases, social pressure included—the stronger the signal is.)
Yes, I agree: for these subjects, the “there’s a lot of stuff we don’t know how to teach in writing” disclaimer I suggested in the grandparent would be a big understatement.
Useless, I can believe. (The extreme limiting case of “there’s a lot of stuff we don’t know how to teach in this format” is “there is literally nothing we know how to teach in this format.”) But harmful? How? Won’t the unexpected syllabus section titles at least disabuse them of their bad assumptions?
Really? The tagline on the website says, “Developing clear thinking for the sake of humanity’s future.” I guess I’m having trouble imagining a developing-clear-thinking-for-the-sake-of-humanity’s-future curriculum for which the things we write about on this website would be irrelevant. The “comfort zone expansion” exercises I’ve heard about would qualify, but Sequences-knowledge seems totally relevant to something like, say, double crux.
(It’s actually pretty weird/surprising that I’ve never personally been to a CfAR workshop! I think I’ve been assuming that my entire social world has already been so anchored on the so-called “rationalist” community for so long, that the workshop proper would be superfluous.)
The idea that CFAR would be superfluous is fairly close to the kind of harm that CFAR worries about. (You might have been right to believe that it would have been superfluous in 2012, but CFAR has changed since then in ways that it hasn’t managed to make very legible.)
I think meditation provides the best example for illustrating the harm. It’s fairly easy to confuse simple meditation instructions (e.g. focus on your breath, sit still with a straight spine) with the most important features of meditation. It’s fairly easy to underestimate the additional goals of meditation, because they’re hard to observe and don’t fit well with more widely accepted worldviews.
My experience suggests that getting value out of meditation is heavily dependent on a feeling (mostly at a system 1 level) that I’m trying something new, and there were times when I wasn’t able to learn from meditation, because I mistakenly thought that focusing on my breath was a much more central part of meditation than it actually is.
The times when I got more value out of meditation were times when I tried new variations on the instructions, or new environments (e.g. on a meditation retreat). I can’t see any signs that the new instructions or new environment were inherently better at teaching meditation. It seems to have been mostly that any source of novelty about the meditation makes me more alert to learning from it.
My understanding is that CFAR is largely concerned that participants will mistakenly believe that they’ve already learned something that CFAR is teaching, and that will sometimes be half-true—participants may know it at a system 2 level, when CFAR is trying to teach other parts of their minds that still reject it.
I think I experienced that a bit, due to having experience with half-baked versions of early CFAR before I took a well-designed version of their workshop. E.g. different parts of my mind have different attitudes to acknowledging my actual motivations when they’re less virtuous than the motivations that my system 2 endorses. I understood that pretty well at some level before CFAR existed, yet there are still important parts of my mind that cling to self-deceptive beliefs about my motives.
CFAR likely can’t teach a class that’s explicitly aimed at that without having lots of participants feel defensive about their motives, in a way that makes them less open to learning. So they approach it via instruction that is partly focused on teaching other things that look more mundane and practical. Those other things often felt familiar enough to me that I reacted by saying: I’ll relax now and conserve my mental energy for some future part of the curriculum that’s more novel. That might have led me to do the equivalent of what I did when I was meditating the same way repeatedly without learning anything new. How can I tell whether that caused me to miss something important?
A key problem is moving from knowing about a technique to action. When you know 20 techniques and use no one of them it’s harder to get you to actually use the 21st technique that you are taught then if you start out with less techniques in your head and have no established pattern of not doing any of the exercises that you were taught.
There’s less that needs unlearning if you haven’t been exposed to material beforehand.
I would still err on the side of being more public with information, but I do understand that there is a tradeoff.
I can offer an explanation that might fit. Rationalists tend toward expertise mode thinking (expert from the torbert action logic framework). Behaviour like reading the book is in line with the expert behaviour.
Cfar techniques and related in-person methods are not always about being the expert, they are about doing the best thing. Being a better expert is not always the same as being the better munchkin, the better person or the person who can step out of their knowledge beliefs.
In theory, the expert thing is the best thing. In theory there’s no difference between theory and practice, in practice, there’s a big difference between theory and practice.
Having said that, I’ve never done cfar, and I teach workshops monthly in Sydney and I think they are wrong to discourage sharing of their resources. As the same time I accept the idea of intellectual property being protected even if that’s not the case they are claiming.
(I’m in the process of writing up my resources into a collection)
I suspect that this is the real reason. Although if the much vaster sequences by Yudkowsky are freely available I don’t see it as a good justification for not making the CFAR handbook available.
It’s pretty uncharitable of you to just accuse CfAR of lying like that! If the actual reason were “Many of the explanations here are intentionally approximate or incomplete because we predict that this handbook will be leaked and we don’t want to undercut our core product,” then the handbook would have just said that.
Wait, are you invoking the principle of charity as an epistemic axiom (“assume people don’t lie”)? Why would that be truth-aligned at all?
If you didn’t mean to invoke the principle of charity, why not just say it’s likely to be incorrect based on priors, CFAR’s reputation, etc, instead of using the word “uncharitable” as an insult?
You caught me—introspecting, I think the grandparent was written in a spirit of semi-deliberate irony. (“Semi” because it just felt like the “right” thing to say there; I don’t think I put a lot of effort into modeling how various readers would interpret it.)
Roland is speculating that the real reason for intentionally incomplete explanations in the handbook is different from the stated reason, and I offered a particularly blunt phrasing (“we don’t want to undercut our core product”) of the hypothesized true reason, and suggested that that’s what the handbook would have said in that case. I think I anticipated that a lot of readers would find my proposal intuitively preposterous: “everyone knows” that no one would matter-of-factly report such a self-interested rationale (especially when writing on behalf of an organization, rather than admitting a vice among friends). That’s why the earlier scenes in the 2009 film The Invention of Lying, or your post “Act of Charity”, are (typically) experienced as absurdist comedy rather than an inspiring and heartwarming portrayal of a more truthful world.
But it shouldn’t be absurd for the stated reason and the real reason to be the same! Particularly for an organization like CfAR which is specifically about advancing the art of rationality. And, I don’t know—I think sometimes I talk in a way that makes me seem more politically naïve than I actually am, because I feel as if the “naïve” attitude is in some way normative? (“You really think someone would do that? Just go on the internet and tell lies?”) Arguably this is somewhat ironic (being deceptive about your ability to detect deception is probably not actually the same thing as honesty), but I haven’t heretofore analyzed this behavioral pattern of mine in enough detail to potentially decide to stop doing it??
I think another factor might be that I feel guilty about being “mean” to CfAR in the great-great-great grandparent comment? (CfAR isn’t a person and doesn’t have feelings, but my friend who works there is and does.) Such that maybe the emotional need to signal that I’m still fundamentally loyal to the “mainstream rationality” tribe (despite the underlying background situation where I’ve been collaborating with you and Ben and Michael to discuss what you see as fatal deficits of integrity in “the community” as presently organized) interacted with my preëxisting tendency towards semi-performative naiveté in a way that resulted in me writing a bad blog comment? It’s a good thing you were here to hold me to account for it!
I thought your comment was fine and the irony was obvious, but this kind of misunderstanding can be easily avoided by making the straightforward reading more boring, like so:
Yeah, I would have expected Jessica to get it, except that I suspect she’s also executing a strategy of habitual Socratic irony (but without my additional innovation of immediately backing down and unpacking the intent when challenged), which doesn’t work when both sides of a conversation are doing it.
I actually didn’t get it. I was confused but I didn’t consciously generate the hypothesis that it was ironic.
I think I don’t share the background assumption that it is overwhelmingly obvious that CFAR wouldn’t tell the truth about this in their handbook. I also reflectively endorse a policy of calling out things that could easily be mistaken for sincere (though not obvious sarcasm), in order to ensure common knowledge.
Quick note for your model of how people interpret various kinds of writing, my initial read of your comment was to put a 60% probability on “Zack is currently undergoing a pendulum swing in the direction away from calling people out on lying, and overcompensating.” (which was wrong and/or overconfident on my part)
I wasn’t, I rather suspect them of being biased.
We’ve all heard this sort of thing many times, of course. The best response is probably Schopenhauer’s:
You are, I assume, referring to the ideas of this person? He appears to be some variety of management consultant. Is there any reason to take this “action logic” of his seriously? It seems to be yet another among the many, many self-help / management consulting / etc. “frameworks” or “systems” etc. Do any of his ideas have any empirical verification, or… well, anything, really?
That’s up to you. I’ve got a lot of value from the structure he outlines. It’s a lot more reasoned than some of the other mysterious odd things I read.
If there is something wrong with the theory and the way it maps to the practice, is it better to read more theory or do more practice and make new theories? I would suggest it depends on the person and what they have found to work in the past. And also with an awareness to the loops of bad habits—“sharpen the saw” type problems. Sometimes it’s more valuable to stop sharpening the saw, and start cutting down the tree. (rationality frame of thinking loves to sharpen more and cut less)