Thank you for your gracious reply. I interpret a couple of overarching themes in which I would like to frame my own: the first is the ‘performance issue’ (i.e. ‘How good is double crux at resolving disagreement/getting closer to the truth’); the second the ‘pedagogical issue’ (i.e. ‘how good is double crux at the second order task of getting people better at resolving disagreement/getting closer to the truth’). I now better understand you take the main support from double crux to draw upon the latter issue, but I’d also like to press on some topics about the former on which I believe we disagree.
How well does double crux perform?
Your first two diagrams precisely capture the distinction I have in mind (I regret not having thought to draw my own earlier). If I read the surrounding text right (I’m afraid not to know what ‘fompmott’ means, and google didn’t help me), you suggest that even if better cognisers find their considerations form a denser web like the second diagram, double-crux amenable ‘sparser’ webs are still common in practice, perhaps due to various non-rational considerations. You also add:
I note that, if correct, this theory would indicate that e.g. your average LessWronger would find less value in double crux than your average CFAR participant (who shares a lot in common with a LessWronger but in expectation is less rigorous and careful about their epistemics). This being because LessWrongers try very deliberately to form belief webs like the first [I think second? - T] image, and when they have a belief web like the third image they try to make that belief feel to themselves as unbalanced and vulnerable as it actually is. Ergo, LessWrongers would find the “Surprise! You had an unjustified belief!” thing happening less often and less unexpectedly.
This note mirrors a further thought I had (c.f. Ozymandias’s helpful remark in a child about sequence versus cluster thinking). Yet I fear this poses a further worry for the ‘performance issue’ of double crux, as it implies that the existence of cruxes (or double cruxes) may be indicative of pathological epistemic practices. A crux implies something like the following:
You hold some belief B you find important (at least, important enough you think it is worth your time to discuss).
Your credence in B depends closely on some consideration C.
Your credence in C is non-resilient (at least sufficiently non-resilient you would not be surprised to change your mind on it after some not-unduly-long discussion with a reasonable interlocutor).*
* What about cases where one has a resilient credence in C? Then the subsequent worries do not apply. However, I suspect these cases often correspond to “we tried to double crux and we found we couldn’t make progress on resolving our disagreement about theories of truth/normative ethics/some other foundational issue”.
It roughly follows from this you should have low resilience in your credence of B. As you note, this is vulnerable, and knowing one had non-resilient credences in important Bs is to be avoided.
As a tool of diagnosis, double crux might be handy (i.e. “This seems to be a crux for me, yet cruxes aren’t common among elite cognisers—I should probably go check whether they agree this is the crux of this particular matter, and if not maybe see what else they think bears upon B besides C”). Yet (at least per the original exposition) it seems to be more a tool for subsequent ‘treatment’. Doing so could make things worse, not better.
If X and Y find they differ on some crux, but also understand that superior cognisers tend not to have this crux, and distribute support across a variety of considerations, it seems a better idea for them to explore other candidate considerations rather than trying to resolve their disagreement re. C. If they instead do the double-cruxy thing and try and converge on C, they may be led up the epistemic garden path. They may agree with one another on C (thus B), and thus increase their resilience of C (thus B), yet they also confirm a mistaken web of belief around B which wrongly accords too much weight to C. If (as I suggest) at least half the battle on having good ‘all things considered’ attitudes to recondite matters comprises getting the right weights for relevant considerations on the matter, double crux may celebrate them converging further away from the truth. (I take this idea to be expressed in kernel in Ozymandias’s worry of double crux displacing more-expectedly-accurate cluster thinking with less-expectedly-accurate sequence thinking).
How good is double crux at ‘levelling people up at rationality’
The substantial independence of the ‘performance issue’ from the pedagogical issue’
In the same way practising scales may not be the best music, but make one better at playing music, double crux may not be the best discussion technique, but make one better at discussions. This seems fairly independent of its ‘object level performance’ (although I guess if the worry above is on the right track, we would be very surprised if a technique that on the object level leads beliefs to track truth more poorly nonetheless has a salutatory second-order effect).
Thus comparisons to practices of elite philosophers (even if they differ) are inapposite—especially, as I understand from one of them, the sort of superior pattern I observe occurs only at a far right tail even among philosophers (i.e. ‘world-class’ as you write, rather then ‘good’, as I write in the OP). It is obviously a great boon if I could get some fraction more like someone like Askell or Shulman without either their profound ability or the time they have invested in these practices.
On demurring the ‘double crux challenge’
I regret I don’t think it would be hugely valuable to ‘try double crux’ with an instructor in terms of resolving this disagreement. One consideration (on which more later) is that conditional on me not being persuaded by a large group of people who self-report double crux is great, I shouldn’t change my mind (for symmetry reasons) if this number increases by one other person, or it increases by including me. Another is that the expected yield may not be great, at least in one direction: although I hope I am not ‘hostile’ to double crux, it seems one wouldn’t be surprised if it didn’t work with me, even if its generally laudable.
Yet I hope I am not quite as recalcitrant as ‘I would not believe until I felt the stigmata with my own hands’. Apart from a more publicly legible case (infra), I’m a bit surprised at the lack of ‘public successes’ of double cruxing (although this may confuse performance versus pedagogy). In addition to Constantin, Raemon points to their own example with gjm. Maybe I’m only seeing what I want to, but I get a similar impression. They exhibit a variety of laudable epistemic practices, but I don’t see a crux or double crux (what they call ‘cruxes’ seem to be more considerations they take to be important).
The methods of rational self-evaluation
You note a head-to-head comparison between double crux and an approximate sham-control seemed to favour double crux. This looks like interesting data, and it seems a pity it emerges in the depths of a comment thread (ditto the ‘large n of successes’) rather than being written up and presented—it seems unfortunate that the last ‘public evaluation report’ is about 2 years old. I would generally urge trying to produce more ‘public evidence’ rather than the more private “we’ve generally seen this work great (and a large fraction of our alums agree!)”
I recognise that “Provide more evidence to satisfy outside sceptics” should not be high on CFAR’s priority list. Yet I think it is instrumental to other important goals instead. Chiefly: “Does what we are doing actually work?”
You noted in your initial reply undercutting considerations to the ‘we have a large n of successes’, yet you framed this in way that these would often need to amount to a claim of epistemic malice (i.e. ‘either CFAR is lying or participants are being socially pressured into reporting a falsehood’). I don’t work at a rationality institute or specialise in rationality, but on reflection I find this somewhat astonishing. My impression of cognitive biases were that they were much more insidious, that falling prey to them was the rule rather than the exception, and that sincere good faith was not adequate protection (is this not, in some sense, what CFAR casus belli is predicated upon?)
Although covered en passant, let me explicitly (although non-exhaustively) list things which might bias more private evidence of the type CFAR often cites:
CFAR staff (collectively) are often responsible for developing the interventions they hope will improve rationality. One may expect them to be invested in them, and more eager to see that they work than see they don’t (c.f. why we prefer double-blinding over single-blinding).
Other goods CFAR enjoys (i.e. revenue/funding, social capital) seem to go up the better the results of their training. Thus CFAR staff have a variety of incentives pushing them to over-report how good their ‘product’ is (c.f. why conflicts of interest are bad, the general worries about pharma-funded drug trials).
Many CFAR participants have to spend quite a lot of money (i.e. fees and travel) to attend a workshop. They may fear looking silly if it turns out after all this it didn’t do anything, and so incentivised to assert it was much more helpful than it actually was (c.f. choice supportive bias).
There are other aspects of CFAR workshops that participants may enjoy independent of the hoped-for improvement of their rationality (e.g. hanging around interesting people like them, personable and entertaining instructors, romantic entanglements). This extraneous benefits may nonetheless bias upwards their estimate of how effective CFAR workshops are at improving their rationality (c.f. halo effect).
I am sure there are quite a few more. One need not look that hard to find lots of promising studies supporting a given intervention undermined by any one of these.
The reference class of interventions with “a large corpus of (mainly self-reported) evidence of benefit, but susceptible to these limitations” is dismal. It includes many branches of complementary medicine. It includes social programs (e.g. ‘scared straight’) that we now know to be extremely harmful. It includes a large number of ineffective global poverty interventions. Beyond cautionary tales, I aver these approximate the modal member of the class: when the data is so subjective, and the limitations this severe, one should expect the thing in question doesn’t actually work after all.
I don’t think this expectation changes when we condition on the further rider “And the practicioners really only care about the truth re. whether the intervention works or not.” What I worry about going on under the hood is a stronger (and by my lights poorly substantiated) claim of rationalist exceptionalism: “Sure, although cognitive biases plague entire fields of science and can upend decades of results, and we’re appropriately quick to point out risk of bias of work done by outsiders, we can be confident that as we call ourselves rationalists/we teach rationality/we read the sequences/etc. we are akin to Penelope refusing her army of suitors—essentially incorruptible. So when we do similarly bias-susceptible sorts of things, we should give one another a pass.”
I accept ‘gold standard RCTs’ are infeasible (very pricey, and how well can one really do ‘sham CFAR’?) yet I aver there is quite a large gap between this ideal of evidence and the actuality (i.e. evidence kept in house, and which emerges via reference in response to criticism) which could be bridged by doing more write-ups, looking for harder metrics that put one more reliably in touch with reality, and so on. I find it surprisingly incongruent that the sort of common cautions about cognitive biases—indeed, common cautions that seem predicates for CFAR’s value proposition (e.g. “Good faith is not enough”, “Knowing about the existence of biases does not make one immune to them”, Feynmann’s dictum about ‘you are the easiest person to fool’), are not reflected in its approach to self-evaluation.
If nothing else, opening up more of CFAR’s rationale, evidence, etc. to outside review may allow more benefits of outside critique. Insofar as it is the case you found this exchange valuable, one may anticipate greater benefit from further interaction with higher-quality sceptics.
Thank you for your gracious reply. I interpret a couple of overarching themes in which I would like to frame my own: the first is the ‘performance issue’ (i.e. ‘How good is double crux at resolving disagreement/getting closer to the truth’); the second the ‘pedagogical issue’ (i.e. ‘how good is double crux at the second order task of getting people better at resolving disagreement/getting closer to the truth’). I now better understand you take the main support from double crux to draw upon the latter issue, but I’d also like to press on some topics about the former on which I believe we disagree.
How well does double crux perform?
Your first two diagrams precisely capture the distinction I have in mind (I regret not having thought to draw my own earlier). If I read the surrounding text right (I’m afraid not to know what ‘fompmott’ means, and google didn’t help me), you suggest that even if better cognisers find their considerations form a denser web like the second diagram, double-crux amenable ‘sparser’ webs are still common in practice, perhaps due to various non-rational considerations. You also add:
This note mirrors a further thought I had (c.f. Ozymandias’s helpful remark in a child about sequence versus cluster thinking). Yet I fear this poses a further worry for the ‘performance issue’ of double crux, as it implies that the existence of cruxes (or double cruxes) may be indicative of pathological epistemic practices. A crux implies something like the following:
You hold some belief B you find important (at least, important enough you think it is worth your time to discuss).
Your credence in B depends closely on some consideration C.
Your credence in C is non-resilient (at least sufficiently non-resilient you would not be surprised to change your mind on it after some not-unduly-long discussion with a reasonable interlocutor).*
* What about cases where one has a resilient credence in C? Then the subsequent worries do not apply. However, I suspect these cases often correspond to “we tried to double crux and we found we couldn’t make progress on resolving our disagreement about theories of truth/normative ethics/some other foundational issue”.
It roughly follows from this you should have low resilience in your credence of B. As you note, this is vulnerable, and knowing one had non-resilient credences in important Bs is to be avoided.
As a tool of diagnosis, double crux might be handy (i.e. “This seems to be a crux for me, yet cruxes aren’t common among elite cognisers—I should probably go check whether they agree this is the crux of this particular matter, and if not maybe see what else they think bears upon B besides C”). Yet (at least per the original exposition) it seems to be more a tool for subsequent ‘treatment’. Doing so could make things worse, not better.
If X and Y find they differ on some crux, but also understand that superior cognisers tend not to have this crux, and distribute support across a variety of considerations, it seems a better idea for them to explore other candidate considerations rather than trying to resolve their disagreement re. C. If they instead do the double-cruxy thing and try and converge on C, they may be led up the epistemic garden path. They may agree with one another on C (thus B), and thus increase their resilience of C (thus B), yet they also confirm a mistaken web of belief around B which wrongly accords too much weight to C. If (as I suggest) at least half the battle on having good ‘all things considered’ attitudes to recondite matters comprises getting the right weights for relevant considerations on the matter, double crux may celebrate them converging further away from the truth. (I take this idea to be expressed in kernel in Ozymandias’s worry of double crux displacing more-expectedly-accurate cluster thinking with less-expectedly-accurate sequence thinking).
How good is double crux at ‘levelling people up at rationality’
The substantial independence of the ‘performance issue’ from the pedagogical issue’
In the same way practising scales may not be the best music, but make one better at playing music, double crux may not be the best discussion technique, but make one better at discussions. This seems fairly independent of its ‘object level performance’ (although I guess if the worry above is on the right track, we would be very surprised if a technique that on the object level leads beliefs to track truth more poorly nonetheless has a salutatory second-order effect).
Thus comparisons to practices of elite philosophers (even if they differ) are inapposite—especially, as I understand from one of them, the sort of superior pattern I observe occurs only at a far right tail even among philosophers (i.e. ‘world-class’ as you write, rather then ‘good’, as I write in the OP). It is obviously a great boon if I could get some fraction more like someone like Askell or Shulman without either their profound ability or the time they have invested in these practices.
On demurring the ‘double crux challenge’
I regret I don’t think it would be hugely valuable to ‘try double crux’ with an instructor in terms of resolving this disagreement. One consideration (on which more later) is that conditional on me not being persuaded by a large group of people who self-report double crux is great, I shouldn’t change my mind (for symmetry reasons) if this number increases by one other person, or it increases by including me. Another is that the expected yield may not be great, at least in one direction: although I hope I am not ‘hostile’ to double crux, it seems one wouldn’t be surprised if it didn’t work with me, even if its generally laudable.
Yet I hope I am not quite as recalcitrant as ‘I would not believe until I felt the stigmata with my own hands’. Apart from a more publicly legible case (infra), I’m a bit surprised at the lack of ‘public successes’ of double cruxing (although this may confuse performance versus pedagogy). In addition to Constantin, Raemon points to their own example with gjm. Maybe I’m only seeing what I want to, but I get a similar impression. They exhibit a variety of laudable epistemic practices, but I don’t see a crux or double crux (what they call ‘cruxes’ seem to be more considerations they take to be important).
The methods of rational self-evaluation
You note a head-to-head comparison between double crux and an approximate sham-control seemed to favour double crux. This looks like interesting data, and it seems a pity it emerges in the depths of a comment thread (ditto the ‘large n of successes’) rather than being written up and presented—it seems unfortunate that the last ‘public evaluation report’ is about 2 years old. I would generally urge trying to produce more ‘public evidence’ rather than the more private “we’ve generally seen this work great (and a large fraction of our alums agree!)”
I recognise that “Provide more evidence to satisfy outside sceptics” should not be high on CFAR’s priority list. Yet I think it is instrumental to other important goals instead. Chiefly: “Does what we are doing actually work?”
You noted in your initial reply undercutting considerations to the ‘we have a large n of successes’, yet you framed this in way that these would often need to amount to a claim of epistemic malice (i.e. ‘either CFAR is lying or participants are being socially pressured into reporting a falsehood’). I don’t work at a rationality institute or specialise in rationality, but on reflection I find this somewhat astonishing. My impression of cognitive biases were that they were much more insidious, that falling prey to them was the rule rather than the exception, and that sincere good faith was not adequate protection (is this not, in some sense, what CFAR casus belli is predicated upon?)
Although covered en passant, let me explicitly (although non-exhaustively) list things which might bias more private evidence of the type CFAR often cites:
CFAR staff (collectively) are often responsible for developing the interventions they hope will improve rationality. One may expect them to be invested in them, and more eager to see that they work than see they don’t (c.f. why we prefer double-blinding over single-blinding).
Other goods CFAR enjoys (i.e. revenue/funding, social capital) seem to go up the better the results of their training. Thus CFAR staff have a variety of incentives pushing them to over-report how good their ‘product’ is (c.f. why conflicts of interest are bad, the general worries about pharma-funded drug trials).
Many CFAR participants have to spend quite a lot of money (i.e. fees and travel) to attend a workshop. They may fear looking silly if it turns out after all this it didn’t do anything, and so incentivised to assert it was much more helpful than it actually was (c.f. choice supportive bias).
There are other aspects of CFAR workshops that participants may enjoy independent of the hoped-for improvement of their rationality (e.g. hanging around interesting people like them, personable and entertaining instructors, romantic entanglements). This extraneous benefits may nonetheless bias upwards their estimate of how effective CFAR workshops are at improving their rationality (c.f. halo effect).
I am sure there are quite a few more. One need not look that hard to find lots of promising studies supporting a given intervention undermined by any one of these.
The reference class of interventions with “a large corpus of (mainly self-reported) evidence of benefit, but susceptible to these limitations” is dismal. It includes many branches of complementary medicine. It includes social programs (e.g. ‘scared straight’) that we now know to be extremely harmful. It includes a large number of ineffective global poverty interventions. Beyond cautionary tales, I aver these approximate the modal member of the class: when the data is so subjective, and the limitations this severe, one should expect the thing in question doesn’t actually work after all.
I don’t think this expectation changes when we condition on the further rider “And the practicioners really only care about the truth re. whether the intervention works or not.” What I worry about going on under the hood is a stronger (and by my lights poorly substantiated) claim of rationalist exceptionalism: “Sure, although cognitive biases plague entire fields of science and can upend decades of results, and we’re appropriately quick to point out risk of bias of work done by outsiders, we can be confident that as we call ourselves rationalists/we teach rationality/we read the sequences/etc. we are akin to Penelope refusing her army of suitors—essentially incorruptible. So when we do similarly bias-susceptible sorts of things, we should give one another a pass.”
I accept ‘gold standard RCTs’ are infeasible (very pricey, and how well can one really do ‘sham CFAR’?) yet I aver there is quite a large gap between this ideal of evidence and the actuality (i.e. evidence kept in house, and which emerges via reference in response to criticism) which could be bridged by doing more write-ups, looking for harder metrics that put one more reliably in touch with reality, and so on. I find it surprisingly incongruent that the sort of common cautions about cognitive biases—indeed, common cautions that seem predicates for CFAR’s value proposition (e.g. “Good faith is not enough”, “Knowing about the existence of biases does not make one immune to them”, Feynmann’s dictum about ‘you are the easiest person to fool’), are not reflected in its approach to self-evaluation.
If nothing else, opening up more of CFAR’s rationale, evidence, etc. to outside review may allow more benefits of outside critique. Insofar as it is the case you found this exchange valuable, one may anticipate greater benefit from further interaction with higher-quality sceptics.