Basically, this:
https://intelligence.org/2016/07/27/alignment-machine-learning/
It’s now MIRI’s official 2nd agenda, with the previous agenda going under the name “agent foundations”.
ProofOfLogic
Karma: 120
Could I get a couple of upvotes so that I could post links? I’d like to put some of the LW-relevant content from weird.solar here now that link posts are a thing.
If double crux felt like the Inevitable Correct Thing, what other things would we most likely believe about rationality in order for that to be the case?
I think this is a potentially useful question to ask for three reasons. One, it can be a way to install double crux as a mental habit—figure out ways of thinking which make it seem inevitable. Two, to the extent that we think double crux really is quite useful, but don’t know exactly why, that’s Bayesian evidence for whatever we come up with as potential justification for it. But, three, pinning down sufficient conditions for double crux can also help us see limitations in its applicability (IE, point toward necessary conditions).
I like the four preconditions Duncan listed:
Epistemic humility.
Good faith.
Confidence in the existence of objective truth.
Curiosity.
I made my list mostly by moving through the stages of the algorithm and trying to justify each one. Again, these are things which I think might or might not be true, but which I think would help motivate one step or another of the double crux algorithm if they were true.
A mindset of gathering information from people (that is, a mindset of honest curiosity) is a good way to combat certain biases (“arguments are soldiers” and all that).
Finding disagreements with others and finding out why they believe what they believe is a good way to gather information from them.
Most people (or perhaps, most people in the intended audience) are biased to argue for their own points as a kind of dominance game / intelligence signaling. This reduces their ability to learn things from each other.
Telling people not to do that, in some appropriate way, can actually improve the situation—perhaps by subverting the signaling game, making things other than winning arguments get you intelligence-signaling-points.
Illusion of transparency is a common problem, and operationalizing disagreements is a good way to fight against the illusion of transparency.
Or: Free-floating beliefs are a common problem, and operationalization is a good way to fight free-floating beliefs.
Or: operationalizing / discussing examples is a good way to make things easier to reason about, which people oftem don’t take enough advantage of.
Seeking your cruxes helps ensure your belief isn’t free-floating: if the belief is doing any work, it must make some predictions (which means it could potentially be falsified). So, in looking for your cruxes, you’re doing yourself a service, not just the other person.
Giving your cruxes to the other person helps them disprove your beliefs, which is a good thing: it means you’re providing them with the tools to help you learn. You have reason to think they know something you don’t. (Just be sure that your conditions for switching beliefs are good!)
Seeking out cruxes shows the other person that you believe things for reasons: your beliefs could be different if things were different, so they are entangled with reality.
In ordinary conversations, people try to have modus ponens without modus tollens: they want a belief that implies lots of things very strongly, but which is immune to attack. Bayesian evidence doesn’t work this way; a hypothesis which makes sharp prediction is necessarily sticking its neck out for the chopping block if the prediction turns out false. So, asking what would change your mind (asking for cruxes) is in a way equivalent to asking for implications of your belief. However, it’s doing it in a way which enforces the equivalence of implication and potential falsifier.
Asking for cruxes from them is a good way to avoid wasting time in a conversation. You don’t want to spend time explaining something only to find that it doesn’t change their mind on the issue at hand. (But, you have to believe that they give honest cruxes, and also that they are working to give you cruxes which could plausibly lead to progress rather than ones which will just be impossible to decide one way or the other.)
It’s good to focus on why you believe what you believe, and why they believe what they believe. The most productive conversations will tend to concentrate on the sources of beliefs rather than the after-the-fact reasoning, because this is often where the most evidence lies.
If you disagree with their crux but it isn’t a crux for you, then you may have info for them, but the discussion won’t be very informative for your belief. Also, the weight of the information you have is less likely to be large. Perhaps discuss it, but look for a double crux.
If they disagree with your crux but it isn’t a crux for them, then there may be information for you to extract from them, but you’re allowing the conversation to be bias toward cherry-picking disproof of your belief; perhaps discuss, but try to get them to stick their neck out more so that you’re mutually testing your beliefs.
Of all of this, my attempt to justify looking for a double crux rather than accepting single-person cruxes sticks out to me as especially weak. Also, I think a lot of the above points get something wrong with respect to good faith, but I’m not quite sure how to articulate my confusion on that.
Things “Meta” Can Mean
Yeah, I think the links thing is pretty important. Getting bloggers in the rationalist diaspora to move back to blogging on LW is something of an uphill battle, whereas them or others linking to their stuff is a downhill one.
“Decisions” as thoughts which lead to actions.
Disagreements can lead to bad real-world consequences for (sort of) two reasons:
1) At least one person is wrong and will make bad decisions which lead to bad consequences. 2) The argument itself will be costly (in terms of emotional cost, friendship, perhaps financial cost, etc).
In terms of #1, an unnoticed disagreement is even worse than an unsettled disagreement; so thinking about #1 motivates seeking out disagreements and viewing them as positive opportunities for intellectual progress.
In terms of #2, the attitude of treating disagreements as opportunities can also help, but only if both people are on board with that. I’m guessing that is what you’re pointing at?
My strategy in life is something like: seek out disagreements and treat them as delicious opportunities when in “intellectual mode”, but avoid disagreements and treat them as toxic when in “polite mode”. This heuristic isn’t always correct. I had to be explicitly told that many people often don’t like arguing even over intellectual things. Plus, because of #1, it’s sometimes especially important to bring up disagreements in practical matters (that don’t invoke “intellectual mode”) even at risk of a costly argument.
It seems like something like “double crux attitude” helps with #2 somewhat, though.
Some thoughts on double crux.
Seems there’s no way to edit the link, so I have to delete.
The Internal Lawyer
Stress Response, Growth Mindset, and Nonviolent Communication
Guilt vs Shame, Pride vs Joy?
Fear or fear? (A Meteuphoric post on distinguishing feelings from considered positions.)
Quick modeling: resolving disagreements.
I didn’t write the article, but I think “quick modeling” is referring to the previous post on that blog: simple rationality. It’s an idiosyncratic view, though; I think the “quick modeling” idea works just as well if you think of it as referring to Fermi-estimate style fast modeling instead (which isn’t that different in any case). The point is really just to have any model of the other person’s belief at all (for a broad notion of “model”), and then try to refine that. This is more flexible than the double crux algorithm.
From my experience with CFAR, I suspect CFAR staff would call the strategy described here a form of double crux anyway. The double crux algorithm is an ideal to shoot for, but the broader spirit of double crux is more like what this article is recommending I think.
Value Journaling
We also have to take into account priors in an individual situation. So, for example, maybe I have found that shamanistic scammers who lie about things related to dreams are pretty common. Then it would make sense for me to apply a special-case rule to disbelieve strange-sounding dream-related claims, even if I tend to believe similarly surprising claims in other contexts (where my priors point to people’s honesty).
But (if my reasoning is correct) the fact is that a real method can work before there is enough evidence to support it. My post attempts to bring to our attention that this will make it really hard to discover certain experiences assuming that they exist.
Discounting the evidence doesn’t actually make it any harder for us to discover those experiences. If we don’t want to lose out on such things, then we should try some practices which we assign low probability, to see which ones work. Assigning low probability isn’t what makes this hard—what makes this hard is the large number of similarly-goofy-sounding things which we have to choose from, not knowing which ones will work. Assigning a more accurate probability just allows us to make a more accurate cost-benefit analysis in choosing how much of our time to spend on such things. The actual amount of effort it takes to achieve the results (in cases where results are real) doesn’t change with the level of rationality of our beliefs.
Malcolm Ocean has also done the “let me see who lives in my head” exercise, inspired by Brienne.
Ah, cool, thanks!
I myself keep a normal journal every day, recording my state of mind and events. This isn’t exactly the same thing, but I think it approximates some of the benefits, and it also feeds my desire to record my life so ephemeral things have some concrete backing. I’d recommend that if gratitude journals don’t feel right.
For me, regular journalling never felt interesting. I’ve kept a “research thoughts” journal for a long time, but writing about everyday events just didn’t feel very motivating—until CFAR convinced me that life debugging was an interesting thing to do. And then I still needed to find this format to make it into a thing I’d do regularly.
“much” to connect with, I think.
Fixed.
Reminds me of the general tone of Nate Soares’ Simplifience stuff.