It occurred to me while reading your comment that I could respond entirely with excerpts from Minding our way. Here’s a go (it’s just fun, if you also find it useful, great!):
You will spend your entire life pulling people out from underneath machinery, and every time you do so there will be another person right next to them who needs the same kind of help, and it goes on and on forever
This is a grave error, in a world where the work is never finished, where the tasks are neverending.
Rest isn’t something you do when everything else is finished. Everything else doesn’t get finished. Rather, there are lots of activities that you do, some which are more fun than others, and rest is an important one to do in appropriate proportions.
Rest isn’t a reward for good behavior! It’s not something you get to do when all the work is finished! That’s finite task thinking. Rather, rest and health are just two of the unending streams that you move through. [...]
the scope of the problem, at least relative to your contribution, is infinite
This behavior won’t do, for someone living in a dark world. If you’re going to live in a dark world, then it’s very important to learn how to choose the best action available to you without any concern for how good it is in an absolute sense. [...]
You will beg for a day in which you go outside and don’t find another idiot stuck under his fucking car
I surely don’t lack the capacity to feel frustration with fools, but I also have a quiet sense of aesthetics and fairness which does not approve of this frustration. There is a tension there.
I choose to resolve the tension in favor of the people rather than the feelings. [...]
somebody else is going to die, you monster
We aren’t yet gods. We’re still fragile. If you have something urgent to do, then work as hard as you can — but work as hard as you can over a long period of time, not in the moment. [...]
You can look at the bad things in this world, and let cold resolve fill you — and then go on a picnic, and have a very pleasant afternoon. That would be a little weird, but you could do it! [...]
So eventually you either give up, or you put earplugs in your ears and go enjoy some time in the woods, completely unable to hear the people yelling for help.
many people seem to think that there is a privileged “don’t do anything” action, that consists of something like curling up into a ball, staying in bed, and refusing to answer emails. It’s much easier to adopt the “buckle down” demeanor when, instead, curling up in a ball and staying in bed feels like just another action. It’s just another way to respond to the situation, which has some merits and some flaws.
(That’s not to say that it’s bad to curl up in a ball on your bed and ignore the world for a while. Sometimes this is exactly what you need to recover. Sometimes it’s what the monkey is going to do regardless of what you decide. [...])
So see the dark world. See everything intolerable. Let the urge to tolerify it build, but don’t relent. Just live there in the intolerable world, refusing to tolerate it. See whether you feel that growing, burning desire to make the world be different. Let parts of yourself harden. Let your resolve grow. It is here, in the face of the intolerable, that you will be able to tap into intrinsic motivation. [...]
You draw boundaries towards questions.
As the links I’ve posted above indicate, no, lists don’t necessarily require questions to begin noticing joints and carving around them.
Questions are helpful however, to convey the guess I might already have and to point at the intension that others might build on/refute. And so...
Your list doesn’t have any questions like that
...I have had some candidate questions in the post since the beginning, and later even added some indication of the goal at the end.
EDIT: You also haven’t acknowledged/objected to my response to your “any attempt to analyse the meaning independent of the goals is confused”, so I’m not sure if that’s still an undercurrent here.
In Where to Draw the Boundaries, Zack points out (emphasis mine):
The one replies:But reality doesn’t come with its joints pre-labeled. Questions about how to draw category boundaries are best understood as questions about values or priorities rather than about the actual content of the actual world. I can call dolphins “fish” and go on to make just as accurate predictions about dolphins as you can. Everything we identify as a joint is only a joint because we care about it.No. Everything we identify as a joint is a joint not “because we care about it”, but because it helps us think about the things we care about.
The one replies:
But reality doesn’t come with its joints pre-labeled. Questions about how to draw category boundaries are best understood as questions about values or priorities rather than about the actual content of the actual world. I can call dolphins “fish” and go on to make just as accurate predictions about dolphins as you can. Everything we identify as a joint is only a joint because we care about it.
No. Everything we identify as a joint is a joint not “because we care about it”, but because it helps us think about the things we care about.
There are more relevant things in there, which I don’t know if you have disagreements with. So maybe it’s more useful to crux with Zack’s main source. In Where to Draw the Boundary, Eliezer gives an example:
And you say to me: “It feels intuitive to me to draw this boundary, but I don’t know why—can you find me an intension that matches this extension? Can you give me a simple description of this boundary?”
I take it this game does not work for you without a goal more explicit than the one I have in the postscript to the question?
(Notice that inferring some aspects of the goal is part of the game; in the specific example Eliezer gave, they’re trying to define Art−which is as nebulous an example as it could be. Self-deception is surely less nebulous than Art.)
I was looking for this kind of engagement, which asserts/challenges either intension or extension:
You come up with a list of things that feel similar, and take a guess at why this is so. But when you finally discover what they really have in common, it may turn out that your guess was wrong. It may even turn out that your list was wrong.
It seemed to me that avoiding fallacies of compression was always a useful thing (independent of your goal, so long as you have the time for computation), even if negligibly. Yet these questions seem to be a bit of a counterexample in mind, namely that I have to be careful when what looks like decoupling might be decontextualizing.
Importantly, I can’t seem to figure out a sharp line between the two. The examples were a useful meditation for me, so I shared them. Maybe I should rename the title to reflect this?
(I’m quite confused by my failure of conveying the point of the meditation, might try redoing the whole post.)
Yes, this is the interpretation.
If I’m doing X wrong (in some way), it’s helpful for me to notice it. But then I notice I’m confused about when decoupling context is the “correct” thing to do, as exemplified in the post.
Rationalists tend to take great pride in decoupling and seeing through narratives (myself included), but I sense there might be some times when you “shouldn’t”, and they seem strangely caught up with embeddedness in a way.
I think I might have made a mistake in putting in too many of these at once. The whole point is to figure out which forms of accusations are useful feedback (for whatever), and which ones are not, by putting them very close to questions we think we’ve dissolved.
Take three of these, for example. I think it might be helpful to figure out whether I’m “actually” enjoying the wine, or if it’s a sort of a crony belief. Disentangling those is useful to make better decisions for myself, in say, deciding to go to a wine-tasting if status-boost with those people wouldn’t help.
Perhaps similarly, I’m better off knowing if my knowledge of whether this food item is organic is interfering with my taste experience.
But then in the movie example, no one would dispute the knowledge is relevant to the experience! Going back to our earlier ones, maybe just the knowledge there was relevant, and “genuinely” making it a better experience?
Maybe my degree of liking is a function of both “knowledge of organic origin” and “chemical interactions with tongue receptors” just like my degree of liking of a movie is a function of both “contextual buildup from the narrative” and “the currently unfolding scene”?
How about when you apply this to “you only upvoted that because of who wrote it”? Maybe that’s a little closer home.
[ETA: posted a Question instead]Question: What’s the difference, conceptually, between each of the following if any?
“You’re only enjoying that food because you believe it’s organic”“You’re only enjoying that movie scene because you know what happened before it”“You’re only enjoying that wine because of what it signals”“You only care about your son because of how it makes you feel”“You only had a moving experience because of the alcohol and hormones in your bloodstream”“You only moved your hand because you moved your fingers”“You’re only showing courage because you’ve convinced yourself you’ll scare away your opponent”
“You’re only enjoying that food because you believe it’s organic”
“You’re only enjoying that movie scene because you know what happened before it”
“You’re only enjoying that wine because of what it signals”
“You only care about your son because of how it makes you feel”
“You only had a moving experience because of the alcohol and hormones in your bloodstream”
“You only moved your hand because you moved your fingers”
“You’re only showing courage because you’ve convinced yourself you’ll scare away your opponent”
Do some of these point legitimately or illegitimately at self-deception?
Are some of these a confusion of levels and others less so?
Are some of these instances of working wishful thinking?
Are some of these better seen as actions rather than rationalizations?
So… it looks like the second AI-Box experiment was technically a loss.
Not sure what to make of it, since it certainly imparts the intended lesson anyway. Was it a little misleading that this detail wasn’t mentioned? Possibly. Although the bet was likely conceded, a little disclaimer of “overtime” would have been nice when Eliezer discussed it.
I was also surprised. Having spoken to a few people with crippling impostor syndrome, the summary seemed to be “people think I’m smart/skilled, but it’s not Actually True.” I think the claim in the article is they’re still in the game when saying that, just another round of downplaying themselves? This becomes really hard to falsify (like internalized misogyny) even if true, so I appreciate the predictions at the end.
I like the idea of it being closer to noise, but there are also reasons to consider the act of advertising theft, or worse:
It feels like the integrity of my will is attacked, when ads work and I know somewhere that I don’t want it to; a divide and conquer attack on my brain, Moloch in my head.
If they get the most out of marketing it to parts of my brain rather than to me as a whole, there is optimization pressure to keep my brain divided, to lower the sanity waterline.
Whenever I’m told to “turn off adblocker”, for that to work for them, it’s premised on me being unable to have an adblocker inside my brain, preying on what’s uncontrollably automatic for me. As if to say: “we both know how this works”. It makes me think of an abuser saying to their victim: “go fetch my belt”.
There’s a game of chicken in “who has to connect potential buyers to sellers, the buyers or the sellers?” and depending on who’s paying to make the transaction happen, we call it “advertisement” or “consultancy”.(You might say “no, that distinction comes from the signal-to-noise ratio”, so question: if increasing that ratio is what works, how come advertisements are so rarely informative?)
As a meta-example, even to this I want to add:
There’s this other economy to keep in mind of readers scrolling past walls of text. Often, I can and want to make what I’m saying cater to multiple attention spans (a la arbital?), and collapsed-by-default comments allow the reader to explore at will.
A strange worry (that may not be true for other people) is attempting to contribute to someone else’s long thread or list feels a little uncomfortable/rude without reading it all/carefully. With collapsed-by-default, you could set up norms that it’s okay to reply without engaging deeply.
It would be nice to have collapsing as part of the formatting
With this I already feel like I’m setting up a large-ish personal garden that would inhibit people from engaging in this conversation even if they want to, because there’s so much going on.
And I can’t edit this into my previous comment without cluttering it.
There’s obviously no need for having norms of “talking too much” when it’s decoupled from the rest of the control system
I do remember Eliezer saying in a small comment somewhere long ago that “the thumb rule is to not occupy more than three places in the Recent Comments page” (paraphrased).
I noticed a thing that might hinder the goals of longevity as described here (“build on what was already said previously”): it feels like a huge cost to add a tiny/incremental comment to something because of all the zero-sum attention games it participates in. It would be nice to do a silent comment, which:
Doesn’t show up in Recent Comments
Collapsed by default
(less confident) Doesn’t show up in author’s notifications (unless “Notify on Silent” is enabled in personal settings)
(kinda weird) Comment gets appended automatically to previous comment (if yours) in a nice, standard format.
The operating metaphor is to allow the equivalent of bulleted lists to span across time, which I suppose would mostly be replies to yourself.
It feels strange to keep editing one comment, and too silent. Also disrupts flow for readers.
I don’t see often that people have added several comments (via edit or otherwise) across months, or even days. Yet people seem to use a lot of nested lists here. Hard to believe that those list-erious ways go away if spread out in time.
Often, people like that will respond well to criticism about X and Y but not about Z.
One (dark-artsy) aspect to add here is that the first time you ask somebody for criticism, you’re managing more than your general identity, you’re also managing your interaction norms with that person. You’re giving them permission to criticize you (or sometimes, even think critically about you for the first time), creating common knowledge that there does exist a perspective from which it’s okay/expected for them to do that. This is playing with the charity they normally extend to you, which might mean that your words and plans will be given less attention than before, even though there might not be any specific criticism in their head. This is especially relevant for low-legibility/fluid hierarchies, which might collapse and impede functioning from the resulting misalignment, perhaps not unlike your own fears of being “crushed”, but at the org level.
Although it’s usually clear that you’d want to get feedback rather than manage this (at least, I think so), it’s important to notice as one kind of anxiety surrounding criticism. This is separate from any narcissistic worries about status, it can be a real systemic worry when you’re acting prosocially.
Incidentally Eliezer, is this really worth your time?
This comment might have caused a tremendous loss of value, if Eliezer took Marcello’s words seriously here and so stopped talking about his metaethics. As Luke points out here, despite all the ink spilled, very few seemed to have gotten the point (at least, from only reading him).
I’ve personally had to re-read it many times over, years apart even, and I’m still not sure I fully understand it. It’s also been the most personally valuable sequence, the sole cause of significant fundamental updates. (The other sequences seemed mostly obvious—which made them more suitable as just incredibly clear references, sometimes if only to send to others.)
I’m sad that there isn’t more.
I’ve read/heard a lot about double crux but never had the opportunity to witness it.
EDIT: I did find one extensive example, but this would still be valuable since it was a live debate.
This one? From the CT-thesis section in A first lesson in meta-rationality.
the objection turns partly on the ambiguity of the terms “system” and “rationality.” These are necessarily vague, and I am not going to give precise definitions. However, by “system” I mean, roughly, a set of rules that can be printed in a book weighing less than ten kilograms, and which a person can consciously follow.11 If a person is an algorithm, it is probably an incomprehensibly vast one, which could not written concisely. It is probably also an incomprehensibly weird one, which one could not consciously follow accurately. I say “probably” because we don’t know much about how minds work, so we can’t be certain.
What we can be certain is that, because we don’t know how minds work, we can’t treat them as systems now. That is the case even if, when neuroscience progresses sufficiently, they might eventually be described that way. Even if God told us that “a human, reasoning meta-systematically, is just a system,” it would be useless in practice. Since we can’t now write out rules for meta-systematic reasoning in less than ten kilograms, we have to act, for now, as if meta-systematic reasoning is non-systematic.
Ideally, I’d make another ninja-edit that would retain the content in my post and the joke in your comment in a reflexive manner, but I am crap at strange loops.
Cold Hands Fallacy/Fake Momentum/Null-Affective Death StallAlthough Hot Hands has been the subject of enough controversy to perhaps no longer be termed a fallacy, there is a sense in which I’ve fooled myself before with a fake momentum. I mean when you change your strategy using a faulty bottomline: incorrectly updating on your current dynamic.
As a somewhat extreme but actual example from my own life: when filling out answersheets to multiple-choice questions (with negative marks for incorrect responses) as a kid, I’d sometimes get excited about having marked almost all of the questions near the end, and then completely, obviously, irrationally decide to mark them all. This was out of some completion urge, and the positive affect around having filled in most of them. This involved a fair bit of self-deception to carry out, since I was aware at some level that I left some of them previously unanswered because I was in fact unsure, and to mark them I had to feel sure.Now, for sure you could make the case that maybe there are times when you’re thinking clearer and when you know the subject or whatever, where you can additionally infer this about yourself correctly and then rationally ramp up the confidence (even if slight) in yourself. But this wasn’t one of those cases, it was the simple fact that I felt great about myself.Anyway the real point of this post is that there’s a flipside (or straightforward generalization) of this: we can talk about this fake inertia for subjects at rest or at motion. What I mean is there’s this similar tendency to not feel like doing something because you don’t have that dynamic right now, hence all the clichés of the form “first blow is half the battle”. In a sense, that’s all I’m communicating here, but seeing it as a simple irrational mistake (as in the example above) really helped me get over this without drama: just remind yourself of the bottomline and start moving in the correct flow, ignoring the uncalibrated halo (or lack thereof) of emotion.