gjm

• gjm 22 Nov 2022 0:11 UTC

  127 points

  120

  on: Here’s the exit.

  If this post had just said “I think some people may feel strongly about AI x-risk for reasons that ultimately come down to some sort of emotional/​physical pain whose origins have nothing to do with AI; here is why I think this, and here are some things you can do that might help find out whether you’re one of them and to address the underlying problem if so”, then I would consider it very valuable and deserving of attention and upvotes and whatnot. I think it’s very plausible that this sort of thing is driving at least some AI-terror. I think it’s very plausible that a lot of people on LW (and elsewhere) would benefit from paying more attention to their bodies.

  … But that’s not what this post does. It says you have to be “living in a[...] illusion” to be terrified by apocalyptic prospects. It says that if you are “feeling stressed” about AI risks then you are “hallucinating”. It says that “what LW is actually about” is not actual AI risk and what to do about it (but, by implication, this alleged “game” of which Eliezer Yudkowsky is the “gamesmaster” that works by engaging everyone’s fight-or-flight reactions to induce terror). It says that, for reasons beyond my understanding, it is impossible to make actual progress on whatever real AI risk problems there might be while in this stressed-because-of-underlying-issues state of mind. It says that “the reason” (italics mine) AI looks like a big threat is because the people to whom it seems like a big threat are “projecting [their] inner hell onto the external world”. And it doesn’t offer the slightest shred of evidence for any of this; we are just supposed to, I dunno, feel in our bodies that Valentine is telling us the truth, or something like that.

  I don’t think this is good epistemics. Maybe there is actually really good evidence that the mechanism Valentine describes here is something like the only way that stress ever arises in human beings. (I wouldn’t be hugely surprised to find that it’s true for the stronger case of terror, and I could fairly easily be convinced that anyone experiencing terror over something that isn’t an immediate physical threat is responding suboptimally to their situation. Valentine is claiming a lot more than that, though.) But in that case I want to see the really good evidence, and while I haven’t gathered any actual statistics on how often people claiming controversial things with great confidence but unwilling to offer good evidence for them turn out to be right and/​or helpful, I’m pretty sure that many of them don’t. Even more so when they also suggest that attempts to argue with them about their claims are some sort of deflection (or, worse, attempts to keep this destructive “game” going) that doesn’t merit engaging with.

  Full disclosure #1: I do not myself feel the strong emotional reaction to AI risk that many people here do. I do not profess to know whether (as Valentine might suggest) this indicates that I am less screwed up psychologically than people who feel that strong emotional reaction, or whether (as Eliezer Yudkowsky might suggest) it indicates that I don’t understand the issues as fully as they do. I suspect that actually it’s neither of those (though either might happen to be true[1]) but just that different people get more or less emotionally involved in things in ways that don’t necessarily correlate neatly with their degree of psychological screwage or intellectual appreciation of the things in question.

  [1] For that matter, the opposite of either might be true, in principle. I might be psychologically screwed up in ways that cut me off from strong emotions I would otherwise feel. I might have more insight into AI risk than the people who feel more strongly that helps me see why it’s not so worrying, or why being scared doesn’t help with it. I think these are both less likely than their opposites, for what it’s worth.

  Full disclosure #2: Valentine’s commenting guidelines discourage commenting unless you “feel the truth of [that Valentine and you are exploring the truth together] in your body” and require “reverent respect”. I honestly do not know, and don’t know how I could tell with confidence, whether Valentine and I are exploring the truth together; at any rate, I do not have the skill (if that’s what it is) of telling what someone else is doing by feeling things in my body. I hope I treat everyone with respect; I don’t think I treat anyone with reverence, nor do I wish to. If any of that is unacceptable to Valentine, so be it.

  Clarification for the avoidance of doubt: I don’t have strong opinions on just what probability we should assign to (e.g.) the bulk of the human race being killed-or-worse as a result of the actions of an AI system within the next century, nor on what psychological response is healthiest for any given probability. The criticisms above are not (at least, not consciously) some sort of disguise for an underlying complaint that Valentine is trying to downplay an important issue, nor for anger that he is revealing that an emperor I admire has no clothes. My complaint is exactly what I say it is: I think this sort of bulveristic “I know you’re only saying this because of your psychological problems, which I shall now proceed to reveal to you; it would be a total waste of time to engage with your actual opinions because they are merely expressions of psychological damage, and providing evidence for my claims is beneath me”[2] game is not only rude (which Valentine admits, and I agree that it is sometimes helpful or even necessary to be rude) but usually harmful and very much not the sort of thing I want to see more of on Less Wrong.

  [2] I do not claim that Valentine is saying exactly those things. But that is very much the general vibe.

  (Also somewhat relevant, though not especially to any of what I’ve written above, and dropped here without further comment: “Existential Angst Factory”.)

  What links here?

  • TekhneMakre's comment on Here’s the exit. by Valentine (22 Nov 2022 3:35 UTC; 5 points)

• gjm 1 May 2013 23:25 UTC

  110 points

  on: What do pro­fes­sional philoso­phers be­lieve, and why?

  Some of the comments here indicate that their authors have severely misunderstood the nature of those seven “major components”, and actually I think the OP may have too.

  They are not clusters in philosopher-space, particular positions that many philosophers share. They are directions in philosopher-space along which philosophers tend to vary. Each could equivalently have been replaced by its exact opposite. They are defined, kinda, by clusters in philosophical idea-space: groups of questions with the property that a philosopher’s position on one tends to correlate strongly with his or her position on another.

  The claim about these positions being made by the authors of the paper is not, not even a little bit, “most philosophers fall into one of these seven categories”. It is “you can generally tell most of what there is to know about a philosopher’s opinions if you know how well they fit or don’t fit each of these seven categories”. Not “philosopher-space is mostly made up of these seven pieces” but “philosopher-space is approximately seven-dimensional”.

  So, for instance, someone asked “Is there a cluster that has more than 1 position in common with LW norms?”. The answer (leaving aside the fact that these things aren’t clusters in the sense the question seems to assume) is yes: for instance, the first one, “anti-naturalism”, is simply the reverse of “naturalism”, which is not far from being The Standard LW Position on everything it covers. The fourth, “anti-realism”, is more or less the reverse of The Standard LW Position on a different group of issues.

  (So why did the authors of the paper choose to use “anti-naturalism” and “anti-realism” rather than “naturalism” and “realism”? I think in each case they chose the more distinctive and less usual of the two opposite poles. Way more philosophers are naturalists and realists than are anti-naturalists and anti-realists. I repeat: these things are not clusters in which a lot of philosophers are found; that isn’t what they’re for.)

  What links here?

  • What do pro­fes­sional philoso­phers be­lieve, and why? by Rob Bensinger (1 May 2013 14:40 UTC; 49 points)
  • Hazard's comment on Open & Wel­come Thread—Jan­uary 2021 by jsalvatier (2 Feb 2021 13:58 UTC; 2 points)

Large lan­guage mod­els learn to rep­re­sent the world

• gjm 14 Apr 2022 13:34 UTC

  89 points

  in reply to: Daniel Bensen’s comment on: Lies Told To Children

  It’s explicitly called out that the red-headed children “tended bigger than the rest of us, with more adult facial structures, to the point where you could’ve maybe mistaken them for very small adults in disguise”. I suspect EY’s intention is that they are in fact small adults in disguise.

• gjm 27 Mar 2013 18:48 UTC

  84 points

  on: Schel­ling Day: A Ra­tion­al­ist Holiday

  I was hoping it would say: “Schelling Day is celebrated every year, on the obvious date.” and leave it at that.

• gjm 29 Jun 2018 21:40 UTC

  81 points

  on: Ma­chine Learn­ing Anal­ogy for Meditation

  Transcript since I find the above basically impossible to read (I have to go and do something else for a bit; will transcribe more when I’m done):

  [note: I have not tried to e.g. turn underlining into italics etc.; this was enough effort as it was; nor does my spacing exactly match the original.]

  ----

  Abram’s Machine-Learning model of the benefits of meditation

  (a synthesis of Zizian “fusion” and Shinzen’s explanation of meditation, also inspired by some of the ideas in Kaj Sotala’s “My attempt to explain Looking and enlightenment in non-mysterious terms” … but this model is no substitute for those sources and does not summarize what they have to say)

  note that I am not an experienced meditator; let that influence your judgement of the validity of what I have to say as it may.

  (also heavily influenced by my CFAR level 2 workshop experience)

  My immediate inspiration for postulating this model was noticing that after just a little meditation, tolerating cold or hot shower temperatures was much easier.

  [picture: person in hot/​cold shower]

  I had previously been paying attentin to what happens in my mind when I flinch away from too-hot or too-cold temperatures in the shower, as a way to pay attention to “thoughts which lead to action”.

  There are several reasons why it might be interesting to pay attention to thoughts which lead to action.

  1. “Where’s the steering wheel on this thing, anyway?” [picture: confusing car dashboard] If you’re experiencing “motivational issues”, then it stands to reason that it might be useful to keep an eye on which thoughts are leading to actions and which are not.

  2. “Who [or what] is steering this thing?” [picture: car with various people in it] Far from being alone in a mysterious spacecraft, it is more like we are on a big road trip with lots of backseat driving and fighting for the wheel, if you buy the multiagent mind picture.

  We often think as if we were unitary, and blame any failings of this picture on a somewhat mysterious limited resource called “willpower”. I’m not implying willpower models are wrong exactly; I’m unsure of what is going on. But bear with me on the multiagent picture...

  I think there is a tendency to gravitate toward narratives where an overarching self with coherent goals drives everything—missing the extent to which we are driven by a variety of urges such as immediate comfort. So, I think it is interesting to watch oneself and look for what really drives actions. You don’t often eat because eating is necessary for continuing proper function of body & brain in order to use them for broader goals; you eat because food tastes good /​ you’re hungry /​ etc.

  Well, maybe. You have to look for yourself. But, it seems easy to mistakenly rationalize goals as belonging to a coherent whole moreso than is the case.

  Why would we be biased to think we are alone in an alien spaceship which we only partly know how to steer, whne in fact we are fighting for the wheel in a crowded road-trip?

  [picture: same car as before, loudmouth backseat driver circled]

  Well, maybe it is because the only way the loudmouth (that is to say, consciousness) gets any respect around here is by maintaining the illusion of control. More on that later.

  3. A third reason to be interested in “thoughts which lead to action” is that it is an agentless notion of decision.

  Normally we think of a decision made by an atomic agent which could have done one of several things; chooses one; and, does it. [picture: person labelled “agent” with “input” and “output” arrows, and “environment” outside] In reality, there is no solid boundary between an agent and its environment; no fixed interface with a well-defined set of actions which act across the interface.

  [picture: brain, spinal cord, muscles, eyeballs, bones, arrows, with circles sketched in various places]

  Instead, there are concentric rings where we might draw such a boundary. The brain? The nerves? The muscles? The skin?

  With a more agentless notion of agency, you can easily look further out.

  Does this person’s thought of political protest cause such a protest to happen? Does the protest lead to the change which it demands?

  Anyway. That is quite enough on what I was thinking in the shower. [picture: recap of some of the pictures from above, in a thought bubble] The point is, after meditation, the thoughts leading to action were quite different, in a way which (temporarily) eliminated any resistance which I had to going under a hot or cold shower which I knew would not harm me but which would ordinarily be difficult to get myself to stand under.

  (I normally can take cold-water showers by applying willpower; I’m talking about a shift in what I can do “easily”, without a feeling of effort.)

  So. My model of this:

  I’m going to be a little bit vague here, and say that we are doing something like some kind of reinforcement learning, and the algorithm we use includes a value table:

  [picture: table, actions on x-axis, states on y-axis, cells of table are estimated values of taking actions in states]

  A value isn’t just the learned estimate of the immediate reward which you get by taking an action in a state, but rather, the estimate of the eventual rewards, in total, from that action.

  This makes the values difficult to estimate.

  An estimate is improved by value iteration: passing current estimates of values back along state transitions to make values better-informed.

  [picture: table like above, with arrows, saying “if (s1,a1)->(s2,a2) is a common transition, propagate backward along the link (s1,a1)<-(s2,a2)”]

  For large state & action sets, this can be too expensive: we don’t have time to propagate along all the possible (state,action) transitions.

  So, we can use attention algorithms to focus selectively on what is highest-priority to propagate.

  The goal of attention is to converge to good value estimates in the most important state,action pairs as efficiently as possible.

  Now, something one might conceivably try is to train the attention algorithm based on reinforcement learning as well. One might even try to run it from the very same value table:

  [picture: value table as before, actions partitioned into “thinking” actions that propagate values and “standard external actions”]

  “The problem with this design is that it can allow for pathological self-reinforcing patterns of attention to emerge. I will provocatively call such self-reinforcing patterns “ego structures”. An ego structure doesn’t so much feed on real control as on the illusion of control.

  [picture: loudmouth-representing-consciousness from before, saying “I told you so!”]

  The ego structure gets it supply of value by directing attention to its apparent successes and away from its apparent failures, including focusing on interpretations of events which make it look like the ego had more control than it did during times of success, and less than it did in cases of failure.

  [picture: car with loudmouth saying “I never said to go left!”]

  Some of this will sound quite familiar to students of cognitive bias. One might normally explain these biases (confirmation bias, optimism bias, attribution bias) as arising from interpersonal incentives (like signalling games).

  I would not discount the importance of those much, but the model here suggests that internal dynamics are also to blame. In my model, biases arise from wireheading effects. In the societal analogies mentioned earlier, we’re looking at regulatory capture and rent-seeking.

  This is rather fuzzy as a concrete mathematical model because I haven’t specified any structure like an “interpretation”—but I suspect details could be filled in appropriately to make it work. (Specifically, model-based reinforcement needs to somehow be tied in.)

  Anyway, where does meditation come in?

  My model is that meditation entices an ego structure with the promise of increased focus (i.e., increased attentional control), which is actually delivered, but at the same time dissolves ego structures by training away any contortions of attention which prevent value iteration from spreading value through the table freely and converging to good estimates efficiently.

  [picture: happy meditating person with value table including “updating” actions in his/​her head]

  How does it provide increased control while dissolving control structures?

  Well, what are you training when you meditate? Overtly, you are training the ability to keep attention fixed on one thing. This is kind of a weird thing to try to get attention to do. The whole point of attention is to help propagate updates as efficiently as possible. Holding attention on one thing is like asking a computer to load the same data repeatedly. It doesn’t accomplish any computation. Why do it?

  [picture: same meditation with an empty-set symbol instead of face + value table]

  Well, it isn’t quite a no-operation. Often, the meditative focus is on something which you try to observe in clarity and detail, like the sensations of the body. This can be useful for other reasons.

  For the sake of my model, though, think of it as the ego structure trying to keep the attention algorithm in what amounts to a no-op, repeatedly requesting attention in a place where the information has already propagated.

  [picture: value table with lots of arrows between the same pair of cells]

  The reason this accomplishes anything is that the ego is not in complete control. Shadows dance beneath the surface.

  [picture: same value table with a bunch of other arrows sketched in too]

  The ego is a set of patterns of attention. It has “attachments”—obligatory mental gymnastics which it has to keep up as part of the power struggle. Indeed, you could say it is only a set of attachments.

  In CFAR terms, an attachment is a trigger-action pattern.

  Examples:

  “Oh, I mustn’t think that way” (rehearsing a negative association to make sure a specific attention pattern stays squished)

  Getting up & getting food to distract yourself when you feel sad

  Rehearsing all the reasons you’ll definitely succeed whenever a failure though comes up

  Meditation forces you to do nothing whenever these thoughts come up, because the only way to maintain attention at a high level is to develop what is called quanimity: any distracting thought is greeted and set aside in the same way, neither holding it up nor squishing it down. No rehearsal of why you must not think that way. No getting up to go to the fridge. No rehearsal of all the reasons why you will definitely succeed.

  Constantly greeting distractions with equanimity and setting them aside fills the value table with zeros where attachments previously lived.

  [picture: value table with a bunch of zeros in the left portion labelled “attentional acts”]

  Note that these are not fake zeros. You are not rewriting your true values out of existence (though it may feel that way to the ego). You are merely experimenting with not responding to thoughts, and observing that nothing terrible happens.

  Another way of thinking about this is un-training the halo effect (though I have not seen any experimental evidence supporting this interpretation). Normally, all entities of conscious experience are imbued with some degree of positive or negative feeling (according to cognitive bias research & experienced meditators alike), which we have flinch-like responses to (trigger-action patterns). Practicing non-response weakens the flinch, allowing more appropriate responses.

  Putting zeros in the table can actually give the ego more control by eliminating some competition. However, in the long term, it destabilizes the power base.

  You might think this makes sustained meditative practice impossible by removing the very motivational structures trying to meditate; and perhaps it sometimes works that way. Another possibility is that the ego is sublimated into a form which serves to sustain the meditative practice, the skills of mental focus /​ mindfulness which have been gained, and the practice of equanimity. This structure serves to ensure that propagation of value through the table remains unclogged by attachments in the future. Such a structure doesn’t need to play games to get credit for what it does, since it is actually useful.

  Regardless, my advice is that you should absolutely not take this model as an invitation to try and dissolve your ego.

  Perhaps take it as an invitation to develop better focus, and to practice equanimity in order to debias halo-effect related problems & make “ugh fields” slowly dissolve.

  I have no particular indication that directly trying to “dissolve ego” is a safe or fruitful goal, however, and some reason to think that it is not. The indirect route to un-wireheading our cognitive strategies through a gently rising tide of sanity seems safest.

  Speaking of the safety of the approach...

  Why doesn’t “zeroing out” the value table destroy our values, again??

  At sinceriously.fyi, Ziz talks about core vs structure.

  Core is where your true values come from. However, core is not complex enough to interface with the world. Core must create structure to think and act on its behalf.

  [picture: central golden circle with complex stuff radiating out from it]

  “Structure” means habits of thinking and doing; models, procedures. Any structure is an approximation of how the values represented by the core play out in some arena of life.

  So, in this model, all the various sub-agents in your mind arise from the core, as parts of the unfolding calculation of the policy maximizing the core’s values.

  These can come into conflict only because they are approximations.

  [picture: that car again, with core+radiating-stuff superimposed on it]

  The model may sound strange at first, but it is a good description of what’s going on in the value-table model I described. (Or rather, the value-table model gives a concrete mechanism for the core/​structure idea.)

  The values in the table are approximations which drive an agent’s policy; a “structure” is a subset of the value table which acts as a coherent strategy in a subdomain.

  Just removing this structure would be bad; but, it would not remove the core values which get propagated around the value table. Structure would re-emerge.

  However, meditation does not truly remove any structure. It only weakens structure by practicing temporary disengagement with it. As I said before, meditation does not introduce any false training data; the normal learning mechanisms are updating on the simple observation of what happens when most of the usual structure is suppressed. This update creates an opportunity to do some “garbage collection” if certain structures prove unnecessary.

  According to this model, all irrationality is coming from the approximation of value which is inherent in structure, and much of the irrationality there is coming from structures trying to grab credit via regulatory capture.

  (“Regulatory capture” refers to getting undue favor from the government, often in the form of spending money lobbying in order to get legislation which is favorably to you; it is like wireheading the government.)

  The reflective value-table model predicts that it is easy to get this kind of irrationality; maybe too easy. For example, addictions can be modeled as a mistaken (but self-reinforcing) attention structure like “But if I think about the hangover I’ll have tomorrow, I won’t want to drink!”

  So long as the pattern successfully blocks vlaue propagation, it can stick.

  (This should be compared with more well-studied models of such irrationality such as hyperbolic discounting.)

  Control of attention is a computationally difficult task, but the premise of Buddhist meditation (particularly Zen) is that you have more to unlearn than to learn. In the model I’m presenting here, that’s because of wireheading by attentional structure.

  However, there is some skill which must be learned. I said earlier that one must learn equanimity. Let’s go into what that means.

  The goal is to form a solid place on which to stand for the purpose of self-evaluation: an attentional structure from which you can judge your other attentional structures impartially.

  [picture: wisdom-seeker on mountaintop nonplussed at being told by lotus-sitting master that all that’s in the way of seeing himself is himself and he should simply stand aside]

  If you react to your own thoughts too judgementally, you will learn to hide them from yourself. Better to simply try to see them clearly, and trust the learning algorithms of the brain to react appropriately. Value iteration will propagate everything appropriately if attention remains unblocked.

  According to some Buddhist teachings, suffering is pain which is not experienced fully; pain with full mindfulness contains no suffering. This is claimed from experience. Why might this be true? What experience might make someone claim this?

  Another idea about suffering is that it results from dwelling on a way that reality differs from how you want it to be which you can’t do anything about.

  Remember, I’m speaking from within my machine-learning model here, which I don’t think captures everything. In particular, I don’t think the two statements above capture everything important about suffering.

  Within the model, though, both statements make sense. We could say that suffering results from a bad attention structure which claims it is still necessary to focus on a thing even though no value-of-information is being derived from i. The only way this can persist is if the attention structure is refusing to look at some aspects of the situation (perhaps because they are too painful), creating a block to value iteration properly scoring the attentional structure’s worth.

  For example, it could be refusal to face the ways in which your brilliant plan to end world hunger will succeed or fail due to things beyond your control. You operate under a model which says that you can solve every potential problem by thinking about it, so you suffer when this is not the case.

  From a rationalist perspective, this may at first sound like a good thing, like the attitude you want. But it ruins the value-of-information calculations, ignores opportunity costs, and stops you from knowing when to give up.

  To act with equanimity is to be able to see a plan as having a 1% chance of success and see it as your best bet anyway, if best bet it is—and in that frame of mind, to be able to devote your whole being toward that plan; and yet, to be able to drop it in a moment if sufficient evidence accumulates in favor of another way.

  So, equanimity is closely tied to your ability to keep your judgements of value and your judgements of probability straight.

  Adopting more Buddhist terminology (perhaps somewhat abusively), we can call the opposite of equanimity “attachment”—to cling to certain value estimates (or certain beliefs) as if they were good in themselves.

  To judge certain states of affairs unacceptable rather than make only relative judgements of better or worse: attachment! You rob yourself of the ability to make tradeoffs in difficult choices!

  To cling to sunk costs: attachment! You rob your future for the sake of maintaining your past image of success!

  To be unable to look at the possibility of failure and leave yourself a line of retreat: attachment! Attachment! Attachment!

  To hunt down and destroy every shred of attachment in oneself—this, too, would be attachment. Unless our full self is already devoted to the task, this will teach some structure to hide itself.

  Instead, equanimity must be learned gently, through nonjudgemental observation of one’s own mind, and trust that our native learning algorithm can find the right structure if we are just able to pay full attention.

  (I say this not because no sect of Buddhism recommends the ruthless route—far from it—nor because I can derive the recommendation from my model; rather, this route seems least likely to lead to ill effects.)

  So, at the five-second level, equanimity is just devoted attention to what is, free from immediate need to judge as positive or negative or to interpret within a pre-conceived story.

  “Between stimulus and response there is a space. In that space is our power to choose our response. In our response lies our growth and our freedom.”—Viktor E. Frankl

  There’s definitely a lot that is missing in this model, and incorrect. However, it does seem to get at something useful. Apply with care.

  -- End --

  What links here?

  • Ma­chine Learn­ing Anal­ogy for Med­i­ta­tion (illus­trated) by abramdemski (28 Jun 2018 22:51 UTC; 97 points)

• gjm 21 Jul 2012 23:04 UTC

  73 points

  on: HP:MOR and the Ra­dio Fallacy

  I think what Harry’s says here is, or at least ought to be, a kind of shorthand for a closely related and much stronger argument.

  It isn’t just that brain damage can take away your mental abilities. It’s that particular kinds of brain damage can take away particular mental abilities, and there’s a consistent correlation between the damage to the brain and the damage to the mind.

  Suppose I show you a box, and you talk to it and it talks back. You might indeed hypothesize that what’s in the box is a radio, and there’s a person somewhere else with whom you’re communicating. But now suppose that you open up the box and remove one electronic component, and the person “at the other end” still talks to you but can no longer remember the names of any vegetables. Then you remove another component, and now they t-t-talk w-with a t-t-t-terrible st-st-st-stutter and keep pausing oddly in the middle of sentences. Another, and they punctuate all their sentences with pointless outbursts of profanity.

  And I have some more of these boxes, and it turns out that they all respond in similar ways to similar kinds of damage.

  How much of this does it take before you regard this as very, very powerful evidence that the mind you’re talking with is implemented by the electronics in the box?

  What links here?

  • CarlShulman's comment on HP:MOR and the Ra­dio Fallacy by RichardChappell (22 Jul 2012 0:44 UTC; 5 points)

• gjm 23 Oct 2014 10:20 UTC

  72 points

  on: 2014 Less Wrong Cen­sus/​Survey

  About two hours ago, I submitted an incomplete census return—it looks as if some keystroke produces an immediate submission, at least on my browser. I’ll be submitting a complete one later today. Yvain, if you want to suppress the incomplete one and need help in identifying it then I can help. I was partway through the calibration questions when I accidentally submitted.

  (I see TrE had the same problem.)

  [EDITED to add: Complete return now submitted.]

  What links here?

  • gjm's comment on 2014 Less Wrong Cen­sus/​Survey by Scott Alexander (23 Oct 2014 14:46 UTC; 59 points)

• gjm 22 Nov 2013 2:54 UTC

  70 points

  on: 2013 Less Wrong Cen­sus/​Survey

  I have taken the survey (and answered, to a good approximation, all the questions).

  Note that if you take the survey and comment here immediately after, Yvain can probably identify which survey is yours. If this possibility troubles you, you may wish to delay. On the other hand, empirically it seems that earlier comments get more karma.

  I conjecture that more than 5% of entrants will experience a substantial temptation to give SQUEAMISH OSSIFRAGE as their passphrase at the end. The purpose of this paragraph is to remark that (1) if you, the reader, are so tempted then that is evidence that I am right, and (2) if so then giving in to the temptation is probably a bad idea.

On “aiming for con­ver­gence on truth”

Buy­ing happiness

• gjm 2 Apr 2022 11:19 UTC

  60 points

  on: Good Heart Week: Ex­tend­ing the Experiment

  Add me to the list of people for whom the incentives go the exact other way: I see this obviously-April-fool thing being done with LW karma, I see it’s even called “Goodhart”, and it all makes me feel rather ugh about posting or commenting. (This isn’t about the money for me, which I see it is for a couple of other people on Team Disincentive; I thought yesterday that more likely than not the bit about paying out money was part of the joke, and that didn’t stop me feeling ugh about the whole thing.)

  To be clear, this doesn’t mean I think it was a bad April fool, and it doesn’t mean I think continuing it a few days would have bad outcomes overall. Almost all of what I do is commenting rather than posting, and while I think my comments are generally pretty good they’re never game-changing; if the tradeoff is that I comment a lot less and someone else posts a bit more then it’s probably a good tradeoff.

• gjm 23 Oct 2014 14:46 UTC

  59 points

  on: 2014 Less Wrong Cen­sus/​Survey

  Submitted. (Yvain, if you’re reading this, you might want to see my note about an accidental incomplete submission.)

• gjm 26 Mar 2016 1:44 UTC

  58 points

  in reply to: Huluk’s comment on: Less­wrong 2016 Survey

  I have taken the survey.

• gjm 25 Jan 2023 22:18 UTC

  57 points

  7

  on: Sapir-Whorf for Rationalists

  I think this is mostly excellent, but one thing rings false to me.

  When discussing costs of new words, you repeatedly say or assume something along the following lines: “The cost to me of introducing a new word is the inconvenience /​ cognitive burden /​ manipulation risk that I take on by using that new word”. That’s true if the question is “should I start using this new word?”. But if this is meant to be relevant to your posting things on LW proposing new terminology, the actual question is something more like “should this community start using this new word?”, and that brings in both new benefits and new costs.

  You do explicitly acknowledge the benefits that come from a community, rather than an isolated individual, acquiring a new concept-word. But nothing in what you write acknowledges the costs:

  • If the rest of the community is using a new term, then I have to choose between getting it into my head and having trouble understanding what everyone else is saying.

  • If a community starts using a new term that isn’t standard elsewhere, then someone encountering the community for the first time will (1) have a little bit more difficulty making sense of what everyone else is saying and (2) be a little bit more likely to think “bunch of weirdos with their own jargon, avoid”.

  These aren’t pure costs. (If I find some neologism useless, and the people holding a particular discussion find it useful enough that I can’t contribute to the discussion without getting my head around the neologism … well, maybe that indicates that I’m not familiar enough with some relevant issues for my opinions to be valuable. Maybe newcomers who aren’t willing to put in some effort to figure out the local jargon are less likely to be useful contributors. Maybe newcomers who would be put off by neologism-dense discussion are less likely to be useful contributors.) But they are costs, and not necessarily negligible ones.

  (The following is pure handwaving unsupported by actual data: my impression is that highly neologistic communities are often unhealthy ones, perhaps partly because they put off other people who might criticize, perhaps because the neologisms actually have exclusion as a goal either consciously or not. I don’t know whether this is a likely failure mode for the LW community, nor even whether it’s actually a thing that happens much outside my imagination, but to me it feels like something worth worrying about a little bit.)

• gjm 23 Oct 2014 11:17 UTC

  53 points

  on: 2014 Less Wrong Cen­sus/​Survey

  Is it deliberate that the size of the MONETARY REWARD is not stated anywhere?

• gjm 5 Apr 2022 0:53 UTC

  52 points

  on: Dr Fauci as Machi­avel­lian Boddhisattva

  I am pretty sure what is going on with those advertisements and others’ responses to your criticism of them is:

  • The other people don’t see the ads as having the subtext you do

    • … and find your reasons for seeing them as having that subtext sufficiently alien that they don’t take seriously the possibility that you really truly think the ads have that subtext.

  • So they don’t think you are “complaining about anti-mask propaganda”

    • … and even if you explicitly say “I am complaining about anti-mask propaganda” they think you’re framing things that way for rhetorical reasons, not that you seriously think the ads are anti-mask propaganda.

  • So what they are construing as an anti-mask position is not in their view a complaint about anti-mask propaganda.

  I have not seen the advertisements in question (your tweet has a picture of the other one but not of these), but my guess from your description of them here is that (1) I would not, seeing them in isolation from your comments, take them to be anti-mask propaganda, and that (2) even given your comments, I would not think it likely that they were intended as, or that they function in practice as, anti-mask propaganda.

  (I am in fact having trouble believing that you seriously think the ads are anti-mask propaganda, rather than claiming that for rhetorical purposes. I may of course be wrong; I mention this just because it seems like evidence that other people might feel the same way, especially if unlike me they haven’t explicitly sat and thought for a minute about what it is likely that you mean, and had the context of this post as well as your tweet.)

  It seems to me that your reasoning here resembles a common (and in my opinion very wrong) pattern of thought: “These people say X; also, Y; X and Y imply Z; therefore these people think Z”. The reason it’s wrong, of course, is that these people may well not believe Y. There is maybe an implicit “no one with half a brain reasoning in good faith could fail to believe Y”, and I think such propositions are usually false.

  Another instance of approximately the same error: “They are not trying to live. They are not trying to save their friends’ lives. If they were, they would have picked up my message about high-dose Vitamin D”. You are firmly convinced that high-dose vitamin D is plainly very helpful, but they may not be, and the reason need not be stupidity or dishonesty. E.g., Scott Alexander, at least as of December 2021, doesn’t think it likely that vitamin D is useful against Covid-19; he may be right or wrong, but to me this is already conclusive evidence that it isn’t obvious to any smart person who’s paying attention that vitamin D is useful against Covid-19. Which means that “if these people were really trying to save their own and their friends’ lives, they would agree with me about vitamin D” is just plain wrong.

• gjm 12 Feb 2023 18:47 UTC

  51 points

  21

  on: Con­flict The­ory of Bounded Distrust

  I have several “local” nitpicks and a “global” objection to the overall narrative that I think is being proposed here. Local issues first.

  Alexander concludes with a characteristically mistake-theoretic plea for mutual understanding

  He does. But one thing you don’t mention which I think makes your presentation of SA’s argument misleading is this: between the things you’ve previously described (about Fox News, liberal academics, and the Bush/​Quayle “revenue enhancements”, all things going on in the contemporary-ish USA) and the bit you go on to quote (referring to “a hostile environment that lies to them all the time) SA’s piece has moved on somewhat, and (1) the “hostile environment” bit is not talking about the contemporary US but about (a somewhat hypothetical version of) the Stalin-era USSR and (2) the business about interactions between “savvy” and “clueless” people is addressing a fundamentally different question from most of the article.

  So (1) to whatever extent you’re taking SA’s article to say that the contemporary USA (or other similar places) is “a hostile environment that lies to us all the time”, I think that is an error (maybe SA would in fact agree with you about that, maybe not, but at any rate it isn’t what he says).

  And (2) he isn’t saying we should necessarily regard the presenters on Fox News, or those liberal academics, or the writers in the Washington Post, or the government of the Stalinist USSR, as being honestly mistaken. That isn’t what the conflict/​mistake dichotomy is about. When he talks about our attitudes to those people, he does so in terms of “they aren’t honest, but are they likely to be lying to me about this, in this particular way, in this particular context?”. The mistake-theory-ish bit you quote comes in only at the end and is about an entirely different question: how should we interact with people whose assessment of the honesty of what those would-be authorities are saying is different from ours?

  We need a conflict theory to understand this type of situation [sc. cop-bribery].

  “Conflict theory” and “mistake theory” don’t mean thinking that everyone all the time is or isn’t working towards the same goal. Obviously different people have different goals, sometimes opposing ones. The terms only make sense in the context of some sort of discussion (e.g., a political one) where the differences between you and your interlocutor may or may not be conflict-y or mistake-y. The bribing-a-cop scenario is not of this type, and “we need a conflict theory to understand this type of situation” seems to me like a category mistake.

  (Remark: I think conflict/​mistake oversimplifies in important ways. 1. We can have the same ultimate goals but still relate in conflict-y ways, if our differing opinions give us opposing instrumental goals and prospects for reaching agreement on those opinions are poor. 2. There are ways to have different goals that aren’t of the form “I want my group to be on top, you want your group to be on top” and while these may still lead to conflict I think it’s a fundamentally less-hostile sort of conflict.)

  When a government denies tax increases but announces “revenue enhancements” [...] regime supporters and dissidents are trying to do different things. Dissidents want to create common knowledge of the regime’s shortcomings [...]

  There are definitely situations where “dissidents are trying to create common knowledge of the regime’s shortcomings” so that when the right time comes everyone can have enough confidence to revolt. But SA’s example of “revenue enhancements” is unambiguously not one of those situations. One didn’t need any particular degree of common knowledge to not vote for George Bush. No one was proposing an armed revolt or anything similarly risky. Saying “aha, Bush did levy new taxes despite saying he wouldn’t” did not put one in danger of being “crushed as an individual”.

  (This is a place where I think you are taking advantage of your earlier conflation of contemporary-US and Stalinist-USSR situations in SA’s article.)

  Further, while the “revenue enhancements” thing is obviously slimy, it’s not remotely in the same category as e.g. the things in the “Kolmogorov complicity” article you link to. Saying that thunder is heard before the corresponding lightning is seen (SA’s example in that article) is flatly incompatible with reality; you can’t actually believe it along with the truth about how physics and thunderstorms work, but you can call a tax a “revenue enhancement” without any actual false beliefs about reality. (You probably can’t think that’s optimal terminology for good thinking without false beliefs, but most people most of the time are not choosing their terminology solely to optimize good thinking, and it’s not at all clear that they should.)

  As for the overall narrative:

  The impression I get from your article is something along the following lines: “SA is a mistake-theorist; he wants us to think of other people as basically on the same side as us, and reason politely with them. His article about bounded distrust applies this thinking to governments, major media sources, etc. But this is all wrong and possibly outright sinister: governments, major media sources, etc., are actively trying to mislead us for their own ends, and the people who want to think in mistake-theory terms in such a situation are the lackeys of Power, the government mouthpieces and suchlike, as opposed to the brave dissidents who see the conflict for what it is.” With a somewhat-plausibly-deniable side order of “Boooo to SA, who has shown himself to be on the side of Power, which is much like the government of the Stalinist USSR”.

  And I think most of this narrative is wrong. SA is indeed a mistake-theorist, but he conspicuously doesn’t take that to mean that the mouthpieces of state/​cultural/​… power should be assumed to be arguing in good faith. His article about bounded distrust, in particular, doesn’t suggest doing that. I see no reason to think that his general preference for mistake theory indicates that he is on the side of Power (whatever specific sort of Power that might be). I do not think any sort of Power he is plausibly on the side of has much in common with the Stalinist USSR.

Sus­pi­ciously bal­anced evidence

• gjm 23 Oct 2014 10:33 UTC

  50 points

  on: 2014 Less Wrong Cen­sus/​Survey

  The question called P(Global Catastrophic Risk) should really be called something more like P(not Global Catastrophic Risk). (Or else the question itself should be inverted, but that would be a Bad Idea since some people have now filled in the survey.)