I think the LW zeitgeist doesn’t really engage with this.
Really? I feel quite the opposite, unless you’re saying we could do still more. I think LW is actually one of the few communities that take this sort of non-dualism/naturalism in arriving at a probabilistic judgement (and all its meta levels) seriously. We’ve been repeatedly exposed to the fact that Newcomblike problems are everywhere since a long time ago, and then relatively recently, with Simler’s wonderful post on crony beliefs (and now, his even more delightful book with Hanson, of course).
ETA: I’m missing quite a few posts that were even older (Wei Dai’s? Drescher’s? yvain had something too IIRC), it’d be nice if someone else who does remember posted them here.
I think your links are a good indication of the way that LW has engaged with a relatively narrow aspect of this, and with a somewhat biased manner. “Crony beliefs” is a good example—starting right from the title, it sets up a dichotomy of “merit beliefs” versus “crony beliefs”, with the not-particularly-subtle-connotation of “merit beliefs are this great thing that models reality and in an ideal world we’d only have merit beliefs, but in the real world, we also have to deal with the fact that it’s useful to have crony beliefs for the purpose of manipulating others and securing social alliances”.
Which… yes, that is one aspect of this. But the more general point of the original post is that there are a wide variety of beliefs which are underdetermined by external reality. It’s not that you intentionally have fake beliefs which out of alignment with the world, it’s that some beliefs are to some extent self-fulfilling, and their truth value just is whatever you decide to believe in. If your deep-level alief is that “I am confident”, then you will be confident; if your deep-level alief is that “I am unconfident”, then you will be that.
Another way of putting it: what is the truth value of the belief “I will go to the beach this evening”? Well, if I go to the beach this evening, then it is true; if I don’t go to the beach this evening, it’s false. Its truth is determined by the actions of the agent, rather than the environment.
The predictive processing thing could be said to take this even further: it hypothesizes that all action is caused by these kinds of self-fulfilling beliefs; on some level our brain believes that we’ll take an action, and then it ends up fulfilling that prediction:
About a third of Surfing Uncertainty is on the motor system, it mostly didn’t seem that interesting to me, and I don’t have time to do it justice here (I might make another post on one especially interesting point). But this has been kind of ignored so far. If the brain is mostly just in the business of making predictions, what exactly is the motor system doing?
Based on a bunch of really excellent experiments that I don’t have time to describe here, Clark concludes: it’s predicting action, which causes the action to happen.
This part is almost funny. Remember, the brain really hates prediction error and does its best to minimize it. With failed predictions about eg vision, there’s not much you can do except change your models and try to predict better next time. But with predictions about proprioceptive sense data (ie your sense of where your joints are), there’s an easy way to resolve prediction error: just move your joints so they match the prediction. So (and I’m asserting this, but see Chapters 4 and 5 of the book to hear the scientific case for this position) if you want to lift your arm, your brain just predicts really really strongly that your arm has been lifted, and then lets the lower levels’ drive to minimize prediction error do the rest.
Under this model, the “prediction” of a movement isn’t just the idle thought that a movement might occur, it’s the actual motor program. This gets unpacked at all the various layers – joint sense, proprioception, the exact tension level of various muscles – and finally ends up in a particular fluid movement
Now, I’ve mostly been talking about cases where the truth of a belief is determined purely by our choices. But as the OP suggests, there are often complex interplays between the agent and the environment. For instance, if you believe that “I will be admitted to Example University if I study hard enough to get in”, then that belief may become self-fulfilling in that it causes you to study hard enough to get in. But at the same time, you may simply not be good enough, so the truth value of this belief is determined both by whether you believe in it, and by whether you actually are good enough.
With regard to the thing about confidence; people usually aren’t just confident in general, they are confident about something in particular. I’m much more confident in my ability to write on a keyboard, than I am in my ability to do brain surgery. You could say that my confidence in my ability to do X, is the probability that I assign to doing X successfully.
And it’s often important that I’m not overconfident. Yes, if I’m really confident in my ability to do something, then other people will give me more respect. But the reason why they do that, is that confidence is actually a bit of a costly signal. So far I’ve said that an agent’s decisions determine the truth-values of many beliefs, but it’s also the other way around: the agent’s beliefs determine the agent’s actions. If I believe myself to be really good at brain surgery even when I’m not, I may be able to talk myself into a situation where I’m allowed to do brain surgery, but the result will be a dead patient. And it’s not going to take many dead patients before people realize I’m a fraud and put me in prison. But if I’m completely deluded and firmly believe myself to be a master brain surgeon, that belief will cause me to continue carrying out brain surgeries, even when it would be better from a self-interested perspective to stop doing that.
So there’s a complicated thing where beliefs have several effects: they determine your predictions about the world and they determine your future actions and they determine the subconscious signals that you send to others. You have an interest in being overconfident for the sake of persuading others, and for the sake of getting yourself to do things, but also in being just-appropriately-confident for the sake of being able to predict the consequences of your own future actions better.
An important framing here is “your beliefs determine your actions, so how do you get the beliefs which cause the best actions”. There have been some posts offering tools for belief-modification which had the goal of causing change, but this mostly hasn’t been stated explicitly, and even some of the posts which have offered tools for this (e.g. Nate’s “Dark Arts of Rationality”) have still talked about it being a “Dark Art” thing which is kinda dirty to engage in. Which I think is dangerous, because getting an epistemically correct map is only half of what you need for success, with the “have beliefs which cause you to take the actions that you need to succeed” being the other half that’s just as important to get right. (Except, as noted, they are not two independent things but intertwined with each other in complicated ways.)
There’s a thing MIRI people talk about, about the distinction between “cartesian” and “naturalized” agents: a cartesian agent is something like AIXI that has a “cartesian boundary” separating itself from the environment, so it can try to have accurate beliefs about the environment, then try to take the best actions on the environment given those beliefs. But a naturalized agent, which is what we actually are and what any AI we build actually is, is part of the environment; there is no cartesian boundary. Among other things this means that the environment is too big to fully model, and it’s much less clear what it even means for the agent to contemplate taking different actions. Scott Garrabrant has said that he does not understand what naturalized agency means; among other things this means we don’t have a toy model that deserves to be called “naturalized AIXI.”
There’s a way in which I think the LW zeitgeist treats humans as cartesian agents, and I think fully internalizing that you’re a naturalized agent looks very different, although my concepts and words around this are still relatively nebulous.