Which means that if having a distorted model helps solve the embedded problem, you’ll probably have a distorted model of what the problem is and/or how you’re solving it.
I think there’s a tricky distinction between “model” and “meta model” being lost here. The model is distorted relative to what and according to what model?
Control systems have to be a model of the system it’s controlling because of the good regulator theorem. If we take a maximally simple control system like a bang bang controller, the structure of the control system has implications about the structure of the plant being controlled. So for example, a bang bang controller is not going to work as a servo position control. It could work for the velocity control loop, or a thermostat, but if you try to use it to control position directly it will oscillate.
If the controller works then it’s fit to reality. The explanation can be wrong in all sorts of ways but the controller itself has to conform to reality or it won’t work.
Bringing this back to the guy trying to attract women, he can say whatever he wants about “She’s gonna like me because I was born on a prime day which is interesting, and chicks dig interesting guys”, and if he uses that to justify the decision to stop communicating to women “Don’t like me” then that will help him out… but the “distortion” is in his meta model, according to my meta model, and his object level model is actually more fit to reality or it wouldn’t work.
So when he says “I gotta believe that she’ll like me, so she’ll like me. Truth tracking isn’t necessarily optimal when you’re embedded in the system!”, I see that as another distorted model. He’s trying to twist his meta meta model in such a way that it doesn’t untwist his meta model in such a way that twists his object level model out of congruence with reality. Instead of just untwisting the relationship between his meta model and object level model, and playing it straight which is gonna work better.
Scratching the next couple paragraphs that came after this, I think I see now that you were using the word “embedded” a bit differently than I interpreted it. Hm. Now I’m less sure I disagree about anything. Because you agree that it’s possible to do truth tracking so that it works, even though “probably important” is a much weaker endorsement of this path than I’d have given. And I strong agree that occlumency is an incredibly important rationality skill (at least, in existing suboptimal cultures).
Maybe I’d just have phrased it “Irrationality is socially strategic (if you’re irrational)”
Okay, maybe this is it: I think occlumency is important because figuring out how to get your meta models sufficiently clean is hard. Like you’re not gonna manage delivering harsh but accurate criticism in a way that people receive as loving and helpful the first time you notice that accurate criticism would be harsh, because you don’t know how to be loving and helpful yet. You don’t yet see your errors for what they are, and so you’re necessarily going to have all sorts of distortions in your meta models to where naive implementations are going to predictably fuck up your object level models if you try to play it the first way that comes to mind. So like, “Back the fuck off. I don’t know what I think yet, and I need to be able to explore thoughts that won’t hold up” is critical. In the same way that training wheels make it hard to learn how to ride a bike. Sure, they keep you upright which is where you generally want to be, but it’s only by leaning that you can learn how to regulate your lean, and that’s a crucial part of balancing for yourself let alone turning.
So we end up with a bell curve meme where the low skill end says “Have you tried not having the problem, you dummy?”, the center of the curve says “Nooo! Saying this is going to cause all sorts of problems because this and that!”, and the high skill end comes back to “Have you tried not having the problem, you dummy?”—which actually works after having untangled all the distortions that lead to that coming off as unloving and un-reality-tracking the first time it comes to mind.
That’s mostly my sense too. I think maybe we don’t disagree about anything substantial. Maybe not anything at all; mostly an aesthetic based on how we naturally view this idea cluster.
Control systems have to be a model of the system it’s controlling because of the good regulator theorem.
Oh man, this was exciting to discover was real. Makes some intuitive sense, but I sure do love when a plausibility argument (conjecture) becomes a necessity argument (proof).
…you agree that it’s possible to do truth tracking so that it works, even though “probably important” is a much weaker endorsement of this path than I’d have given.
I’m not sure what I said that you’re interpreting as my having said “probably important”.
To be clear, I think the core conjecture of LW-style rationality is that there’s a way to rearrange our problem-solving processes to be truth tracking in each, and maybe, every case; and that doing so is significantly more effective than the non-truth-tracking alternatives. If true, I think that’s way more significant than “probably important” would indicate.
In particular, there’s a plausible (though maybe unlikely) pathway here where getting this right could still outpace AI development via introducing a memetic “invasive species” and cause a tidal wave of sanity. It’s one of the few shots I still see as maybe working at this point if we’re in a fast takeoff scenario.
I’m not sure what I said that you’re interpreting as my having said “probably important”.
Hah
I got it from here:
I agree that there’s a way for him to rearrange his relationship to the situation such that truth-seeking isn’t bad anymore. Strong agree. I think developing that type of rearrangement is probably an important part of a mature art of rationality.
Ah yeah. Definitely a case of understatement on my part. My gut sense is more like “Obviously this is absolutely fucking critical and its neglect is one of the key things that’s been holding back real development of this art and the community that’s been trying to practice it for well over a decade.” But I think my stating it that way would tend to create more defensiveness than clarity, and we’d end up in a quagmire of argument that has more to do with feelings and social status than truth. The understated thing is at least true, and I think respects the cultural norms around shared epistemology better.
I’m trying to tease out where we differ, exactly.
I think it’s here:
I think there’s a tricky distinction between “model” and “meta model” being lost here. The model is distorted relative to what and according to what model?
Control systems have to be a model of the system it’s controlling because of the good regulator theorem. If we take a maximally simple control system like a bang bang controller, the structure of the control system has implications about the structure of the plant being controlled. So for example, a bang bang controller is not going to work as a servo position control. It could work for the velocity control loop, or a thermostat, but if you try to use it to control position directly it will oscillate.
If the controller works then it’s fit to reality. The explanation can be wrong in all sorts of ways but the controller itself has to conform to reality or it won’t work.
Bringing this back to the guy trying to attract women, he can say whatever he wants about “She’s gonna like me because I was born on a prime day which is interesting, and chicks dig interesting guys”, and if he uses that to justify the decision to stop communicating to women “Don’t like me” then that will help him out… but the “distortion” is in his meta model, according to my meta model, and his object level model is actually more fit to reality or it wouldn’t work.
So when he says “I gotta believe that she’ll like me, so she’ll like me. Truth tracking isn’t necessarily optimal when you’re embedded in the system!”, I see that as another distorted model. He’s trying to twist his meta meta model in such a way that it doesn’t untwist his meta model in such a way that twists his object level model out of congruence with reality. Instead of just untwisting the relationship between his meta model and object level model, and playing it straight which is gonna work better.
Scratching the next couple paragraphs that came after this, I think I see now that you were using the word “embedded” a bit differently than I interpreted it. Hm. Now I’m less sure I disagree about anything. Because you agree that it’s possible to do truth tracking so that it works, even though “probably important” is a much weaker endorsement of this path than I’d have given. And I strong agree that occlumency is an incredibly important rationality skill (at least, in existing suboptimal cultures).
Maybe I’d just have phrased it “Irrationality is socially strategic (if you’re irrational)”
Okay, maybe this is it: I think occlumency is important because figuring out how to get your meta models sufficiently clean is hard. Like you’re not gonna manage delivering harsh but accurate criticism in a way that people receive as loving and helpful the first time you notice that accurate criticism would be harsh, because you don’t know how to be loving and helpful yet. You don’t yet see your errors for what they are, and so you’re necessarily going to have all sorts of distortions in your meta models to where naive implementations are going to predictably fuck up your object level models if you try to play it the first way that comes to mind. So like, “Back the fuck off. I don’t know what I think yet, and I need to be able to explore thoughts that won’t hold up” is critical. In the same way that training wheels make it hard to learn how to ride a bike. Sure, they keep you upright which is where you generally want to be, but it’s only by leaning that you can learn how to regulate your lean, and that’s a crucial part of balancing for yourself let alone turning.
So we end up with a bell curve meme where the low skill end says “Have you tried not having the problem, you dummy?”, the center of the curve says “Nooo! Saying this is going to cause all sorts of problems because this and that!”, and the high skill end comes back to “Have you tried not having the problem, you dummy?”—which actually works after having untangled all the distortions that lead to that coming off as unloving and un-reality-tracking the first time it comes to mind.
That’s mostly my sense too. I think maybe we don’t disagree about anything substantial. Maybe not anything at all; mostly an aesthetic based on how we naturally view this idea cluster.
Oh man, this was exciting to discover was real. Makes some intuitive sense, but I sure do love when a plausibility argument (conjecture) becomes a necessity argument (proof).
I’m not sure what I said that you’re interpreting as my having said “probably important”.
To be clear, I think the core conjecture of LW-style rationality is that there’s a way to rearrange our problem-solving processes to be truth tracking in each, and maybe, every case; and that doing so is significantly more effective than the non-truth-tracking alternatives. If true, I think that’s way more significant than “probably important” would indicate.
In particular, there’s a plausible (though maybe unlikely) pathway here where getting this right could still outpace AI development via introducing a memetic “invasive species” and cause a tidal wave of sanity. It’s one of the few shots I still see as maybe working at this point if we’re in a fast takeoff scenario.
Hah
I got it from here:
Ah yeah. Definitely a case of understatement on my part. My gut sense is more like “Obviously this is absolutely fucking critical and its neglect is one of the key things that’s been holding back real development of this art and the community that’s been trying to practice it for well over a decade.” But I think my stating it that way would tend to create more defensiveness than clarity, and we’d end up in a quagmire of argument that has more to do with feelings and social status than truth. The understated thing is at least true, and I think respects the cultural norms around shared epistemology better.