My take on predictive processing is a bit different than the textbooks, and in terms of decision theories, it doesn’t wind up radically different from logical inductor decision theory, which Scott talked about in 2017 here, and a bit more here. Or at least, take logical inductor decision theory, make everything about it kinda more qualitative, and subtract the beautiful theoretical guarantees etc.
It’s obvious but worth saying anyway that pretty much all the decision theory scenarios that people talk about, like Newcomb’s problem, are scenarios where people find themselves unsure what to do, and disagree with each other. Therefore the human brain doesn’t give straight answers—or if it does, the answers are not to be found at the “base algorithm” level, but rather the “learned model” level (which can involve metacognition). Or I guess it’s possible that the base-algorithm-default and the learned models are pushing in different directions.
Scott’s 2017 post gives two problems with this decision theory. In my view humans absolutely suffer from both. Like, my friend always buys the more expensive brand of cereal because he’s concerned that he wouldn’t like the less expensive brand. But he’s never tried it! The parallel to the 5-and-10 problem is obvious, right?
The problem about whether to change the map, territory, or both is something I discussed a bit here. Wishful thinking is a key problem—and just looking at the algorithm as I understand it, it’s amazing that humans don’t have even more wishful thinking than we do. I think wishful thinking is kept mostly under control in a couple ways: (1) self-supervised learning effectively gets a veto over what we can imagine happening, by-and-large preventing highly-implausible future scenarios from even entering consideration in the Model Predictive Control competition; (2) The reward-learning part of the algorithm is restricted to the frontal lobe (home of planning and motor action), not the other lobes (home of sensory processing). (Anatomically, the other lobes have no direct connection to the basal ganglia.) This presumably keeps some healthy separation between understanding sensory inputs and “what you want to see”. (I didn’t mention that in my post because I only learned about it more recently; maybe I should go back and edit, it’s a pretty neat trick.) (3) Actually, wishful thinking is wildly out of control in certain domains like post hoc rationalizations. (At least, the ground-level algorithm doesn’t do anything to keep it under control. At the learned-model level, it can be kept under control by learned metacognive memes, e.g. by Reading The Sequences.).
The embedded agency sequence says somewhere that there are still mysteries in human decisionmaking, but (at some risk of my sounding arrogant) I’m not convinced. Everything people do that I can think of, seems to fit together pretty well into the same algorithmic story. I’m very open to discussion about that. Of course, insofar as human decisionmaking has room for improvement, it’s worth continuing to think through these issues. Maybe there’s a better option that we can use for our AGIs.
Or if not, I guess we can build our human-brain-like AGIs and tell them to Read The Sequences to install a bunch of metacognitive memes in themselves that patch the various problems in their own cognitive algorithms. :-P (Actually, I wrote that as a joke but maybe it’s a viable approach...??)
It’s obvious but worth saying anyway that pretty much all the decision theory scenarios that people talk about, like Newcomb’s problem, are scenarios where people find themselves unsure what to do, and disagree with each other. Therefore the human brain doesn’t give straight answers—or if it does, the answers are not to be found at the “base algorithm” level, but rather the “learned model” level (which can involve metacognition).
One point I personally put a lot of weight on: while people are unsure/disagree about particular scenarios, people do mostly seem to agree on what the relevant arguments are, or what the main “options” are for how to think about particular scenarios. That suggests that we do share a common underlying decision-making algorithm, but that algorithm itself sometimes produces uncertain answers.
In particular, for a predictive-processing-like decision theory, it makes sense that sometimes there would be multiple possible self-consistent models. In those cases, we should expect humans to be unsure/disagree, but we’d still expect people to agree on what the relevant arguments/options are—i.e. the possible models.
sometimes there would be multiple possible self-consistent models
I’m not sure what you’re getting at here; you may have a different conception of predictive-processing-like decision theory than I do. I would say “I will get up and go to the store” is a self-consistent model, “I will sit down and read the news” is a self-consistent model, etc. etc. There are always multiple possible self-consistent models—at least one for each possible action that you will take.
Oh, maybe you’re taking the perspective where if you’re hungry you put a high prior on “I will eat soon”. Yeah, I just don’t think that’s right, or if there’s a sensible way to think about it, I haven’t managed to get it despite some effort. I think if you’re hungry, you want to eat because it leads to a predicted reward, not because you have a prior expectation that you will eat. After all, if you’re stuck on a lifeboat in the middle of the ocean, you’re hungry but you don’t expect to eat. This is an obvious point, frequently brought up, and Friston & colleagues hold strong that it’s not a problem for their theory, and I can’t make heads or tails of what their counterargument is. I discussed my version (where rewards are also involved) here, and then here I went into more depth for a specific example.
My take on predictive processing is a bit different than the textbooks, and in terms of decision theories, it doesn’t wind up radically different from logical inductor decision theory, which Scott talked about in 2017 here, and a bit more here. Or at least, take logical inductor decision theory, make everything about it kinda more qualitative, and subtract the beautiful theoretical guarantees etc.
It’s obvious but worth saying anyway that pretty much all the decision theory scenarios that people talk about, like Newcomb’s problem, are scenarios where people find themselves unsure what to do, and disagree with each other. Therefore the human brain doesn’t give straight answers—or if it does, the answers are not to be found at the “base algorithm” level, but rather the “learned model” level (which can involve metacognition). Or I guess it’s possible that the base-algorithm-default and the learned models are pushing in different directions.
Scott’s 2017 post gives two problems with this decision theory. In my view humans absolutely suffer from both. Like, my friend always buys the more expensive brand of cereal because he’s concerned that he wouldn’t like the less expensive brand. But he’s never tried it! The parallel to the 5-and-10 problem is obvious, right?
The problem about whether to change the map, territory, or both is something I discussed a bit here. Wishful thinking is a key problem—and just looking at the algorithm as I understand it, it’s amazing that humans don’t have even more wishful thinking than we do. I think wishful thinking is kept mostly under control in a couple ways: (1) self-supervised learning effectively gets a veto over what we can imagine happening, by-and-large preventing highly-implausible future scenarios from even entering consideration in the Model Predictive Control competition; (2) The reward-learning part of the algorithm is restricted to the frontal lobe (home of planning and motor action), not the other lobes (home of sensory processing). (Anatomically, the other lobes have no direct connection to the basal ganglia.) This presumably keeps some healthy separation between understanding sensory inputs and “what you want to see”. (I didn’t mention that in my post because I only learned about it more recently; maybe I should go back and edit, it’s a pretty neat trick.) (3) Actually, wishful thinking is wildly out of control in certain domains like post hoc rationalizations. (At least, the ground-level algorithm doesn’t do anything to keep it under control. At the learned-model level, it can be kept under control by learned metacognive memes, e.g. by Reading The Sequences.).
The embedded agency sequence says somewhere that there are still mysteries in human decisionmaking, but (at some risk of my sounding arrogant) I’m not convinced. Everything people do that I can think of, seems to fit together pretty well into the same algorithmic story. I’m very open to discussion about that. Of course, insofar as human decisionmaking has room for improvement, it’s worth continuing to think through these issues. Maybe there’s a better option that we can use for our AGIs.
Or if not, I guess we can build our human-brain-like AGIs and tell them to Read The Sequences to install a bunch of metacognitive memes in themselves that patch the various problems in their own cognitive algorithms. :-P (Actually, I wrote that as a joke but maybe it’s a viable approach...??)
One point I personally put a lot of weight on: while people are unsure/disagree about particular scenarios, people do mostly seem to agree on what the relevant arguments are, or what the main “options” are for how to think about particular scenarios. That suggests that we do share a common underlying decision-making algorithm, but that algorithm itself sometimes produces uncertain answers.
In particular, for a predictive-processing-like decision theory, it makes sense that sometimes there would be multiple possible self-consistent models. In those cases, we should expect humans to be unsure/disagree, but we’d still expect people to agree on what the relevant arguments/options are—i.e. the possible models.
I’m not sure what you’re getting at here; you may have a different conception of predictive-processing-like decision theory than I do. I would say “I will get up and go to the store” is a self-consistent model, “I will sit down and read the news” is a self-consistent model, etc. etc. There are always multiple possible self-consistent models—at least one for each possible action that you will take.
Oh, maybe you’re taking the perspective where if you’re hungry you put a high prior on “I will eat soon”. Yeah, I just don’t think that’s right, or if there’s a sensible way to think about it, I haven’t managed to get it despite some effort. I think if you’re hungry, you want to eat because it leads to a predicted reward, not because you have a prior expectation that you will eat. After all, if you’re stuck on a lifeboat in the middle of the ocean, you’re hungry but you don’t expect to eat. This is an obvious point, frequently brought up, and Friston & colleagues hold strong that it’s not a problem for their theory, and I can’t make heads or tails of what their counterargument is. I discussed my version (where rewards are also involved) here, and then here I went into more depth for a specific example.