Also, you seem to have slid from “motivated cognition works to produce true beliefs/optimize the world” to the much weaker claim of “some people use motivated cognition, you need to understand it to predict there behavior”. This is a big jump, and feels mote and bailey.
Most parts of the post are explicitly described as “this is how motivated cognition helps us, even if it’s wrong”. Stronger claims return later. And your weaker claim (about predicting people) is still strong and interesting enough.
No you don’t. Penroses theory is totally abstract computability theory. If it were true, then so what? The best for humans facts are something like “alignment is easy, FAI built next week”. This only works if penrose somehow got a total bee in his bonnet about uncomputability, it greatly offended his sensibilities that humans couldn’t know everything. Even though we empirically don’t. Even though pragmatic psycological bounds are a much tighter constraint than computability. In short, your theory of “motivated cognition” doesn’t help predict much. Because you need to assume penroses motivations are just as wacky.
There I talk about the most interesting possibility in the context of physics and math, not Alingment. And I don’t fully endorse Penrose’s “motivation”, even without Alingment his theory is not the most interesting/important thing to me. I treat Penrose’s theory as a local maximum of optimism, not the global maximum. You’re right. But this still helps to remember/highlight his opinions.
I’m not sure FAI is the global maximum of optimism too:
There may be things that are metaphysically more important. (Something about human intelligence and personality.)
We have to take facts into account too. And facts tell that MC doesn’t help to avoid death and suffering by default. Maybe it could help if it were more widespread.
Those two factors make me think FAI wouldn’t be guaranteed if we suddenly learned that “motivated cognition works (for the most part)”.
Most parts of the post are explicitly described as “this is how motivated cognition helps us, even if it’s wrong”. Stronger claims return later. And your weaker claim (about predicting people) is still strong and interesting enough.
There I talk about the most interesting possibility in the context of physics and math, not Alingment. And I don’t fully endorse Penrose’s “motivation”, even without Alingment his theory is not the most interesting/important thing to me. I treat Penrose’s theory as a local maximum of optimism, not the global maximum. You’re right. But this still helps to remember/highlight his opinions.
I’m not sure FAI is the global maximum of optimism too:
There may be things that are metaphysically more important. (Something about human intelligence and personality.)
We have to take facts into account too. And facts tell that MC doesn’t help to avoid death and suffering by default. Maybe it could help if it were more widespread.
Those two factors make me think FAI wouldn’t be guaranteed if we suddenly learned that “motivated cognition works (for the most part)”.