let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
I still think you need to have multiple types of belief here, because this fixed prior can’t be used to make later deductions about the world. For example, suppose that I’m stranded in the desert with no food. It’s a new situation, I’ve never been there before. If my prior strongly believes I won’t be hungry 10 hour in the future, I can infer that I’m going to be rescued; and if my prior strongly believes I won’t be sleepy 10 hours from now, then I can infer I’ll be rescued without needing to do anything except take a nap. But of course I can’t (and won’t) infer that.
(Maybe you’ll say “well, you’ve learned from previous experience that the prior is only true if you can actually figure out a way of making it true”? But then you may as well just call it a “goal”, I don’t see the sense in which it’s a belief.)
This type of thing is why I’m wary about “starting with the existing HGM maths”. I agree that it’s rare for humans to ignore symbolic reasoning… but the HGM math might ignore symbolic reasoning! And it could happen in a way that’s pretty hard to spot. If this were my main research priority I’d do it anyway (although even then maybe I’d write this sequence first) but as it is my main goal here is to have a minimum viable epistemology which refutes bayesian rationalism, and helps rationalists reason better about AI.
I’d be interested in your favorite links to the HGM math though, sounds very useful to read up more on it.
I still think you need to have multiple types of belief here, because this fixed prior can’t be used to make later deductions about the world. For example, suppose that I’m stranded in the desert with no food. It’s a new situation, I’ve never been there before. If my prior strongly believes I won’t be hungry 10 hour in the future, I can infer that I’m going to be rescued; and if my prior strongly believes I won’t be sleepy 10 hours from now, then I can infer I’ll be rescued without needing to do anything except take a nap. But of course I can’t (and won’t) infer that.
(Maybe you’ll say “well, you’ve learned from previous experience that the prior is only true if you can actually figure out a way of making it true”? But then you may as well just call it a “goal”, I don’t see the sense in which it’s a belief.)
This type of thing is why I’m wary about “starting with the existing HGM maths”. I agree that it’s rare for humans to ignore symbolic reasoning… but the HGM math might ignore symbolic reasoning! And it could happen in a way that’s pretty hard to spot. If this were my main research priority I’d do it anyway (although even then maybe I’d write this sequence first) but as it is my main goal here is to have a minimum viable epistemology which refutes bayesian rationalism, and helps rationalists reason better about AI.
I’d be interested in your favorite links to the HGM math though, sounds very useful to read up more on it.