I broadly agree with something like “we use a lot of explicit S2 algorithms built on top of the modelling machinery described”, so yes, what I mean more directly apply to the low level, than to humans explicitly thinking about what steps to take.
I think practically useful epistemology for humans needs to deal with both “how is it implemented” and “what’s the content”. To use ML metaphor: human cognition is build out of both “trained neural nets” and “chain-of-thought type inferences in language” running on top of such nets. All S2 reasoning is a prediction in somewhat similar way as all GPT3 reasoning is a prediction—the NN predictor learns how to make “correct predictions” of language, but because the domain itself is partially symbolic world model, this maps to predictions about the world.
In my view some parts of traditional epistemology are confused in trying to do epistemology for humans basically only at the level of the language reasoning, which is a bit like if you try to fix LLM cognition just by writing smart prompts, and ignore there is this huge underlying computation which does the heavy lifting.
I’m certainly in favour of attempts to do epistemology for humans which are compatible with what the underlying computation actually does.
I do agree you can go too far in the opposite direction, ignoring the symbolic reason … but seems rare when people think about humans?
2. My personal take on dark room problem is it is in case of humans mostly fixed by “fixed priors” on interoceptive inputs. I.e. your body has evolutionary older machinery to compute hunger. This gets fed into the predictive processing machinery as input, and the evolutionary sensible belief (“not hungry”) gets fixed. (I don’t think calling this “priors” was good choice of terminology...).
This setup at least in theory rewards both prediction and action, and avoids dark room problems for practical purposes: let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
Given that evolution has good reasons to “fix priors” on multiple evolutionary relevant inputs, I would not expect actual humans to seek dark rooms, but I would expect the PP system occasionally seeking a way how to block or modify the interoceptive signals
3. My impression about how you use ‘frames’ is … the central examples are more like somewhat complex model ensembles including some symbolic/language based components, rather than e.g. “there is gravity” frame or “model of apple” frame. My guess is this will likely be useful for practical use, but with attempts to formalize it, I think a better option is to start with the existing HGM maths.
let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
I still think you need to have multiple types of belief here, because this fixed prior can’t be used to make later deductions about the world. For example, suppose that I’m stranded in the desert with no food. It’s a new situation, I’ve never been there before. If my prior strongly believes I won’t be hungry 10 hour in the future, I can infer that I’m going to be rescued; and if my prior strongly believes I won’t be sleepy 10 hours from now, then I can infer I’ll be rescued without needing to do anything except take a nap. But of course I can’t (and won’t) infer that.
(Maybe you’ll say “well, you’ve learned from previous experience that the prior is only true if you can actually figure out a way of making it true”? But then you may as well just call it a “goal”, I don’t see the sense in which it’s a belief.)
This type of thing is why I’m wary about “starting with the existing HGM maths”. I agree that it’s rare for humans to ignore symbolic reasoning… but the HGM math might ignore symbolic reasoning! And it could happen in a way that’s pretty hard to spot. If this were my main research priority I’d do it anyway (although even then maybe I’d write this sequence first) but as it is my main goal here is to have a minimum viable epistemology which refutes bayesian rationalism, and helps rationalists reason better about AI.
I’d be interested in your favorite links to the HGM math though, sounds very useful to read up more on it.
I broadly agree with something like “we use a lot of explicit S2 algorithms built on top of the modelling machinery described”, so yes, what I mean more directly apply to the low level, than to humans explicitly thinking about what steps to take.
I think practically useful epistemology for humans needs to deal with both “how is it implemented” and “what’s the content”. To use ML metaphor: human cognition is build out of both “trained neural nets” and “chain-of-thought type inferences in language” running on top of such nets. All S2 reasoning is a prediction in somewhat similar way as all GPT3 reasoning is a prediction—the NN predictor learns how to make “correct predictions” of language, but because the domain itself is partially symbolic world model, this maps to predictions about the world.
In my view some parts of traditional epistemology are confused in trying to do epistemology for humans basically only at the level of the language reasoning, which is a bit like if you try to fix LLM cognition just by writing smart prompts, and ignore there is this huge underlying computation which does the heavy lifting.
I’m certainly in favour of attempts to do epistemology for humans which are compatible with what the underlying computation actually does.
I do agree you can go too far in the opposite direction, ignoring the symbolic reason … but seems rare when people think about humans?
2. My personal take on dark room problem is it is in case of humans mostly fixed by “fixed priors” on interoceptive inputs. I.e. your body has evolutionary older machinery to compute hunger. This gets fed into the predictive processing machinery as input, and the evolutionary sensible belief (“not hungry”) gets fixed. (I don’t think calling this “priors” was good choice of terminology...).
This setup at least in theory rewards both prediction and action, and avoids dark room problems for practical purposes: let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
Given that evolution has good reasons to “fix priors” on multiple evolutionary relevant inputs, I would not expect actual humans to seek dark rooms, but I would expect the PP system occasionally seeking a way how to block or modify the interoceptive signals
3. My impression about how you use ‘frames’ is … the central examples are more like somewhat complex model ensembles including some symbolic/language based components, rather than e.g. “there is gravity” frame or “model of apple” frame. My guess is this will likely be useful for practical use, but with attempts to formalize it, I think a better option is to start with the existing HGM maths.
I still think you need to have multiple types of belief here, because this fixed prior can’t be used to make later deductions about the world. For example, suppose that I’m stranded in the desert with no food. It’s a new situation, I’ve never been there before. If my prior strongly believes I won’t be hungry 10 hour in the future, I can infer that I’m going to be rescued; and if my prior strongly believes I won’t be sleepy 10 hours from now, then I can infer I’ll be rescued without needing to do anything except take a nap. But of course I can’t (and won’t) infer that.
(Maybe you’ll say “well, you’ve learned from previous experience that the prior is only true if you can actually figure out a way of making it true”? But then you may as well just call it a “goal”, I don’t see the sense in which it’s a belief.)
This type of thing is why I’m wary about “starting with the existing HGM maths”. I agree that it’s rare for humans to ignore symbolic reasoning… but the HGM math might ignore symbolic reasoning! And it could happen in a way that’s pretty hard to spot. If this were my main research priority I’d do it anyway (although even then maybe I’d write this sequence first) but as it is my main goal here is to have a minimum viable epistemology which refutes bayesian rationalism, and helps rationalists reason better about AI.
I’d be interested in your favorite links to the HGM math though, sounds very useful to read up more on it.