Right, I probably should’ve expected this objection and pre-addressed it more thoroughly.
I think this is a bit of “missing the forest for trees”. In my view, every single human concept and every single human train of thoughts is an example of human world-models’ autosymbolicity. What are “cats”, “trees”, and “grammar”, if not learned variables from our world-model that we could retrieve, understand the semantics of, and flexibly use for the purposes of reasoning/problem-solving?
We don’t have full self-interpretability by default, yes. We have to reverse-engineer our intuitions and instincts (e. g., grammar from your example), and for most concepts, we can’t break their definitions down into basic mathematical operations. But in modern adult humans, there is a vast interpreted structure that contains an enormous amount of knowledge about the world, corresponding to, well, literally everything a human consciously knows. Which, importantly, includes every fruit of our science and technology.
If we understood an external superhuman world-model as well as a human understands their own world-model, I think that’d obviously get us access to tons of novel knowledge.
Which, importantly, includes every fruit of our science and technology.
I don’t think this is the right comparison, since modern science / technology is a collective effort and so can only cumulate thinking through mostly-interpretable steps. (This may also be true for AI, but if so then you get interpretability by default, at least interpretability-to-the-AIs, at which point you are very likely better off trying to build AIs that can explain that to humans.)
In contrast, I’d expect individual steps of scientific progress that happen within a single mind often are very uninterpretable (see e.g. “research taste”).
If we understood an external superhuman world-model as well as a human understands their own world-model, I think that’d obviously get us access to tons of novel knowledge.
Sure, I agree with that, but “getting access to tons of novel knowledge” is nowhere close to “can compete with the current paradigm of building AI”, which seems like the appropriate bar given you are trying to “produce a different tool powerful enough to get us out of the current mess”.
Perhaps concretely I’d wildly guess with huge uncertainty that this would involve an alignment tax of ~4 GPTs, in the sense that if you had an interpretable world model from GPT-10 similar in quality to a human’s understanding of their own world model, that would be similarly useful as GPT-6.
Right, I probably should’ve expected this objection and pre-addressed it more thoroughly.
I think this is a bit of “missing the forest for trees”. In my view, every single human concept and every single human train of thoughts is an example of human world-models’ autosymbolicity. What are “cats”, “trees”, and “grammar”, if not learned variables from our world-model that we could retrieve, understand the semantics of, and flexibly use for the purposes of reasoning/problem-solving?
We don’t have full self-interpretability by default, yes. We have to reverse-engineer our intuitions and instincts (e. g., grammar from your example), and for most concepts, we can’t break their definitions down into basic mathematical operations. But in modern adult humans, there is a vast interpreted structure that contains an enormous amount of knowledge about the world, corresponding to, well, literally everything a human consciously knows. Which, importantly, includes every fruit of our science and technology.
If we understood an external superhuman world-model as well as a human understands their own world-model, I think that’d obviously get us access to tons of novel knowledge.
I don’t think this is the right comparison, since modern science / technology is a collective effort and so can only cumulate thinking through mostly-interpretable steps. (This may also be true for AI, but if so then you get interpretability by default, at least interpretability-to-the-AIs, at which point you are very likely better off trying to build AIs that can explain that to humans.)
In contrast, I’d expect individual steps of scientific progress that happen within a single mind often are very uninterpretable (see e.g. “research taste”).
Sure, I agree with that, but “getting access to tons of novel knowledge” is nowhere close to “can compete with the current paradigm of building AI”, which seems like the appropriate bar given you are trying to “produce a different tool powerful enough to get us out of the current mess”.
Perhaps concretely I’d wildly guess with huge uncertainty that this would involve an alignment tax of ~4 GPTs, in the sense that if you had an interpretable world model from GPT-10 similar in quality to a human’s understanding of their own world model, that would be similarly useful as GPT-6.