To solve problems, researchers may use algorithms that terminate in a finite number of steps, or iterative methods that converge to a solution (on some specified class of problems), or heuristics that may provide approximate solutions to some problems (although their iterates need not converge).
I don’t quite understand this. What does the sentence “computational optimization can compute all computable functions” mean? Additionally, in my conception of “computational optimization” (which is admittedly rather vague), learning need not take place.
I might have overloaded the phrase “computational” here. My intention was to point out what can be encoded by such a system. Maybe “coding” is a better word? E.g., neural coding. These systems can implement Turing machines so can potentially have the same properties of turing machines.
these two options are conceptually quite different and might influence the meaning of the analogy. If intelligence computes only a “target direction”, then this corresponds to a heuristic approach in which locally, the correct direction in action space is chosen. However, if you view intelligence as an actual optimization algorithm, then what’s chosen is not only a direction but a whole path.
I’m wondering if our disagreement is conceptual or semantic. Optimizing a direction instead of an entire path is just a difference in time horizon in my model. But maybe this is a different use of the word “optimize”?
You write “Learning consists of setting the right weights between all the neurons in all the layers. This is analogous to my understanding of human intelligence as path-finding through reality”
Learning is a thing you do once, and then you use the resulting neural network repeatedly. In contrast, if you search for a path, you usually use that path only once.
If I learn the optimal path to work, then I can use that multiple times. I’m not sure I agree with the distinction you are drawing here … Some problems in life only need to be solved exactly once, but that’s the same as any thing you learn only being applicable once. I didn’t mean to claim the processes are identical, but that they share an underlying structure. Though indeed, this might an empty intuitive leap with no useful implementation. Or maybe not a good matching at all.
I do not know what you mean by “mapping a utility function to world states”. Is the following a correct paraphrasing of what you mean?
“An aligned AGI is one that tries to steer toward world states such that the neurally encoded utility function, if queried, would say ‘these states are rather optimal’ ”
Yes, thank you.
I don’t quite understand the analogy to hyperparameters here. To me, it seems like childbirth’s meaning is in itself a reward that, by credit assignment, leads to a positive evaluation of the actions that led to it, even though in the experience the reward was mostly negative. It is indeed interesting figuring out what exactly is going on here (and the shard theory of human values might be an interesting frame for that, see also this interesting post looking at how the same external events can trigger different value updates), but I don’t yet see how it connects to hyperparameters.
A hyperparameter is a parameter across parameters. So say with childbirth, you have a parameter pain on physical pain which is a direct physical signal, and you have a hyperparameter ‘Satisfaction from hard work’ that takes ‘pain’ as input as well as some evaluative cognitive process and outputs reward accordingly. Does that make sense?
What if instead of trying to build an AI that tries to decode our brain’s utility function, we build the process that created our values in the first place and expose the AI to this process?
Digging in to shard theory is still on my todo list. [bookmarked]
Many models that do not overfit also memorize much of the data set.
Is this on the sweet spot just before overfitting or should I be thinking of something else?
I might have overloaded the phrase “computational” here. My intention was to point out what can be encoded by such a system. Maybe “coding” is a better word? E.g., neural coding. These systems can implement Turing machines so can potentially have the same properties of turing machines.
I see. I think I was confused since, in my mind, there are many Turing machines that simply do not “optimize” anything. They just compute a function.
I’m wondering if our disagreement is conceptual or semantic. Optimizing a direction instead of an entire path is just a difference in time horizon in my model. But maybe this is a different use of the word “optimize”?
I think I wanted to point to a difference in the computational approach of different algorithms that find a path through the universe. If you chain together many locally found heuristics, then you carve out a path through reality over time that may lead to some “desirable outcome”. But the computation would be vastly different from another algorithm that thinks about the end result and then makes a whole plan of how to reach this. It’s basically the difference between deontology and consequentialism. This post is on similar themes.
I’m not at all sure if we disagree about anything here, though.
If I learn the optimal path to work, then I can use that multiple times. I’m not sure I agree with the distinction you are drawing here … Some problems in life only need to be solved exactly once, but that’s the same as any thing you learn only being applicable once.
I would say that if you remember the plan and retrieve it later for repeated use, then you do this by learning and the resulting computation is not planning anymore. Planning is always the thing you do at the moment to find good results now, and learning is the thing you do to be able to use a solution repeatedly.
Part of my opinion also comes from the intuition that planning is the thing that derives its use from the fact that it is applied in complex environments in which learning by heart is often useless. The very reason why planning is useful for intelligent agents is that they cannot simply learn heuristics to navigate the world.
To be fair, it might be that I don’t have the same intuitive connection between planning and learning in my head that you do, so if my comments are beside the point, then feel free to ignore :)
A hyperparameter is a parameter across parameters. So say with childbirth, you have a parameter pain on physical pain which is a direct physical signal, and you have a hyperparameter ‘Satisfaction from hard work’ that takes ‘pain’ as input as well as some evaluative cognitive process and outputs reward accordingly. Does that make sense?
Conceptually it does, thank you! I wouldn’t call these parameters and hyperparameters, though. Low-level and high-level features might be better terms.
Again, I think the shard theory of human values might be an inspiration for these thoughts, as well as this post on AGI motivation which talks about how valence gets “painted” on thoughts in the world model of a brain-like AGI.
Is this on the sweet spot just before overfitting or should I be thinking of something else?
I personally don’t have good models for this. Ilya Sutskever mentioned in a podcast that under some models of bayesian updating, learning by heart is optimal and a component of perfect generalization. Also from personal experience, I think that people who generalize very well also often have lots of knowledge, though this may be confounded by other effects.
I thought I added it but apparently hadn’t pressed submit. Thank you for pointing that out!
I was going by the Wikipedia definition:
I might have overloaded the phrase “computational” here. My intention was to point out what can be encoded by such a system. Maybe “coding” is a better word? E.g., neural coding. These systems can implement Turing machines so can potentially have the same properties of turing machines.
I’m wondering if our disagreement is conceptual or semantic. Optimizing a direction instead of an entire path is just a difference in time horizon in my model. But maybe this is a different use of the word “optimize”?
If I learn the optimal path to work, then I can use that multiple times. I’m not sure I agree with the distinction you are drawing here … Some problems in life only need to be solved exactly once, but that’s the same as any thing you learn only being applicable once. I didn’t mean to claim the processes are identical, but that they share an underlying structure. Though indeed, this might an empty intuitive leap with no useful implementation. Or maybe not a good matching at all.
Yes, thank you.
A hyperparameter is a parameter across parameters. So say with childbirth, you have a parameter pain on physical pain which is a direct physical signal, and you have a hyperparameter ‘Satisfaction from hard work’ that takes ‘pain’ as input as well as some evaluative cognitive process and outputs reward accordingly. Does that make sense?
Digging in to shard theory is still on my todo list. [bookmarked]
Is this on the sweet spot just before overfitting or should I be thinking of something else?
Thank you for you extensive comment! <3
I see. I think I was confused since, in my mind, there are many Turing machines that simply do not “optimize” anything. They just compute a function.
I think I wanted to point to a difference in the computational approach of different algorithms that find a path through the universe. If you chain together many locally found heuristics, then you carve out a path through reality over time that may lead to some “desirable outcome”. But the computation would be vastly different from another algorithm that thinks about the end result and then makes a whole plan of how to reach this. It’s basically the difference between deontology and consequentialism. This post is on similar themes.
I’m not at all sure if we disagree about anything here, though.
I would say that if you remember the plan and retrieve it later for repeated use, then you do this by learning and the resulting computation is not planning anymore. Planning is always the thing you do at the moment to find good results now, and learning is the thing you do to be able to use a solution repeatedly.
Part of my opinion also comes from the intuition that planning is the thing that derives its use from the fact that it is applied in complex environments in which learning by heart is often useless. The very reason why planning is useful for intelligent agents is that they cannot simply learn heuristics to navigate the world.
To be fair, it might be that I don’t have the same intuitive connection between planning and learning in my head that you do, so if my comments are beside the point, then feel free to ignore :)
Conceptually it does, thank you! I wouldn’t call these parameters and hyperparameters, though. Low-level and high-level features might be better terms.
Again, I think the shard theory of human values might be an inspiration for these thoughts, as well as this post on AGI motivation which talks about how valence gets “painted” on thoughts in the world model of a brain-like AGI.
I personally don’t have good models for this. Ilya Sutskever mentioned in a podcast that under some models of bayesian updating, learning by heart is optimal and a component of perfect generalization. Also from personal experience, I think that people who generalize very well also often have lots of knowledge, though this may be confounded by other effects.