This is a fantastic point well articulated, reminiscent of some conversations we had a few months ago at Lightcone.
I’d say that a “general-purpose search” process is something which:
Takes in a problem or goal specification (from a fairly broad range of possible problems/goals)
… and returns a plan which solves the problem or scores well on the goal
I think we probably agree on what things there actually are, but I think this particular definition of ‘general purpose search’ is slightly too general to be a most useful pointer/carving.
This because it seems to include things like matrix inversion for least-squares solutions (unless ‘from a fairly broad range of possible problems/goals’ is taken to preclude this meaningfully?) which I deem importantly different. I’d class matrix inversion least-squares as a (powerful) heuristic[1] (a ‘proposal’ in my deliberation terminology), but not as (proper) search itself.
I think it remains useful to distinguish algorithms which evaluate/promote or otherwise weigh proposals[2]. This is what I’ve started calling ‘proper deliberation’ and it’s generally what I mean when I talk about search.
In the case of applying matrix inversion to ordinary least squares, for me, the ‘general deliberation’ consists of something like
noticing the relevant features of the problem (this is ‘abstraction/pattern-matching magic’)
cognitively retrieving the OLS abstraction and matrix-inversion as a cached heuristic (this is ‘propose’)
thinking ‘yes, this will work’ (this is ‘promote’)
applying matrix inversion to solve
A clever/practised enough deliberator does steps 1, 2 and 3 ‘right’ and doesn’t need to iterate for this particular problem (my point here is that if your heuristics are good enough you can deliberate with only one proposal and say ‘yep, good enough, let’s go’). But counterfactually step 2 might make various alternative proposals, or step 3 might think ‘actually there are too many dimensions in this case for inversion to be tractable’ or something, and thus there’s an evaluation and an internal update.
I don’t require this to be a ‘full consequentialist model-based valuation’, but that would be one example. See my deliberation simple examples for less sophisticated versions which are quite pervasive and nevertheless embody the ‘propose;promote’ breakdown
This is a fantastic point well articulated, reminiscent of some conversations we had a few months ago at Lightcone.
I think we probably agree on what things there actually are, but I think this particular definition of ‘general purpose search’ is slightly too general to be a most useful pointer/carving.
This because it seems to include things like matrix inversion for least-squares solutions (unless ‘from a fairly broad range of possible problems/goals’ is taken to preclude this meaningfully?) which I deem importantly different. I’d class matrix inversion least-squares as a (powerful) heuristic[1] (a ‘proposal’ in my deliberation terminology), but not as (proper) search itself.
I think it remains useful to distinguish algorithms which evaluate/promote or otherwise weigh proposals[2]. This is what I’ve started calling ‘proper deliberation’ and it’s generally what I mean when I talk about search.
In the case of applying matrix inversion to ordinary least squares, for me, the ‘general deliberation’ consists of something like
noticing the relevant features of the problem (this is ‘abstraction/pattern-matching magic’)
cognitively retrieving the OLS abstraction and matrix-inversion as a cached heuristic (this is ‘propose’)
thinking ‘yes, this will work’ (this is ‘promote’)
applying matrix inversion to solve
A clever/practised enough deliberator does steps 1, 2 and 3 ‘right’ and doesn’t need to iterate for this particular problem (my point here is that if your heuristics are good enough you can deliberate with only one proposal and say ‘yep, good enough, let’s go’). But counterfactually step 2 might make various alternative proposals, or step 3 might think ‘actually there are too many dimensions in this case for inversion to be tractable’ or something, and thus there’s an evaluation and an internal update.
Peter Barnett and Ian McKenzie coined ‘God-level heuristic’ for really solid mathematically-justified heuristics like this, which I quite like
I don’t require this to be a ‘full consequentialist model-based valuation’, but that would be one example. See my deliberation simple examples for less sophisticated versions which are quite pervasive and nevertheless embody the ‘propose;promote’ breakdown