So, for optimizing a process with many variables (like tomato sauce), estimate the direction you might improve each variable and move a small amount in that direction, instead of exhaustively testing each variable independently? Because we know that actually works pretty well.
There are some things that Alice does that a gradient descent optimizer doesn’t, though, which might also be important. Particularly: she recognizes which variables are likely to affect which features, and she adds a new variable (carrot) from a rather large search space.
I wonder if Alice is vulnerable to a local minimum trap—she might converge upon pretty good tomato sauce that she can’t improve upon, while Bob exhaustively searches for (and might eventually find) a perfect tomato sauce. I agree with the point, though—if you try Bob’s strategy, you’ll be eating a lot of bad sauce in the process of exploring all possible ingredient combinations.
Bob strategy doesn’t lead to exploring all possible ingredient combinations. The scientific approach of standardizing things generally reduces the search space.
If you look at pharmaceutical interentions you find find often one active ingredient per formula.
On the other hand if you look at the food supplement community you have a lot of formulations that mix a lot more different active ingredients together.
So, for optimizing a process with many variables (like tomato sauce), estimate the direction you might improve each variable and move a small amount in that direction, instead of exhaustively testing each variable independently? Because we know that actually works pretty well.
There are some things that Alice does that a gradient descent optimizer doesn’t, though, which might also be important. Particularly: she recognizes which variables are likely to affect which features, and she adds a new variable (carrot) from a rather large search space.
I wonder if Alice is vulnerable to a local minimum trap—she might converge upon pretty good tomato sauce that she can’t improve upon, while Bob exhaustively searches for (and might eventually find) a perfect tomato sauce. I agree with the point, though—if you try Bob’s strategy, you’ll be eating a lot of bad sauce in the process of exploring all possible ingredient combinations.
Bob strategy doesn’t lead to exploring all possible ingredient combinations. The scientific approach of standardizing things generally reduces the search space.
If you look at pharmaceutical interentions you find find often one active ingredient per formula.
On the other hand if you look at the food supplement community you have a lot of formulations that mix a lot more different active ingredients together.