The fact still stands that ice cream is what we mass produce and send to grocery stores.
Yeah, I guess this exact observation is critical to making Eliezer’s analogy accurate.
IMO “predicting that bear fat with honey and salt tastes good” is analogous to “predicting that harnessing a star’s power will be an optimization target” — something we probably can successfully do.
And “predicting bear fat (or some kind of rendered animal fat) with honey and salt will be a popular treat”—the thing we couldn’t have done a-priori—is analogous to “predicting solar-to-electricity generator panels will be a popular fixture on many planets” (since the details probably will turn out to have some unpredictable twists), and also to “predicting that making humans satisfied with outcomes will be an optimization target for AIs in the production environment as a result of their training”.
I think this analogy is probably right, but the sense in which it’s right seems sufficiently non-obvious/detailed/finicky that I don’t think we can expect most people to get it?
Plus IMO it further undermines the pedagogical value of this example to observe that a drinkable form of ice cream (shakes) is also popular, plus there’s gelato / frozen yogurt / soft serve, and then thick sweet yogurts and popsicles… it’s a pretty continuous treat-fitness landscape.
I do think Eliezer is importantly right that the exact peak market-winning point in this landscape, would be hard to predict a-priori. But is the hardness also explained by the peak being dependent on chaotic historical/cultural forces?
And that’s why I personally don’t bring up the bear fat thing in my AI danger explanations.
I agree. I never saw this analogy as being about the specific bear_fat_and_honey snack. Bear fat and honey is just a vivid example to say that even though we want fat, sugar and salt, some elements of the class of foods with fat, sugar and salt are ranked as more appetizing. And it is this preference ranking what the observing aliens could not predict from knowing about preference for fat, sugar and salt.
Makes sense. Only problem is, bear fat + sugar + salt seems qualitatively pretty similar to ice cream. It doesn’t seem like it neglected the qualitative spirit of why ice cream is good, which just adds to the fine parsing needed to get value out of this.
Yeah, I guess this exact observation is critical to making Eliezer’s analogy accurate.
IMO “predicting that bear fat with honey and salt tastes good” is analogous to “predicting that harnessing a star’s power will be an optimization target” — something we probably can successfully do.
And “predicting bear fat (or some kind of rendered animal fat) with honey and salt will be a popular treat”—the thing we couldn’t have done a-priori—is analogous to “predicting solar-to-electricity generator panels will be a popular fixture on many planets” (since the details probably will turn out to have some unpredictable twists), and also to “predicting that making humans satisfied with outcomes will be an optimization target for AIs in the production environment as a result of their training”.
I think this analogy is probably right, but the sense in which it’s right seems sufficiently non-obvious/detailed/finicky that I don’t think we can expect most people to get it?
Plus IMO it further undermines the pedagogical value of this example to observe that a drinkable form of ice cream (shakes) is also popular, plus there’s gelato / frozen yogurt / soft serve, and then thick sweet yogurts and popsicles… it’s a pretty continuous treat-fitness landscape.
I do think Eliezer is importantly right that the exact peak market-winning point in this landscape, would be hard to predict a-priori. But is the hardness also explained by the peak being dependent on chaotic historical/cultural forces?
And that’s why I personally don’t bring up the bear fat thing in my AI danger explanations.
I agree. I never saw this analogy as being about the specific bear_fat_and_honey snack. Bear fat and honey is just a vivid example to say that even though we want fat, sugar and salt, some elements of the class of foods with fat, sugar and salt are ranked as more appetizing. And it is this preference ranking what the observing aliens could not predict from knowing about preference for fat, sugar and salt.
Makes sense. Only problem is, bear fat + sugar + salt seems qualitatively pretty similar to ice cream. It doesn’t seem like it neglected the qualitative spirit of why ice cream is good, which just adds to the fine parsing needed to get value out of this.