But I don’t think the right conclusion is “Unpredictable!” so much as “So put in the work if you care to predict it?”.
I still think there’s a bit of post-hoc reasoning here; it’s easy to rationalise why we would like ice cream, specifically, after the fact, and harder to make novel predictions that are that spot-on. Though as you say prediction can bring you a bit further than expected.
There’s also the matter of information. How much information are the aliens even given to work from? To predict “chocolate ice cream” you would need data on the chemical composition of our biosphere, the ecological niches occupied by various animals, how mammalian biology and child-rearing works, how parasites work, how our biochemical energy producing mechanisms work, how DNA bases, insect nervous systems, and human nervous systems work (to guess that caffeine or similar compounds might be produced and enjoyed) and who knows what else. That’s a lot of info, probably much more than we comparably have for hypothetical future ASIs. Absent all that, you get stuck with stupid predictions like “gasoline” or “bear fat with honey and salt”.
I still think there’s a bit of post-hoc reasoning here; it’s easy to rationalise why we would like ice cream, specifically, after the fact, and harder to make novel predictions that are that spot-on. Though as you say prediction can bring you a bit further than expected.
There’s also the matter of information. How much information are the aliens even given to work from? To predict “chocolate ice cream” you would need data on the chemical composition of our biosphere, the ecological niches occupied by various animals, how mammalian biology and child-rearing works, how parasites work, how our biochemical energy producing mechanisms work, how DNA bases, insect nervous systems, and human nervous systems work (to guess that caffeine or similar compounds might be produced and enjoyed) and who knows what else. That’s a lot of info, probably much more than we comparably have for hypothetical future ASIs. Absent all that, you get stuck with stupid predictions like “gasoline” or “bear fat with honey and salt”.