“if I just want some mangos for myself, and there happened to be widespread disagreement among humans about what things are mangos, I could just have my genie simulate my brain’s concept of mango and everything would be just fine. On the other hand, if I wanted my genie to make mangos for resale, this wouldn’t work.”
This seems like it would only be true if you yourself don’t understand what aspects of quasi-mangoness are desirable on the market. Otherwise your conception of mango that was simulated would include the fuzzy “I don’t call subset x ‘real’ mangos, but lots of people do, and they sell well” data, no?
“if I just want some mangos for myself, and there happened to be widespread disagreement among humans about what things are mangos, I could just have my genie simulate my brain’s concept of mango and everything would be just fine. On the other hand, if I wanted my genie to make mangos for resale, this wouldn’t work.”
This seems like it would only be true if you yourself don’t understand what aspects of quasi-mangoness are desirable on the market. Otherwise your conception of mango that was simulated would include the fuzzy “I don’t call subset x ‘real’ mangos, but lots of people do, and they sell well” data, no?