One cheap test might be to translate into Dutch and then back to English with the same prompt twice. How garbled does the output end up? Does it elide important nuances?
(Though this might overestimate the quality given that the original pieces in English are likely in the training data. If you’re fluent in another language, I’d be quite curious about the results in the target language).
That is a sensible test! However, in my testing, I just asked a fluent-in-Dutch co-worker about the results, and he was satisfied. There were some problems, but it was basically OK.
Depending on how high a bar you want to meet, it may be worthwhile to also try with a language farther from English than Dutch is. If I really wanted to be thorough, I might try to take two bilingual people, give one the English and one the non-English, and ask them to discuss the results with one another and/or with you. Or, even better, give it to people who don’t speak English, and don’t know it was translated from English, and therefore can’t subconsciously use knowledge of English to correct translation problems. But for a lot of use cases I agree that these would be unnecessary.
One cheap test might be to translate into Dutch and then back to English with the same prompt twice. How garbled does the output end up? Does it elide important nuances?
(Though this might overestimate the quality given that the original pieces in English are likely in the training data. If you’re fluent in another language, I’d be quite curious about the results in the target language).
That is a sensible test! However, in my testing, I just asked a fluent-in-Dutch co-worker about the results, and he was satisfied. There were some problems, but it was basically OK.
That’s good news!
Depending on how high a bar you want to meet, it may be worthwhile to also try with a language farther from English than Dutch is. If I really wanted to be thorough, I might try to take two bilingual people, give one the English and one the non-English, and ask them to discuss the results with one another and/or with you. Or, even better, give it to people who don’t speak English, and don’t know it was translated from English, and therefore can’t subconsciously use knowledge of English to correct translation problems. But for a lot of use cases I agree that these would be unnecessary.