43% is the percentage of questions scoring 2 which is fully correct...They are not answerable by general veterinary knowledge, breed specific production parameters for White Fulani or field recognition cues for trypanosomiasis as a Fulani herdsman would use them do not appear in Western literature.
I am a little confused then, how is it possible to score as high as 43% in giving ‘fully correct’ answers on topics which are not answerable by what sounds like the only material any LLM would have access to in training? If you could answer them by ‘general veterinary knowledge’ or documented breed-level knowledge, then maybe I wouldn’t be surprised, but you specifically claim that to not be the case. Are the LLM doing this using online ‘Nigerian veterinary curriculum materials ’ or what?
The 43% reflects a spectrum within the benchmark, not a flat score across equally inaccessible questions. Here’s the breakdown
Tropical Disease Knowledge: 60
Local Treatment Context: 52.9%
Production & General Context: 48.6%
Breed Knowledge: 42.9%
Terminology: 41.4%
Ethnoveterinary practices: 35.7%
The model performs where training data fragments exist ,FAO reports, tropical medicine literature, publicly available Nigerian curriculum. It drops on the categories built from oral tradition and field specific practice. That gradient is intentional and it’s part of the finding.
That’s exactly what the category breakdown shows. Where general veterinary knowledge applies Tropical Disease the model scores 60%. Where it cannot, Ethnoveterinary field practice, oral tradition, specific knowledge it drops to 35.7%. The gradient answers your question directly. The model is doing exactly what you’d predict: performing on accessible literature and failing on the knowledge that has no systematic documentation.
I am a little confused then, how is it possible to score as high as 43% in giving ‘fully correct’ answers on topics which are not answerable by what sounds like the only material any LLM would have access to in training? If you could answer them by ‘general veterinary knowledge’ or documented breed-level knowledge, then maybe I wouldn’t be surprised, but you specifically claim that to not be the case. Are the LLM doing this using online ‘Nigerian veterinary curriculum materials ’ or what?