Our AI can parse documents with xx% accuracy [...] the people receiving these claims (primarily investors, but also buyers/ procurement teams) have no real means to reliably verify these claims
What they should do, in my opinion, is have a few actual documents prepared (maybe with actual names of people and companies replaced), and try them.
To make it easier for them, an AI could be used to evaluate the correctness of the answer. Make the AI under the answer a few questions related to the document. Then take its output, and use your current AI to check the answers—by giving it the output, and the human-written answers by your local expert, and asking whether the answers match, and to summarize the nature of disagreement (just in case the mismatches is caused by the tested AI providing a better or more nuanced answer).
However I suspect that in real world the actual mechanism that companies use to decide which products to buy is… much less related to the actual quality of the product, which makes this kind of testing irrelevant.
Just some guesses:
What they should do, in my opinion, is have a few actual documents prepared (maybe with actual names of people and companies replaced), and try them.
To make it easier for them, an AI could be used to evaluate the correctness of the answer. Make the AI under the answer a few questions related to the document. Then take its output, and use your current AI to check the answers—by giving it the output, and the human-written answers by your local expert, and asking whether the answers match, and to summarize the nature of disagreement (just in case the mismatches is caused by the tested AI providing a better or more nuanced answer).
However I suspect that in real world the actual mechanism that companies use to decide which products to buy is… much less related to the actual quality of the product, which makes this kind of testing irrelevant.