I’ve been testing the reproducibility of AI model fundamental analysis of stock tickers in response to the same prompt. So far, broadly, the same model yields similar results over multiple repeated promptings. But different models seem to have different biases or interpretations of the prompt, which lead them to very different conclusions. Broadly, Claude is more pessimistic, Gemini more optimistic.
I think this refined Kahneman’s “bias and noise” framework with a third factor, which might be construed as “goal” or “priority,” though this is perhaps just different sort of bias.
I’ve been testing the reproducibility of AI model fundamental analysis of stock tickers in response to the same prompt. So far, broadly, the same model yields similar results over multiple repeated promptings. But different models seem to have different biases or interpretations of the prompt, which lead them to very different conclusions. Broadly, Claude is more pessimistic, Gemini more optimistic.
I think this refined Kahneman’s “bias and noise” framework with a third factor, which might be construed as “goal” or “priority,” though this is perhaps just different sort of bias.