Thanks!
I wonder if you wouldn’t get better results by just grabbing the top system prompts for similar tasks and hard-sticking them onto the models, maybe with a bit of manual prompt engineering, such that e.g. Gemini always sees a very explicit order to assume user error when something goes wrong.
So this would forego our ability to assess how well they do autonomously and make the scenario more similar to having custom or dynamic prompting per task. That is also interesting, but aims a different objective, I think.
The other issue is that open-ended real-world tasks are a bit unfair to give to production LLMs, on the basis that anything these LLMs could do on their own would’ve already been done by an enterprising human using the very same LLMs, but with the intent to succeed[1] rather than the intent to evaluate the model’s performance.
I’m not sure if just seeing how far they get is “unfair” but I agree they are much less likely to succeed indeed!
If I love Jack and Jill similarly as I love my children, I expect to error out, yes. In practice, a real jack and jill have many other properties, and flipping any switch might be more down to how many healthy years they have or how many dependents or some such. I’m not sure. Also depends how much time you give me to deliberate.