Dumb question: why can’t you do a find replace on your evals to replace the date with today’s date?
We thought about that, but then it’s not reproducible if we want to run it for new models later
But won’t the newer models already know that they have knowledge beyond the date you say? I don’t see how hard coding a date in your evals would ever lead to a newer model not knowing it was being lied to
We set it to some date in the future
Dumb question: why can’t you do a find replace on your evals to replace the date with today’s date?
We thought about that, but then it’s not reproducible if we want to run it for new models later
But won’t the newer models already know that they have knowledge beyond the date you say? I don’t see how hard coding a date in your evals would ever lead to a newer model not knowing it was being lied to
We set it to some date in the future