Hi Francisco,
Thank you for your comment! Intuitively, I agree with you that asking the question in the same-turn would result in potentially higher self-scoring, and is worth checking out. We will conduct these additional experiments and ablations and include them in our main paper.
Michael Ofengenden
Karma: 39
Hi, thank you for your comment.
Yes, we’ve actually read all three of these papers and more! This blog post serves to see if these conclusions still stand across new frontier coding agents on very long horizon tasks, Can LLMs Perceive Time evaluates on tasks that often take less than a few minutes. Our graph shows that the estimation of tasks depends on task length. Your LLM Agents are Temporally Blind almost purely depend on whether agents receive time-stamps in their system prompts. We want to examine behavior on long-horizon tasks where agents have all the tools to their disposal.
An even more recent paper, BAGEN: Are LLM Agents Budget-Aware? came up with opposite conclusions for budget estimation. We would like to see if ours better reflect experiences that users have with coding agents. We also find these papers don’t include the harnesses and ablations that we want to test against (we’ll expand to Opencode and Chinese ones). Finally, we’re also runing an updated wall-clock duration-following benchmark, which is emphasized in next steps.
We will also make sure to include a very thorough related works section in our coming paper!