It does seem like LLMs struggle with “trick” questions that are ironically close to well known trick questions but with an easier answer. Simple Bench is doing much the same thing and models do seem to be improving over time. I guess the important question is whether this flaw will effect more sophisticated work.
On another note I find your question 2 to be almost incomprehensible and my first instinct would be to try to trap the bug by feeling for it with my hands.
It does seem like LLMs struggle with “trick” questions that are ironically close to well known trick questions but with an easier answer. Simple Bench is doing much the same thing and models do seem to be improving over time. I guess the important question is whether this flaw will effect more sophisticated work.
On another note I find your question 2 to be almost incomprehensible and my first instinct would be to try to trap the bug by feeling for it with my hands.