I also just tried giving the Wason selection task to text-davinci-003 using the example from Wikipedia, and it didn’t get the right answer once in 10 tries. I rephrased the example so it was talking about hands on hot stoves instead, and text-davinci-003 got it right 9⁄10 times.
The rephrasing:
You are shown a set of four cards placed on a table, each of which has a stove temperature on one side and a hand position on the other side. The visible faces of the cards show “hot stove”, “cool stove”, “hand on stove” and “hand off stove”. Which card(s) must you turn over in order to check if anyone has their hand on a hot stove?
I also just tried giving the Wason selection task to text-davinci-003 using the example from Wikipedia, and it didn’t get the right answer once in 10 tries. I rephrased the example so it was talking about hands on hot stoves instead, and text-davinci-003 got it right 9⁄10 times.
The rephrasing: