Interesting coincidence—my own “torture test” for the model is a variant of your question, with a couple of additional twists.
We're running a physics experiment:
- we have a cube with volume 1000ml and weighting 500g
- we have a cubic container with the side of 110mm
- container is filled with 231ml of water
- the cube is placed in it.
Describe the final state of the container in terms of the water level and the distance from the bottom of the container to the bottom of the cube.
Models tend to screw up both figuring out the water level, and reasoning about the distance to the bottom of the cube. If they do get it right, I follow up with a question “explain how the cube that displaces 500ml can float in only 231ml of water.” o1 and gemini 2.0 got it mostly right. Other models were indeed struggling. 231ml of water is just enough to submerge the cube if you glue it to the bottom and a lot of “thinking” models got tripped by that and reasoned that the cube will not float.
Interesting coincidence—my own “torture test” for the model is a variant of your question, with a couple of additional twists.
Models tend to screw up both figuring out the water level, and reasoning about the distance to the bottom of the cube. If they do get it right, I follow up with a question “explain how the cube that displaces 500ml can float in only 231ml of water.” o1 and gemini 2.0 got it mostly right. Other models were indeed struggling. 231ml of water is just enough to submerge the cube if you glue it to the bottom and a lot of “thinking” models got tripped by that and reasoned that the cube will not float.